v0.1.0

lance-csr 0.1.0

lance-csr is a fork of lance-graph at commit 3014793. It is a research test bed for making multi-hop path traversal fast with a CSR adjacency index and Apache DataFusion over the Lance format. This first release replaces the tree with work developed on top of that commit. Thanks to the lance-graph maintainers and contributors for the foundation: the Cypher parser, semantic analysis, the DataFusion planner, the graph configuration and the Python bindings.

Breaking changes

• The Python distribution is lance-csr, and the import is lance_csr (previously lance-graph / lance_graph). The Rust crates keep their lance-graph* names for now.
• Python 3.12 or newer is required.
• Removed: Delta Lake and Unity Catalog support, the catalog connector layer, and the knowledge-graph application (CLI, web service, Streamlit app, LLM extraction). No deltalake release supports DataFusion 54, and the rest sat outside the Cypher engine.

Dependencies

• Lance 13 (Rust and pylance >=13,<14), DataFusion 54, Arrow 58 and PyO3 0.28. lance-graph's Lance 1.0 reader panics on files written by current pylance.
• The Rust toolchain is pinned in rust-toolchain.toml, and the extension crate now has its own Cargo.lock.

Engine: Lance snapshots, refresh and streaming

• CypherEngine opens each Lance dataset as a lazy DataFusion table provider instead of copying it into memory. You choose per table: pass lance.dataset(...) to read from disk, or an Arrow table to keep it in memory.
• Each dataset is pinned to the version you pass in. refresh() builds and validates a full replacement, then swaps it in atomically. Queries already running keep their snapshot, and a failed refresh changes nothing.
• New methods: execute_batches() (streams a pyarrow.RecordBatchReader), explain(), refresh(), dataset_versions(), csr_stats() and node_index_stats().
• In-memory tables are split into batches across all cores, so operators that don't repartition still run in parallel.

CSR traversal (use_csr=True)

• The engine builds a compressed sparse row (CSR) adjacency index for each relationship type in both directions, from the relationship table alone, so node-table writes never invalidate it. The index runs inside DataFusion as extension operators, and the default (use_csr=False) behavior is unchanged.
• Hops: single hops and every hop of a variable-length path, whether directed, incoming or undirected, expand through the CSR instead of a relationship scan plus hash join. Any hop the CSR can't express falls back to a join in the same query.
• Target lookup by row position: target node columns are fetched directly by Lance row id (or Arrow row index) instead of through a second hash join. Columns no expression needs are pruned, and the target id is rebuilt from the neighbor value rather than read.
• Filters and closing edges: WHERE conditions on the target run inside the expansion. They fetch only the filter's columns, then the remaining columns for rows that pass. When the target is already bound (a cycle), the CSR keeps only the matching neighbor instead of expanding everything.
• Incremental maintenance: refresh() diffs Lance versions fragment by fragment and applies appended edges and deletions (as tombstones) on top of the shared base index. It rebuilds once changes exceed 25% of the base, or when the diff can't be computed. There's no API change: commit with pylance and call refresh().
• Hashing: the CSR's integer-keyed maps use ahash instead of SipHash, which dominated expansion time in profiles.

Planning

• Re-rooting: a MATCH starts at its most selective node (an id lookup first, then equality, then range filters), reversing path segments as needed.
• Statistics: DataFusion gets distinct counts for node-id and edge-endpoint columns, so its join sizing and build-side choices are right, including for equality filters pushed into Lance scans.
• Continued paths: when a pattern continues earlier rows (OPTIONAL MATCH, or MATCH after WITH) and both ends are bound, expansion starts from the end with the smaller mean fan-out. On LDBC IC5 this cut the query from about 1.7 s to about 0.24 s. A path bound only at its far end is now read from that end.

Cypher support

Enough to run all 14 LDBC Interactive complex queries:

• Patterns: relationship-type alternation [:A|B], also inside variable-length paths; multi-label nodes (m:Post:Comment) as a union of labels; optional endpoint labels per relationship type (with_relationship(type, src, dst, source_label, target_label)).
• Clauses: chains of WITH, WITH DISTINCT, WITH ... WHERE, whole nodes carried through WITH, OPTIONAL MATCH, and WHERE between reading clauses.
• Predicates: EXISTS { ... } and pattern predicates (NOT (a)-[:R]-(b)), in WHERE and in CASE conditions.
• Paths: shortestPath and allShortestPaths, as a breadth-first search over the CSR (or over an edge scan without it), plus length(p) and nodes(p).
• Expressions: arithmetic with precedence and parentheses, searched and simple CASE, coalesce, list literals, indexing, size, range, toString, date_part, to_epoch_ms, collect(DISTINCT ...), and the NULL literal in any case.

Correctness fixes to behavior inherited from lance-graph

Each fix has a regression test.

• Lance inputs were silently capped at 10,000 rows, because the bridge called a pylance method that doesn't exist and fell back to take(range(10000)).
• Undirected patterns dropped every match that needs an incoming edge. For example, a person with 30 neighbors returned 29.
• Hops inside variable-length paths could be silently dropped, because path temporaries had no label mapping.
• Multi-label nodes scanned only their first label.
• collect() kept nulls.
• Aggregates with camelCase aliases (count(*) AS postCount) failed to plan, and so did a camelCase scalar carried through a second WITH.
• WITH didn't end variable scope.
• WITH b (a whole node) failed to plan.
• Overflowing numeric literals panicked, and LIMIT -1 / SKIP -1 wrapped around to 2^64 − 1.
• Empty Arrow inputs raised an error instead of returning empty results.
• Subclasses of lance.LanceDataset were loaded into memory and ignored by refresh().

Packaging

• Prebuilt abi3 wheels for Linux x86-64 (manylinux_2_28, via zig) and native Apple Silicon (macOS 11+), plus an sdist. Every wheel is tested on CPython 3.12, 3.13 and 3.14 in a clean environment without Rust.
• The CPU baseline is unchanged from lance-graph: haswell (AVX2/FMA) on Linux x86-64 and apple-m1 on macOS.
• Releases publish to PyPI through trusted publishing.

Known limitations

• The CSR handles integer node ids only. Other id types, inline target property filters and bound relationship variables use the join path.
• A CSR expansion keeps its input's partitioning, so a pattern rooted at one node runs its fan-out on one thread. Splitting large expansions at run time is future work.
• Relationship uniqueness is enforced for undirected variable-length paths of up to 3 hops, and only while no two edges join the same pair of nodes. It isn't enforced across separate pattern segments, which matches lance-graph.
• OPTIONAL MATCH and EXISTS plan their input twice.
• Alternation expands to at most 64 branches. Variable-length alternation is limited to 20 hops and to directed, unnamed segments.
• shortestPath supports a single relationship type, with no relationship variable or properties.