v0.6.0

What's Changed

Breaking:

• #109 (Rust): CsrIndex::edge_rows() and edge_rows_at() return an iterator over packed values instead of a slice, and CsrIndex::from_parts is replaced by from_packed. The Rust API is marked experimental.
• #109 (Python and Rust): refresh() is renamed to checkout_latest(), after Lance's LanceDataset.checkout_latest(). refresh() still works, but Python warns with DeprecationWarning and Rust marks it #[deprecated]. It will be removed in a later release.

Query results that change: none. Queries that failed to plan before (see #107) now run.

CSR:

• #109: with persist_csr=True, an engine over Lance datasets stores each CSR it builds inside the relationship dataset's directory, at _lance_csr/<version>.arrow, and later engines load it instead of building. Writes since the stored version apply as a delta. On GDC SF1 read lazily, engine start drops from 1.65 s to 0.46 s, and the stored files take 134 MiB (zstd-compressed). delete_stored_csr() removes them. Off by default.
• #109: each edge's row address is bit-packed in blocks of 128 values. The in-memory CSR at SF1 shrinks from 926 to 732 MiB in both memory and lazy mode, whether or not the CSR is stored.
• #109: csr_stats() reports update "loaded" for a CSR read from a stored file.

Planner:

• #108: pattern predicates such as WHERE NOT (friend)-[:knows]-(person) walk the CSR with a new PathExists operator instead of planning their input a second time, and list comprehensions keep each input row once instead of copying its list per element. IC10 is about 75% faster (97.5 → 24.2 ms in memory, 130.1 → 34.0 ms in lazy) and IC7 about 34% (14.4 → 9.8 ms, 23.0 → 14.6 ms). The complex suite's geometric mean improves by 10% in memory and 11% in lazy.
• #107: a pattern predicate in OPTIONAL MATCH ... WHERE that reads a variable from an earlier clause now plans instead of failing with No field named .... Physical planning no longer formats whole input plans for extension nodes.

Known trade-offs:

• IC14 is about 3% slower (49.0 to 50.6 ms against 47.1 to 48.4 ms in direct A/B runs), because its [n IN nodes(path) | n.ID] now goes through the new comprehension lowering (#108).
• IC3 in memory mode ran about 3% slower with #109 in a five-process A/B (25.6 to 27.6 ms against 25.1 to 25.7 ms). Its hops don't read edge rows, so the cause isn't known.

Other: #104 (crates publish to crates.io automatically after PyPI), #106 (a DataFusion internals section in the docs), plus a trimmed README that links the benchmark repositories (#109).

Benchmarks: from each PR, measured against main at the time with the CONTRIBUTING.md harness, both suites, both modes, on an Apple M5. Every answer check passed.

PRs

• ci: publish the crates to crates.io after PyPI by @prrao87 in #104
• docs: add a collapsed datafusion internals section by @prrao87 in #106
• fix(planner): plan optional match pattern predicates that read earlier variables by @prrao87 in #107
• perf(planner): walk pattern predicates on the csr and keep comprehension rows once by @prrao87 in #108
• refactor(engine): rename refresh() to checkout_latest() to match lance by @prrao87 in #110
• feat(csr)!: store the csr inside each lance dataset and pack edge rows by @prrao87 in #109
• chore: bump version 0.5.1 → 0.6.0 by @prrao87 in #111

Full Changelog: v0.5.1...v0.6.0