Release v0.5.42

[0.5.42] - 2026-05-04

End-to-end tiered storage: section data (LPG, RDF Ring, vector topology) can spill to mmap-backed disk under memory pressure or explicit configuration, with per-section tier overrides, introspection, and reload. Plus per-block columnar zone maps for selective range scans, paged HNSW topology, packed RDF Ring on disk, a WAL overlay for mutating mmap'd compact stores and a streaming top-K operator that fuses ORDER BY ... LIMIT into a single bounded-heap pass.

Added

• Per-section storage tier configuration: Config::with_section_tier(SectionType, TierOverride) pins individual sections to RAM or disk. ForceDisk spills at db open; ForceRam is hard-enforced (skipped in every spill loop). Auto (default) defers to the buffer manager.
• db.storage_tiers() introspection: returns a map of section type to current tier (InMemory / OnDisk / Uninitialized). New MemoryConsumer::current_tier trait method backs it with authoritative state.
• db.reload_eligible(target_fraction): brings spilled sections back into RAM in priority order, stopping when projected usage exceeds the target. Returns the count reloaded.
• Python bindings for tier control: GrafeoDB(path, section_tiers={'VectorStore': 'force_disk'}), db.storage_tiers(), db.reload_eligible(). Eight new pytest cases.
• Paged HNSW topology (vector v2): VectorStoreSection switches from bincode to a packed GVST envelope with per-index GTOP paged topology. MmapTopology serves neighbor lookups directly from Bytes slices; HnswIndex dispatches search through TopologyBackend { Heap, Mmap }. Heap drops >10x under mmap, recall is bit-identical.
• Packed RDF Ring (ring v2): bincode RdfRingSection replaced by a GRFR envelope (sorted term dictionary + three packed wavelet trees + two permutations + CRC32). swap_to_mmap shares refcounted Bytes slices from the spill mmap.
• Per-block columnar zone maps + iterator bounds: LPG compact-store columns lay out as 4 KiB blocks with per-block min/max. find_in_range_iter skips blocks via zone maps; RangeScanOperator streams matches into the push pipeline. Large speedups on selective range scans.
• WAL overlay for mutable mmap'd LPG: when the compact base is OnDisk, mutations route to a live LpgStore overlay; periodic merge folds it back via LayeredStore::merge_overlay_in_place. New merge_guard makes concurrent reads + writes + merges race-free. Persistent file + spilled base + overlay mutations round-trip across restarts.
• SectionType::OverlayDeletions: persists LayeredStore's deletion log so deletions of base entities survive close/reopen without an explicit compact(). Omitted from the container directory when empty. Marked required: false; meaningful once the directory parser learns to skip non-required unknowns (planned for 0.5.43).
• tracing events on tier transitions (with tracing feature): info events under grafeo::buffer and grafeo::tier for spill, reload, and ForceDisk applications.
• Storage Tiers documentation page: docs/architecture/memory/storage-tiers.md covering tiers, overrides, introspection, reload, and tracing conventions.
• PageFetcher trait + MmapPageFetcher impl: indirection layer that lets sections receive paged byte access without knowing the source, opening a future swap to vmcache or an explicit pager.
• Streaming top-K operator (TopKOperator): bounded-heap of size k, O(k) memory regardless of input cardinality, ~12x faster than the unfused Sort + Limit at N=1M. Planner fuses literal LIMIT k over Sort into a single physical operator; vector/text top-K still takes precedence and PROFILE-mode plans bypass the fusion for entry-count parity. (#326, @temporaryfix)
• var.prop IN [literals] property-index fast path: per-value index lookups unioned via NodeListOperator, deduped, label/MVCC-filtered. ~56x speedup on WHERE id IN $ids against a snapshot-loaded database. (#326, @temporaryfix)
• LogicalOperator::map_children: child-recursive optimizer passes descend without enumerating every variant.

Changed

• Vector store and RDF Ring section formats bumped to v2 (paged GVST and packed GRFR envelopes). Existing v1 bincode files keep loading via magic-byte detection and upgrade to v2 on the next checkpoint.
• TierOverride::ForceDisk enforcement is now targeted: previously triggered spill_all() for every consumer; now spill_consumer_by_name(...) per matching section, so unrelated consumers stay InMemory.
• Filter pushdown extended through Filter, LeftJoin, Apply, Union, Unwind: predicates inside OPTIONAL MATCH and correlated subqueries now reach the scan. New pre-pushdown pass propagates filter predicates across LeftJoin shared-variable boundaries so the right-side subtree of an OPTIONAL MATCH picks up the same WHERE constraints as the main MATCH (~8x on feed.hydrate-shaped queries). Sibling Filter commute is gated by a variable-scope check so predicates over path variables like LENGTH(p) >= 1 don't push past a label filter into a subtree where the path isn't yet bound. (#326, @temporaryfix)

Fixed

• MERGE ... ON CREATE/ON MATCH SET could not reference the MERGE variable (#317): expressions like ON MATCH SET c.description = coalesce(c.description, 'fallback') failed at the binder, and any non-trivial action expression that did pass through was silently lowered to Null. Binder now scopes the MERGE variable into ON CREATE / ON MATCH (per ISO/IEC 39075:2024 §15.5); planner emits a PropertySource::Expression for action expressions; operators evaluate them against an augmented row containing the merged entity. Reported by @Fraenkstan.
• TierOverride::ForceRam was silently a no-op: prior versions accepted the config but the buffer manager spilled ForceRam consumers anyway under pressure. The spill loop now consults a force_ram_consumers set and skips matching consumers in run_eviction_internal, spill_all, and spill_consumer_by_name.
• Persistent reopen lost the compact base after compact(): directory parser had no SectionType::CompactStore arm and load_from_sections had no handler, so reopens fell back to a legacy path that surfaced as a checksum mismatch. New extract_compact_base + wire_layered_after_load helpers rebuild the LayeredStore and consumer wiring on open.
• Concurrent merge could lose writes: stress test with 4 readers + writer + merger lost 426/500 writes in one run. Fixed via merge_guard: RwLock<()> (mutators take read, merge takes write).
• GRAFEO-X001: snapshot checksum mismatch opening v2 section files (#323, #324): read_snapshot was unconditionally a v1 reader and CRC'd zero bytes against the v2 directory CRC when the engine's open path fell through. Reader now early-returns on a v2 header (snapshot_length == 0 with non-empty header), letting the engine's v2 dispatch take over. Reported by @teipsum against an embedded production database.
• Silent corruption masking in read_section_directory (#323 follow-up): the v2 directory parser swallowed from_bytes errors with a wildcard, routing truncated directories and torn-page writes into the v1 read logic and surfacing as a misleading snapshot-checksum error. Parser now propagates errors with the failing offset, treats "v2 header but file too short" as an error rather than a v1 fallback, and verifies the directory page CRC. Regression tests cover both paths.
• LayeredStore deletions silently reverted on reopen (#323 follow-up): base nodes/edges deleted after compact() were tracked only in memory, so reopen made them reappear until the next compact(). New OverlayDeletions section persists the deletion log; open path seeds the layered store from it. Round-trip integration tests lock in the behaviour.
• Per-query spill directory leak (#323 follow-up): <spill_path>/query_<id>/ directories were created per query and never removed; one production day accumulated 358 empty subdirectories. SpillManager/AsyncSpillManager now expose with_owned_dir() so Drop removes the directory non-recursively (preserving unexpected siblings).
• active_db_header docstring drift: the function picks the higher-iteration slot unconditionally; checksum validation lives in the readers. Doc rewritten to match.
• Equality-scan regression from the Bytes-backed codec refactor: BitPackedInts, DictionaryEncoding, and ColumnCodec::{Float64,RawI64} switched to a two-variant Inline(Vec<T>) / Mapped(Bytes) store so in-RAM builds keep native slice iteration and mmap loads stay zero-copy. Closes the 19-29% CodSpeed regression on compact/find_nodes_by_property/{int_eq,dict_eq}.
• Property-index fast path silently disabled after compact(): LayeredStore::has_property_index now delegates to the overlay (was using the trait default false). Snapshot-loaded databases stop falling back to the ~3.5x-slower label-first scan. (#326, @temporaryfix)
• ORDER BY against a WITH alias dropped by RETURN: e.g. WITH x AS s ... RETURN x ORDER BY s failed with Variable 's' not found because the augmented-projection path skipped Variable sort keys. They now pass through alongside Property keys. (#326, @temporaryfix)
• PROFILE panicked on fused fast-path operators: property/range/IN-list paths absorbed their child NodeScan into one physical op, leaving build_profile_tree short an entry. Affected paths now emit a synthetic NodeScan entry; the factorized expand-chain fusion is also gated under PROFILE. (#326, @temporaryfix)

Deprecated

• TieredStore trait (grafeo_common::memory::buffer::TieredStore): never implemented anywhere; the Section trait (with swap_to_mmap + reload_to_ram) plus MemoryConsumer cover the same lifecycle. Marked #[deprecated(since = "0.5.42")] and scheduled for removal in 0.6.0. The StorageTier enum it shipped alongside is kept.

---

Thanks to @teipsum (Michael Lewis Cram) for reporting #323 with byte-level analysis from a production embedded database, and for #324's read_snapshot fix which lands cherry-picked with authorship preserved. The follow-up work in this release (stricter read_section_directory validation, durable LayeredStore deletions via the new OverlayDeletions section, and the per-query spill directory cleanup) addresses the deeper root causes the original report surfaced.

Thanks to @temporaryfix for the top-K operator, IN-list fast path, LeftJoin filter propagation, and the cluster of fixes around compact() and PROFILE in #326, reshaped to fit grafeo's map_children recursion pattern before merging.

Full Changelog: v0.5.41...v0.5.42