fix(mem_wal): keep sealed memtables queryable until flush commits

[hamersaw]

Problem

A concurrent reader could miss rows in the window between freeze_memtable and the flush task's manifest commit. During that window the frozen generation is no longer the active memtable, but is not yet recorded as a flushed generation in the manifest — so it falls out of the read union entirely.

Fix

• Keep frozen-awaiting-flush memtables in WriterState.frozen_memtables so they stay in the read union (Arc refcount, not a copy — the flush task holds them alive for the drain anyway).
• Drop them by generation only on flush-commit success. On failure they are retained until a later flush or WAL replay on reopen, so a transient flush error doesn't reopen the hole. Keyed by generation, so non-FIFO completion is fine.
• Add ShardWriter::in_memory_memtable_refs and LsmScanner::with_in_memory_memtables as the read-path entry point — captures active + frozen atomically under one state.read() (no torn freeze).
• Rename ActiveMemTableRef → InMemoryMemTableRef (frozen is just the immutable case; same shape). Back-compat alias and with_active_memtable test/back-compat entry point retained.
• LsmDataSourceCollector emits in-memory sources in ascending generation order (see below).

Source-ordering correctness fix

Adding the scan-level test surfaced a latent bug in the first cut: collect() pushed active (newest generation) before frozen (older). The planner relies on sources being in ascending-generation order — it reverses them to generation-DESC so the newest row gets the lowest stream index and wins the dedup tiebreaker. With active emitted first, a pk that was written, sealed, then re-written into the new active memtable before the sealed one flushed would resolve to the stale frozen value. The collector now sorts in-memory sources by generation, restoring active-wins across the active/frozen seam.

Tests

• collector: collect()/collect_for_shards() emit active + every frozen memtable, in ascending generation order (frozen deliberately registered out of order).
• planner: end-to-end scan over base + flushed + frozen + active asserting frozen rows are in the read union, frozen beats older flushed generations, and active beats frozen for the same pk.
• write: frozen handle dropped from the queryable set on flush-commit success; retained with rows intact on a fenced/failed flush (hole cannot reopen).
• Migrated the planner/vector_search test harnesses and the WAL-only accessor test onto the new frozen-aware entry point.

All 265 mem_wal tests pass; cargo clippy -p lance --tests -D warnings clean.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 90.23747% with 37 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/mem_wal/scanner/builder.rs	40.00%	18 Missing ⚠️
...ust/lance/src/dataset/mem_wal/scanner/collector.rs	83.13%	14 Missing ⚠️
rust/lance/src/dataset/mem_wal/write.rs	95.53%	3 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[jackye1995]

looks good to me