cleanup_old_versions can delete files held by long-running readers

[LuciferYang]

Description

A Dataset handle at an older version (via checkout_version) has no way to signal cleanup_old_versions that it's in use. If cleanup runs while a reader is mid-scan, the manifest and data files for that version can be deleted out from under it, and the scan fails with object-store NotFound errors.

Reproduce

1. Open a dataset and checkout_version(N) where N is older than lance.auto_cleanup.older_than.
2. Start a scan. While the scan is running, run cleanup_old_versions with a before_version that includes N (or let auto-cleanup fire on the next commit).
3. The scan fails mid-stream with NotFound from the object store.

Current workarounds

Create a temporary tag before the read and delete it after. Works, but every reader has to cooperate and it gets awkward on auto-cleanup-per-commit deployments — cleanup runs on every write.

What I'd want

A short-lived, renewable signal that a reader can publish for the version it's using: cleanup treats the version as retained while the signal is live, and a crashed reader doesn't pin the version forever. Something like an advisory lease with a TTL.

Happy to put up a PR if this direction sounds reasonable.

[LuciferYang]

cc @Xuanwo

[summaryzb]

Large, production-scale tables tend to have many concurrent writers — backfills, index builds, compactions, schema evolutions, row-level updates. A version-count-based TTL is more likely to trigger the exact failure mode this issue describes, while a time-based TTL ends
up retaining too many manifest and transaction files. Production deployments therefore urgently need an active version-liveness detection mechanism.

[jackye1995]

It is expected that version cleanup is outside the transaction domain, because object storage cannot atomically list and delete at the same time. The only way to make them atomic is to be like Iceberg to use a full manifest to track all historical versions, but this is an intentional decision to not do that because that has proven to be bad for commit performance. It also has big implications to other designs decisions like branching and tagging, see https://www.lancedb.com/blog/branching-and-shallow-clone for more details.

If you produce a lot of versions at the same time, I think the right way is to set a high number of versions to keep, or just fully turn off auto cleanup?

For storage that listing is lexicographically ordered, this should have no performance impact from listing perspective (see #5947 benchmark results). For ones not, I guess I should merge my PR #5997, but I currently don't have time to do that (it is a spec change so it require a vote). If you are impacted, would appreciate if you could help push that through 🙏

[jackye1995]

Of course if you do small commit too frequently, it would still be problematic as the above benchmark shows that manifest file would gradually grow. It would not be as bad as Iceberg because you don't roll up the historical version and protobuf can hold data much more efficiently, but there is a point it would break where commit becomes too slow and manifest size at read time starts to take too much memory (although I think that breaking point is pretty high). But if you really reach that breaking point, I think the resolution is https://lance.org/format/table/mem_wal/. Although I have to be honest that I've been only implementing this spec for LanceDB enterprise solution and open source only has the basic building block at this point in rust, would be nice if we could integrate it with engines like Spark or Flink.

[LuciferYang]

@jackye1995 Thanks for the detailed reply. I'd like to dig into the MemWAL suggestion a bit, because I think it might be addressing a different problem than the one this issue describes.

The race here is that a reader holding checkout_version(N) has no way to tell cleanup_old_versions to hold off on N. As far as I can tell, MemWAL doesn't really change that contract. It does reduce how often the base manifest advances, which makes cleanup less aggressive in practice. But base versions still exist, and a long-running reader on one of them still has no liveness signal to publish.

There's a second concern worth surfacing: a similar race seems to appear inside MemWAL itself. Flushed MemTables and WAL files need GC once a generation has been merged and indexes have caught up, and a long-running LSM scan that's still reading a flushed MemTable could hit the same kind of NotFound failure. The spec gates GC on retained base-version references, which covers time-travel reads but doesn't seem to cover active scans.

It's also worth noting that OSS MemWAL is still partial today (the merger, GC, WAL replay on ShardWriter::open, JNI bindings, and ShardSpec evaluation aren't in the tree yet), so it wouldn't be a near-term path even if it fit the problem.

So my take is that MemWAL fits the manifest-growth pain you described well, but probably isn't the right resolution for this issue, and may bring a similar race along with it at the LSM layer. Both cases come down to the same underlying need: letting a live reader pin a version against cleanup, while making sure a crashed reader doesn't pin it forever. A small, generic advisory-lease primitive in lance-core could serve both layers.