refactor(table): move transaction helpers down into lance-table - #8053
Draft
wjones127 wants to merge 5 commits into
Draft
refactor(table): move transaction helpers down into lance-table#8053wjones127 wants to merge 5 commits into
wjones127 wants to merge 5 commits into
Conversation
The function only compares an IndexMetadata name against the two system index name constants, both of which already live in lance-table's system_index module. Moving it there puts it next to the constants it reads; lance-index re-exports it so callers are unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
KeyExistenceFilter and friends were defined in the lance crate but depend only on arrow, lance-core's bloom filter, and the transaction protobuf in lance-table. The filter is serialized into that protobuf, so the table layer is where it belongs. lance::dataset::write::merge_insert::inserted_rows becomes a re-export, so callers are unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deciding which rows an overlay makes stale with respect to an index reads only fragment and index metadata: the coverage bitmaps, the overlay committed_version, and the indexed field ids. None of that needs the read path, so it moves to lance-table alongside the overlay format itself. lance::dataset::overlay keeps the read-resolution half and re-exports the three functions its callers use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
load_mem_wal_index_details, open_mem_wal_index, new_mem_wal_index_meta and update_mem_wal_index_compacted_sstables read and write the MemWAL index's IndexMetadata entry. Every type they touch already lives in lance-table's system_index module, so they join the data structures they serialize. lance::index::mem_wal keeps the dataset-level operations and re-exports the four helpers at their previous visibility. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build_manifest and restore_old_manifest took ManifestWriteConfig, whose timestamp field is the lance crate's mockable SystemTime. That mock is cfg(test) of the lance crate, so resolving the timestamp inside a lower crate would silently un-mock it. This adds lance_table::format::ManifestBuildConfig, which carries the timestamp already resolved to nanoseconds, and has ManifestWriteConfig convert into it at the call sites. The conversion stays in the lance crate, so the clock is still mockable. Conversion happens per attempt inside the commit retry loops, matching the previous behavior of resolving the timestamp on each build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
geruh
reviewed
Jul 28, 2026
geruh
left a comment
geruh
left a comment
Contributor
There was a problem hiding this comment.
Hey Will, did a pass here and looks like a straightforward refactor with everything moved and everything is lined up. Also, checked out locally and ran some tests. Just a left a few small nits, nothing blocking. Let me know what you think.
| ); | ||
| } | ||
| } | ||
| pub use lance_table::format::key_existence::*; |
Contributor
There was a problem hiding this comment.
nit: should we do a named import here? like what you're doing in /index/mem_wal.rs?
| } | ||
|
|
||
| // A missing `fragment_bitmap` means the index predates fragment-bitmap tracking; treat it as | ||
| // covering every fragment (matching `DatasetPreFilter::new`) so overlay-stale rows can't slip |
Contributor
There was a problem hiding this comment.
nit: the DatasetPreFilter location moved, so should we update to lance::index::prefilter::DatasetPreFilter::new
| //! Writers no longer update the index on every write. Instead, they update | ||
| //! shard manifests directly. This module provides functions to: | ||
| //! - Load the MemWAL index | ||
| //! - Update compacted SSTables (called during merge-insert commits) |
Contributor
There was a problem hiding this comment.
Seems like these comments were dropped in the merge. Are they no longer needed?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lance/src/dataset/transaction.rsis 6767 lines and is the next thing we want to move down intolance-table. Its production code turns out to have no dependency onDataset,Session, DataFusion, or async I/O — only six couplings to code abovelance-table. This PR clears five of them so the move itself can be a plain file rename.Each commit moves one self-contained piece down to the layer that already owns the types it touches, and leaves a re-export behind so no caller changes:
is_system_indexcompared anIndexMetadataname against two constants that already live inlance-table'ssystem_indexmodule. It now sits next to them;lance-indexre-exports it.KeyExistenceFilterand friends, previouslymerge_insert/inserted_rows.rs) depends only on arrow,lance-core's bloom filter, and the transaction protobuf. It is serialized into that protobuf, so it moves tolance-table.committed_version, and indexed field ids.lance::dataset::overlaykeeps the read-resolution half.IndexMetadataentry. Every type they touch was already inlance-table, so they join the data structures they serialize.ManifestBuildConfigis new.build_manifesttookManifestWriteConfig, whosetimestampfield is thelancecrate's mockableSystemTime— and that mock iscfg(test)of thelancecrate, so resolving the timestamp inside a lower crate would silently un-mock it. The new config carries the timestamp already resolved to nanoseconds, andManifestWriteConfigconverts into it at the call sites, keeping the clock mockable.The sixth coupling,
ManifestWriteConfigitself, deliberately stays inlancefor that reason.Not included
The move of
transaction.rsintolance-table, and its split into a module tree, come as two follow-up PRs stacked on this one. Splitting them keeps the cross-crate move reviewable as a detected rename rather than a 6767-line add/delete pair.io/commit/conflict_resolver.rsis the other half of the transaction story and a natural later target, but it depends onDatasetandDatasetIndexExt, so it stays put.Testing
The one behavioral question here is whether the mock clock still works, since that is what
ManifestBuildConfigexists to protect. Verified locally: theMockClock-based suites (dataset::cleanup,dataset::delta,dataset::tests::dataset_versioning) pass, 67 tests. Theto_build_config()conversion is called inside the two commit retry loops rather than hoisted above them, so each retry still resolves its own timestamp as before.