feat: add refs metadata table#2814
Open
gmhelmold wants to merge 2 commits into
Open
Conversation
cf93440 to
0cf4c2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
First table slice of the metadata-tables epic, as announced on the epic thread (refs / history / metadata_log_entries are the opening slices).
What changes are included in this PR?
Adds the
refsmetadata table, exposing a table's snapshot references (branches and tags) as rows, matching Iceberg's referenceRefsTable.java.Schema (field ids 1–6, identical to the reference
SNAPSHOT_REF_SCHEMA):typeis the uppercase ref kind —"BRANCH"/"TAG"— matching Java'sSnapshotRefType.name().min_snapshots_to_keep,max_snapshot_age_in_ms, andmax_reference_age_in_ms; a tag fills onlymax_reference_age_in_ms, leaving the other two null.TableMetadata::refsis aHashMap, so the scan collects into aBTreeMapto keep output deterministic.pub(crate)field directly, so the public surface is unchanged.Registry wiring:
MetadataTableType::Refs(+as_str/TryFrom), theMetadataTable::refs()accessor, and the DataFusionTableProviderschema/scan arms. The$refsDataFusion table name is enumerated automatically viaMetadataTableType::all_types()(strumEnumIter).Are these changes tested?
Yes.
inspect::refs::tests:test_refs_tableuses the sharedTableTestFixture(amainbranch auto-injected by V2 normalization with all-null retention, plus atesttag). It pins row order, theBRANCH/TAGtype strings, and the tag'smax_reference_age_in_ms.test_refs_table_retention_fieldsuses inline metadata (TempDir-based, deterministic) whosemainbranch carries explicit retention — the load-bearing golden for the branch column mapping (min_snapshots_to_keep=5,max_snapshot_age_in_ms=86400000,max_reference_age_in_ms=259200000), which the all-null fixture branch does not exercise.test_provider_list_table_namesnow listsmy_table$refs;show_tables.sltlists the$refstables;insert_into.sltselects$refson a snapshot-less table and expects zero rows (V2 normalization only injectsmainonce a current snapshot exists).Red→green verified locally: flipping the branch type string
"BRANCH"→"branch"fails both unit goldens on the type column; restoring passes.cargo clippy --all-targets --all-features --workspace -- -D warningsclean.Are there any user-facing changes?
New feature.
SELECT * FROM <table>$refs(via DataFusion) andtable.inspect().refs()now expose a table's snapshot references.