Skip to content

feat: add refs metadata table#2814

Open
gmhelmold wants to merge 2 commits into
apache:mainfrom
gmhelmold:feat/refs-metadata-table
Open

feat: add refs metadata table#2814
gmhelmold wants to merge 2 commits into
apache:mainfrom
gmhelmold:feat/refs-metadata-table

Conversation

@gmhelmold

Copy link
Copy Markdown

Which issue does this PR close?

First table slice of the metadata-tables epic, as announced on the epic thread (refs / history / metadata_log_entries are the opening slices).

What changes are included in this PR?

Adds the refs metadata table, exposing a table's snapshot references (branches and tags) as rows, matching Iceberg's reference RefsTable.java.

Schema (field ids 1–6, identical to the reference SNAPSHOT_REF_SCHEMA):

id column type required
1 name string
2 type string
3 snapshot_id long
4 max_reference_age_in_ms long optional
5 min_snapshots_to_keep int optional
6 max_snapshot_age_in_ms long optional
  • type is the uppercase ref kind — "BRANCH" / "TAG" — matching Java's SnapshotRefType.name().
  • Retention columns follow the reference mapping: a branch fills min_snapshots_to_keep, max_snapshot_age_in_ms, and max_reference_age_in_ms; a tag fills only max_reference_age_in_ms, leaving the other two null.
  • Rows are emitted name-ordered. TableMetadata::refs is a HashMap, so the scan collects into a BTreeMap to keep output deterministic.
  • No public getter is added; the scan reads the existing pub(crate) field directly, so the public surface is unchanged.

Registry wiring: MetadataTableType::Refs (+ as_str / TryFrom), the MetadataTable::refs() accessor, and the DataFusion TableProvider schema/scan arms. The $refs DataFusion table name is enumerated automatically via MetadataTableType::all_types() (strum EnumIter).

Are these changes tested?

Yes.

  • Two unit goldens in inspect::refs::tests:
    • test_refs_table uses the shared TableTestFixture (a main branch auto-injected by V2 normalization with all-null retention, plus a test tag). It pins row order, the BRANCH/TAG type strings, and the tag's max_reference_age_in_ms.
    • test_refs_table_retention_fields uses inline metadata (TempDir-based, deterministic) whose main branch carries explicit retention — the load-bearing golden for the branch column mapping (min_snapshots_to_keep=5, max_snapshot_age_in_ms=86400000, max_reference_age_in_ms=259200000), which the all-null fixture branch does not exercise.
  • DataFusion / sqllogictest goldens: test_provider_list_table_names now lists my_table$refs; show_tables.slt lists the $refs tables; insert_into.slt selects $refs on a snapshot-less table and expects zero rows (V2 normalization only injects main once a current snapshot exists).

Red→green verified locally: flipping the branch type string "BRANCH""branch" fails both unit goldens on the type column; restoring passes. cargo clippy --all-targets --all-features --workspace -- -D warnings clean.

Are there any user-facing changes?

New feature. SELECT * FROM <table>$refs (via DataFusion) and table.inspect().refs() now expose a table's snapshot references.

@gmhelmold gmhelmold marked this pull request as draft July 12, 2026 16:42
@gmhelmold gmhelmold force-pushed the feat/refs-metadata-table branch from cf93440 to 0cf4c2a Compare July 12, 2026 16:43
@gmhelmold gmhelmold marked this pull request as ready for review July 13, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant