Skip to content

feat(dir): add metadata update APIs#98

Closed
jackye1995 wants to merge 1 commit into
jack/sharded-dir-catalogfrom
jack/dir-metadata-api
Closed

feat(dir): add metadata update APIs#98
jackye1995 wants to merge 1 commit into
jack/sharded-dir-catalogfrom
jack/dir-metadata-api

Conversation

@jackye1995
Copy link
Copy Markdown
Owner

Stacked on lance-format#7008.

Adds native DirectoryNamespace APIs to update table and namespace metadata stored in the __manifest JSON metadata column. Metadata remains JSON, but the API treats it as a string-to-string map, with description as the UI-facing long-description key.

The sharded manifest path updates only the target shard fragment for the changed table or namespace row.

S3 benchmark, c7i.48xlarge, 1000 shards, 1 writer, 100 measured write-update-table-metadata ops per point. Initial rows were bootstrapped with one 50-100 word description and 10 key-value tags; measured writes update the same metadata shape.

Initial rows TPS P50 ms P99 ms Errors
100 6.049 151.2 295.9 0
1,000 6.185 149.4 320.6 0
500 5.907 161.8 347.3 0
10,000 5.009 174.8 470.5 0
50,000 5.718 160.7 330.8 0
100,000 3.551 279.8 385.9 0
200,000 2.927 347.7 536.9 0
300,000 2.863 323.7 601.2 0
500,000 2.875 338.2 507.5 0
750,000 2.300 418.2 725.4 0
1,000,000 2.462 381.7 735.4 0

Raw artifacts are recorded locally under ~/ai/analysis/lance/jack-dir-metadata-api/update-table-namespace-metadata-api/bench_dir_metadata_s3_20260602T042414Z.

@github-actions github-actions Bot added the enhancement New feature or request label Jun 2, 2026
@jackye1995
Copy link
Copy Markdown
Owner Author

Superseded by lance-format#7050.

@jackye1995 jackye1995 closed this Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant