Plumbing to add extra custom metadata to snapshot summary#1241
Plumbing to add extra custom metadata to snapshot summary#1241rdblue merged 5 commits intoapache:masterfrom
Conversation
| public static final String PUBLISHED_WAP_ID_PROP = "published-wap-id"; | ||
| public static final String SOURCE_SNAPSHOT_ID_PROP = "source-snapshot-id"; | ||
| public static final String REPLACE_PARTITIONS_PROP = "replace-partitions"; | ||
| public static final String EXTRA_METADATA_PREFIX = "extra-metadata."; |
There was a problem hiding this comment.
happy to change if a different prefix string would be more suitable
There was a problem hiding this comment.
Sounds good to me. What about context. or snapshot.property.? I want it to be somewhat obvious what the prefix indicates, but as short as possible.
There was a problem hiding this comment.
snapshot.property sounds good. I'll make changes.
| .option("extra-metadata.extra-key", "someValue") | ||
| .option("extra-metadata.another-key", "anotherValue") | ||
| .save(tableLocation); | ||
|
|
||
| Table table = tables.load(tableLocation); | ||
|
|
||
| Assert.assertTrue(table.currentSnapshot().summary().get("extra-key").equals("someValue")); |
There was a problem hiding this comment.
plumbing to pass extra information from write options
|
Looks good to me. I just want to find a good prefix for the feature. It would also be good to document this in the write options docs: http://iceberg.apache.org/configuration/#write-options |
|
Ack, will update docs too 👍 |
|
@rdblue Added docs. Btw, I changed the prefix slightly to use |
|
Thanks @moulimukherjee! |
3 participants
Plumbing to add extra custom metadata to snapshot summary via write options #1242
Context: At Stripe we make use of airflow to run spark jobs, and want to pass a separate external ID and some versioning information linking an airflow run for each snapshot created. Currently to do this, we have to fork the Writer just to override the commitOperation.
The changes in this PR would allow custom metadata via write options which start with a particular prefix (
extra-metadata.).cc? @rdblue