Spec: Add key-id and deprecate key-metadata in table and partition statistics - #17533
Spec: Add key-id and deprecate key-metadata in table and partition statistics#17533gaborkaszab wants to merge 1 commit into
Conversation
github-actions
Bot
added
the
Specification
i understand the rational of using key-id but what i don't understand is why storing key-metadata unsecure ? because we contain wrapped key as if encrypted value instead of raw ? are there more details to this .... cc @ggershinsky |
|
Hey @singhpk234 , The problem with See @ggershinsky 's and @RussellSpitzer 's comments here. |
|
Ah i see, this is unwrapped key metadata which was used to encrypt this file, and not the encrypted keymedata, then it makes sense to have have this wrapped and then have key-id from the table.metadata_keys .... we will wrap this KEK then like we do with MLK then ? |
|
@singhpk234 yes, this would work exactly the same way as manifest list encryption keys. |
| | _required_ | _required_ | _required_ | **`file-size-in-bytes`** | `long` | Size of the statistics file. | | ||
| | _required_ | _required_ | _required_ | **`file-footer-size-in-bytes`** | `long` | Total size of the statistics file's footer (not the footer payload size). See [Puffin file format](puffin-spec.md) for footer definition. | | ||
| | _optional_ | _optional_ | | ~~**`key-metadata`**~~ | | Base64-encoded implementation-specific key metadata for encryption. (**Deprecated**: use `key-id` instead) | | ||
| | _optional_ | _optional_ | _optional_ | **`key-id`** | `string` | ID of the encryption key that encrypts the statistics file key metadata | |
There was a problem hiding this comment.
Should it be the current KEK or the KEK of the snapshot MLK was encrypted with ? there is possibility that a new KEK is created before we ran an offline process to collect stats ? in that case we use KEK of the MLK for snapshot != KEK of the partition file.
I think it should be fine i think but i want to bring this up for discussion
There was a problem hiding this comment.
This is not the MLK's encryption key.
In my understanding TableMetadata.encryptionKeys contains the following types of encrypted keys:
- The table's encryption key that is used to encrypt all the other keys in the list (KEK)
- The encryption keys used for encrypting manifest lists
- The encryption keys used for encrypting snapshot files (this is newly introduced in API, Core, Spark 4.1: Add key_metadata and encrypted write support for StatisticsFile #17417)
In this case this key-id refers to a particular encrypted key for type 3)
If my understanding is incorrect, please let me know.
There was a problem hiding this comment.
nit, a typo "snapshot" -> "statistics".
3 participants
Storing key-metadata in TableStatistics and PartitionStatistics is unsecured, because that is persisted into table metadata, that itself is not encrypted. As an existing design for manifest list, we encrypt the key-metadata, and save the key-id of the encryption key that we used to encrypt key-metadata.