API, Core: Introduce a generic EncryptedFile abstraction to replace ManifestListFile by gaborkaszab · Pull Request #17545 · apache/iceberg

gaborkaszab · 2026-08-07T09:20:30Z

ManifestListFile and its implementation contains nothing that is specific to manifest lists. It is more generally related to files that use TableMetadata.encryptionKeys to store encrypted encryption key metadata that are referred to by a key ID.

This PR introduces a more general interface that can be used accross multiple file types like manifest lists, V4 root manifests, table statistics and partition statistics. The less general functionality specific to manifest lists is deprecated or removed where possible.

gaborkaszab · 2026-08-07T10:15:47Z

+   * @deprecated will be removed in 2.0.0; use {@link #newInputFile(EncryptableFile)} instead.
+   */
+  @Deprecated
+  default InputFile newInputFile(ManifestListFile manifestList) {


I dropped BaseManifestListFile because that was package private. Now, there is no implementation in the library that can call this function, still I don't think we can drop this, because it'd break API for users that happen to implement their own ManifestListFile. Not likely, but technically feasible.

stevenzwu · 2026-08-07T15:43:40Z

+ * A file that may be encrypted. If it is encrypted, its encrypted key metadata is tracked in the
+ * table metadata encryption keys and is referenced by a key ID.
+ */
+public interface EncryptableFile {


Naming: "Encryptable" describes a capability every file has, not the specific concept/state represented here. I would suggest EncryptedFile and let the Javadoc cover the "may or may not carry a keyId" case.

Changed to EncryptedFile, however, I'm still not comfortable with the vile name. The name suggests the file is encrypted, but it's may or may not. Maybe FileWithEncryptedKey ?

stevenzwu · 2026-08-07T15:43:49Z

+   * The file key metadata can be encrypted. Returns ID of encryption key or null if it's not
+   * encrypted.
+   */
+  String encryptionKeyID();


encryptionKeyID() → keyId() for consistency with Snapshot.keyId() and EncryptedKey.keyId() (ID caps is out of style — even the local variables in EncryptionUtil are manifestListKeyId).

Javadoc suggestion: /** Returns the encryption key ID for this file, or null if the file is not encrypted. */ — the interface-level javadoc already covers the mapping to TableMetadata.encryptionKeys(), so the method sentence can stay short.

Thanks for the comment suggestion! Applied.

About the function name: I'm not sure we can change the function name, because that would break the API for ManifestListFile that is derived from this interface.
Alternatively, we can avoid deriving ManifestListFile from the new interface, but then we won't be able to cast it to the new interface and delegate to the new function signature like in FileIO or in EncryptionUtil:
newInputFile((EncryptableFile) manifestList)

stevenzwu · 2026-08-07T17:19:04Z

+/**
+ * A manifest list file that may be encrypted.
+ *
+ * @deprecated will be removed in 2.0.0; use {@link EncryptableFile} instead.


nit: do we need to provide since? @deprecated since 1.12.0. Will be removed in 2.0.0?

I added for now, but I personally don't feel it's useful to the reader.

stevenzwu · 2026-08-07T17:52:48Z

        "Snapshot key metadata encryption requires a StandardEncryptionManager");
    StandardEncryptionManager sem = (StandardEncryptionManager) em;
-    String manifestListKeyId = manifestList.encryptionKeyID();
+    String fileKeyId = file.encryptionKeyID();


nit: encryptionKeyId might be more clear. similarly, maybe also fileKey -> encryptionKey below.

Renamed them

…ManifestListFile ManifestListFile and its implementation contains nothing that is specific to manifest lists. It is more generally related to files that use TableMetadata.encryptionKeys to store encrypted encryption key metadata that are referred to by a key ID. This PR introduces a more general interface that can be used accross multiple file types like manifest lists, V4 root manifests, table statistics and partition statistics. The less general functionality specific to manifest lists is deprecated or removed where possible.

gaborkaszab

Thanks for taking a look, @stevenzwu !

gaborkaszab · 2026-08-08T10:28:03Z

+ * A file that may be encrypted. If it is encrypted, its encrypted key metadata is tracked in the
+ * table metadata encryption keys and is referenced by a key ID.
+ */
+public interface EncryptableFile {


Changed to EncryptedFile, however, I'm still not comfortable with the vile name. The name suggests the file is encrypted, but it's may or may not. Maybe FileWithEncryptedKey ?

gaborkaszab · 2026-08-08T10:29:37Z

+   * The file key metadata can be encrypted. Returns ID of encryption key or null if it's not
+   * encrypted.
+   */
+  String encryptionKeyID();


Thanks for the comment suggestion! Applied.

About the function name: I'm not sure we can change the function name, because that would break the API for ManifestListFile that is derived from this interface.
Alternatively, we can avoid deriving ManifestListFile from the new interface, but then we won't be able to cast it to the new interface and delegate to the new function signature like in FileIO or in EncryptionUtil:
newInputFile((EncryptableFile) manifestList)

gaborkaszab · 2026-08-08T10:40:28Z

+/**
+ * A manifest list file that may be encrypted.
+ *
+ * @deprecated will be removed in 2.0.0; use {@link EncryptableFile} instead.


I added for now, but I personally don't feel it's useful to the reader.

gaborkaszab · 2026-08-08T10:45:57Z

        "Snapshot key metadata encryption requires a StandardEncryptionManager");
    StandardEncryptionManager sem = (StandardEncryptionManager) em;
-    String manifestListKeyId = manifestList.encryptionKeyID();
+    String fileKeyId = file.encryptionKeyID();


Renamed them

github-actions Bot added API core labels Aug 7, 2026

This was referenced Aug 7, 2026

API, Core, Spark, Flink: extract SnapshotFile abstraction #17523

Open

API, Core, Spark 4.1: Add key_metadata and encrypted write support for StatisticsFile #17417

Open

gaborkaszab commented Aug 7, 2026

View reviewed changes

stevenzwu added this to V4: metadata tree Aug 7, 2026

stevenzwu moved this to In review in V4: metadata tree Aug 7, 2026

stevenzwu reviewed Aug 7, 2026

View reviewed changes

stevenzwu changed the title ~~API, Core: Refactor: Introduce a more general abstraction to replace ManifestListFile~~ API, Core: Introduce a generic EncryptedFile abstraction to replace ManifestListFile Aug 7, 2026

gaborkaszab force-pushed the main_encryptable_file branch from 0a3b903 to c08b3fd Compare August 8, 2026 10:50

gaborkaszab commented Aug 8, 2026

View reviewed changes

gaborkaszab requested a review from stevenzwu August 8, 2026 10:50

Uh oh!

Conversation

gaborkaszab commented Aug 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwu Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stevenzwu Aug 7, 2026 •

edited

Loading