-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Manifest list encryption #7770
base: main
Are you sure you want to change the base?
Manifest list encryption #7770
Changes from all commits
5c10c8a
02d4f0b
a18dea0
d55c28f
60d4de9
7ebd06e
0ec8799
829fcab
0415900
90abab3
5486f6f
153b01f
d9f6cee
92a1942
d72060d
1cf95f7
1cd1e1e
7919e6f
26a756d
a4a504a
e2e452d
64ffd83
41e95e0
0bbe4bb
39057d4
8493804
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one | ||
* or more contributor license agreements. See the NOTICE file | ||
* distributed with this work for additional information | ||
* regarding copyright ownership. The ASF licenses this file | ||
* to you under the Apache License, Version 2.0 (the | ||
* "License"); you may not use this file except in compliance | ||
* with the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, | ||
* software distributed under the License is distributed on an | ||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
* KIND, either express or implied. See the License for the | ||
* specific language governing permissions and limitations | ||
* under the License. | ||
*/ | ||
package org.apache.iceberg; | ||
|
||
import java.nio.ByteBuffer; | ||
import org.apache.iceberg.encryption.EncryptionManager; | ||
|
||
public interface ManifestListFile { | ||
|
||
/** Location of manifest list file. */ | ||
String location(); | ||
|
||
/** Snapshot ID of the manifest list. */ | ||
long snapshotId(); | ||
|
||
/** | ||
* The manifest list key metadata is encrypted with a "key encryption key" (KEK). Returns the KEK | ||
* ID for this manifest file. | ||
*/ | ||
String keyMetadataKeyId(); | ||
|
||
/** Returns the manifest list key metadata, encrypted with its KEK. */ | ||
ByteBuffer encryptedKeyMetadata(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think that these should expose both encrypted and unencrypted versions of key metadata. If this object stores the encrypted key metadata, then it should only exposed the encrypted version. Similarly, other classes that store the unencrypted metadata should not be responsible for encrypting. These are simple classes and should stay that way rather than being responsible for encryption or carrying multiple versions of the same thing. |
||
|
||
/** Decrypt and return the encrypted key metadata */ | ||
ByteBuffer decryptKeyMetadata(EncryptionManager em); | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -162,6 +162,16 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) { | |
*/ | ||
String manifestListLocation(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @rdblue @RussellSpitzer what do you think? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There are dozens of calls to this method today (tests etc), so we'll likely need to keep it for a while. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, I saw that. We can consolidate in a separate PR. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, I agree. We need to keep this because it is part of the public API and used in many cases. |
||
|
||
/** | ||
* This snapshot's manifest list file info: size, encryption key metadata and location | ||
* | ||
* @return manifest list file info | ||
*/ | ||
default ManifestListFile manifestListFile() { | ||
throw new UnsupportedOperationException( | ||
this.getClass().getName() + " doesn't implement manifestListFile method"); | ||
} | ||
|
||
/** | ||
* Return the id of the schema used when this snapshot was created, or null if this information is | ||
* not available. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one | ||
* or more contributor license agreements. See the NOTICE file | ||
* distributed with this work for additional information | ||
* regarding copyright ownership. The ASF licenses this file | ||
* to you under the Apache License, Version 2.0 (the | ||
* "License"); you may not use this file except in compliance | ||
* with the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, | ||
* software distributed under the License is distributed on an | ||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
* KIND, either express or implied. See the License for the | ||
* specific language governing permissions and limitations | ||
* under the License. | ||
*/ | ||
package org.apache.iceberg; | ||
|
||
import java.io.Serializable; | ||
import java.nio.ByteBuffer; | ||
import org.apache.iceberg.encryption.EncryptionManager; | ||
import org.apache.iceberg.encryption.EncryptionUtil; | ||
import org.apache.iceberg.util.ByteBuffers; | ||
|
||
class BaseManifestListFile implements ManifestListFile, Serializable { | ||
private final String location; | ||
private final long snapshotId; | ||
private final String keyMetadataKeyID; | ||
// stored as a byte array to be Serializable | ||
private final byte[] encryptedKeyMetadata; | ||
|
||
BaseManifestListFile( | ||
String location, | ||
long snapshotId, | ||
String keyMetadataKeyID, | ||
ByteBuffer encryptedKeyMetadata) { | ||
this.location = location; | ||
this.snapshotId = snapshotId; | ||
this.encryptedKeyMetadata = ByteBuffers.toByteArray(encryptedKeyMetadata); | ||
this.keyMetadataKeyID = keyMetadataKeyID; | ||
} | ||
|
||
@Override | ||
public String location() { | ||
return location; | ||
} | ||
|
||
@Override | ||
public long snapshotId() { | ||
return snapshotId; | ||
} | ||
|
||
@Override | ||
public String keyMetadataKeyId() { | ||
return keyMetadataKeyID; | ||
} | ||
|
||
@Override | ||
public ByteBuffer encryptedKeyMetadata() { | ||
return ByteBuffer.wrap(encryptedKeyMetadata); | ||
} | ||
|
||
@Override | ||
public ByteBuffer decryptKeyMetadata(EncryptionManager em) { | ||
return EncryptionUtil.decryptSnapshotKeyMetadata(this, em); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happened to
length
orsize
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is now kept in the key metadata.
Also, we need it in one place only - when decrypting the manifest list file (stream).