Skip to content

Fix reading Encrypted Deletion Vectors #16157

@ayushtkn

Description

@ayushtkn

Apache Iceberg version

None

Query engine

None

Please describe the bug 🐞

When using Table Level Encryption, If there are Deletion Vectors the Read fails

        Caused by:
        java.lang.IllegalStateException: Null key metadata buffer
            at org.apache.iceberg.encryption.StandardKeyMetadata.castOrParse(StandardKeyMetadata.java:111)
            at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.keyMetadata(StandardEncryptionManager.java:302)
            at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.decrypted(StandardEncryptionManager.java:313)
            at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.location(StandardEncryptionManager.java:333)
            at org.apache.iceberg.encryption.EncryptingFileIO.bulkDecrypt(EncryptingFileIO.java:71)
            at org.apache.iceberg.spark.source.BaseReader.inputFiles(BaseReader.java:186)
            at org.apache.iceberg.spark.source.BaseReader.getInputFile(BaseReader.java:180)
            at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:108)
            at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:43)
            at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:144)
            at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:148)
            at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:186)

OR:

Caused by: java.lang.IllegalArgumentException: Invalid bitmap data length: 4096, expected 36                                     
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:321)               
        at org.apache.iceberg.deletes.BitmapPositionDeleteIndex.readBitmapDataLength(BitmapPositionDeleteIndex.java:184)         
        at org.apache.iceberg.deletes.BitmapPositionDeleteIndex.deserialize(BitmapPositionDeleteIndex.java:148)                  
        at org.apache.iceberg.deletes.PositionDeleteIndex.deserialize(PositionDeleteIndex.java:113)                              
        at org.apache.iceberg.data.BaseDeleteLoader.readDV(BaseDeleteLoader.java:179)                                            
        at org.apache.iceberg.data.BaseDeleteLoader.loadPositionDeletes(BaseDeleteLoader.java:165)                               
        at org.apache.iceberg.data.DeleteFilter.deletedRowPositions(DeleteFilter.java:255)                                       
        at org.apache.iceberg.data.DeleteFilter.applyPosDeletes(DeleteFilter.java:266)                                           
        at org.apache.iceberg.data.DeleteFilter.filter(DeleteFilter.java:189)       

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions