Spark: fix rewrite_position_delete_files reports an error when the partition column contains "." #8111
Conversation
…n column contains "."
lintingbin
changed the title
|
Thanks very much for the pr and the fix, can you add a test ? I have some test in https://github.com/apache/iceberg/blob/master/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewritePositionDeleteFiles.java to catch the removeDanglingDelete scenarios , I think it would fit here well. Thanks in advance |
…", change partition_col to `partition_col.type`.
|
@szehon-ho In order to test the case where the partition column contains ".", I changed the partition_col in the https://github.com/apache/iceberg/blob/master/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewritePositionDeleteFiles.java to partition_col.type. |
...nsions/src/test/java/org/apache/iceberg/spark/extensions/TestRewritePositionDeleteFiles.java
Outdated
Show resolved
Hide resolved
| @@ -122,8 +123,8 @@ public void testDatePartition() throws Exception { | |||
| @Test | |||
| public void testBooleanPartition() throws Exception { | |||
| createTable("boolean"); | |||
| insertData(i -> i % 2 == 0, 2); | |||
| testDanglingDelete(2); | |||
| insertData(PARTITION_COL, i -> i % 2 == 0, 2); | |||
There was a problem hiding this comment.
Thanks, can also you make these methods have a default version that delegates with PARTITION_COL?:
private void insertData(Function<Integer, ?> partitionValueFunction) throws Exception {
insertData(partitionValueFunction, PARTITION_COL, NUM_DATA_FILES);
}
There was a problem hiding this comment.
Done. testDanglingDelete also made the same optimization.
|
Merged, thanks @lintingbin2009 ! |
Solve the "AnalysisException: No such struct field" mentioned in #8109.