Flink 1.20: DynamicIcebergSink Meets NoClassDefFoundError error when it comes to checkpoint #14118
Description
Apache Iceberg version
1.10.0 (latest release)
Query engine
Flink
Please describe the bug 🐞
java.lang.NoClassDefFoundError: org/apache/hadoop/util/Setsappears when it comes to checkpoint. I write data to s3, below is some config. Also this seems a Class ofhadoop-common, buthadoop-commondoes not have this class.
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
Map<String, String> properties = new HashMap<>();
properties.put("type", "iceberg");
properties.put("clients", "5");
properties.put("property-version", "1");
properties.put("catalog-type", "hive");
properties.put("warehouse", ICEBERG_WAREHOUSE);
properties.put("uri", HIVE_METASTORE);
CatalogLoader catalogLoader = CatalogLoader.hive("test_dynamicsink", hadoopConf, properties);
- The doc shows we can use code below, but it seems that there is no non-parameter constructor provided to do this
DynamicRecord record = new DynamicRecord();
// Set table name based on business logic
TableIdentifier tableIdentifier = TableIdentifier.of(database, tableName);
record.setTableIdentifier(tableIdentifier);
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time