Skip empty files while migrating tables to iceberg

[naanagon]

Feature Request / Improvement

ORC:

java.lang.IllegalArgumentException: ORC schema does not contain Iceberg IDs
  at org.apache.iceberg.orc.ORCSchemaUtil.convert(ORCSchemaUtil.java:293)
  at org.apache.iceberg.orc.OrcMetrics.buildOrcMetrics(OrcMetrics.java:143)
  at org.apache.iceberg.orc.OrcMetrics.fromInputFile(OrcMetrics.java:93)
  at org.apache.iceberg.orc.OrcMetrics.fromInputFile(OrcMetrics.java:87)
  at org.apache.iceberg.data.TableMigrationUtil.getOrcMetrics(TableMigrationUtil.java:296)
  at org.apache.iceberg.data.TableMigrationUtil.lambda$listPartition$4(TableMigrationUtil.java:244)

Parquet:

java.lang.RuntimeException: mig_parquet_sample/date_key=2026-09-02/1_$folder$ is not a Parquet file (length is too low: 0)
	at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:599)
	at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:578)
	at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:971)
	at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:961)
	at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:718)
	at org.apache.iceberg.parquet.ParquetUtil.fileMetrics(ParquetUtil.java:64)
	at org.apache.iceberg.data.TableMigrationUtil.getParquetMetrics(TableMigrationUtil.java:287)
	at org.apache.iceberg.data.TableMigrationUtil.lambda$listPartition$3(TableMigrationUtil.java:236)

Sometimes empty files are created in hive, we need to manually remove all empty files before migrating to iceberg.

We can fix this by skipping empty files

Query engine

Spark

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[RussellSpitzer]

Can you explain how Hive is doing this? I'm worried because an empty parquet file is not a parquet file (it has no footer or magic bytes) so this seem like something went wrong