Metadata file not found when restart flink job from latest checkPoint with iceberg v2

[litiliu]

We have a flink job that write upsert stream into a partitioned icebergV2 table ,and the flink version is 1.13. When that job get failed, we restart it from the latest checkPoint. But we got that exception: Files does not exists.
Why does that exception occur, and what should we do to avoid that problem?

[chenwyi2]

is this solved? i have this problem, too

[pvary]

Do you have a concurrent compaction/cleanup job which removes orphan files?

[chenwyi2]

yes, we have removes orphan files,but only remove files which are older than 3 days

[pvary]

On HDFS you might have an audit log to check when the files were removed and find out what happend with them based on the timing.
The other thing could be some messed up metadata where the flink.max-committed-checkpoint-id is wrong. Out of order commits, or multiple jobs without jobId writing to the same table.

[chenwyi2]

what happened if the table metadata has been updated to latest checkpoing and jobid ,but flushing snapshot state to state backend was failed, for examplethe, in the IcebergFilesCommitter calss, the latest commiting commitOperation has done but snapshotState was failed, what wiil hadppen? this can cause filenotfound ?