EXPORT PARTITION to Iceberg never commits when the exported parts are gone from the committing replica

[Selfeer]

Description

ALTER TABLE ... EXPORT PARTITION ID ... TO TABLE <iceberg_dest> runs in two stages:
the data files are uploaded to object storage, and then a commit makes them visible in
the Iceberg table.

If the source parts that were exported disappear from the replica between those two
stages, the commit can never finish. The data files are already in the bucket, but the
export stays in PENDING, retrying over and over, until
export_merge_tree_partition_task_timeout_seconds (default 86400, 1 day) runs out.
It then ends as KILLED. The uploaded files stay in the bucket forever, referenced by
no Iceberg snapshot — the work is lost and has to be redone manually.

Source parts disappearing is completely normal operation: a merge replaces them, and the
old ones are cleaned up after old_parts_lifetime (default 480s). A server restart in
that window has the same effect. So any Iceberg export that is slow enough to still be
committing when a merge and cleanup pass through — or that is interrupted by a restart —
can get stuck like this.

This issue:

• Affects Iceberg destinations only — plain object-storage (hive) exports commit fine.
• Is worst on single-replica setups, where no other replica can be asked for the
  missing parts.
• Contradicts the restart tolerance the feature is supposed to have — an export
  interrupted by a restart is expected to resume, not to hang for a day and die.
• Was introduced by PR #2074.

---

How to reproduce the behavior

Setup

-- Single-replica ReplicatedMergeTree source with aggressive old-part cleanup,
-- so the merged-away parts are removed quickly instead of after 8 minutes.
CREATE TABLE source (id Int64, event_date Date)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/source', 'r1')
ORDER BY id
PARTITION BY toYYYYMM(event_date)
SETTINGS old_parts_lifetime = 1;

INSERT INTO source VALUES (1, '2024-03-20'), (2, '2024-03-20');

CREATE TABLE dest (id Int64, event_date Date)
ENGINE = IcebergS3('http://minio:9000/warehouse/dest/', 'minio_user', 'minio_password')
PARTITION BY toRelativeDayNum(event_date)
SETTINGS allow_insert_into_iceberg = 1;

Steps

1. Hold the export at the commit stage, so there is a reliable window between "files
   uploaded" and "commit done". (In real life this window is simply however long the
   commit takes, or a restart landing inside it.)

SYSTEM ENABLE FAILPOINT export_partition_commit_always_throw;

2. Start the export. The data files get uploaded; the commit keeps retrying. Short
   backoff and timeout values just make the outcome observable in minutes instead of a
   day.

ALTER TABLE source EXPORT PARTITION ID '202403' TO TABLE dest
SETTINGS export_merge_tree_partition_retry_initial_backoff_seconds = 1,
         export_merge_tree_partition_retry_max_backoff_seconds     = 3,
         export_merge_tree_partition_task_timeout_seconds          = 120;

3. Do ordinary things to the source table: insert more data and let a merge replace the
   exported parts. Then wait for cleanup to remove them.

INSERT INTO source VALUES (3, '2024-03-05');
OPTIMIZE TABLE source PARTITION ID '202403' FINAL;

-- Wait until the exported parts are gone entirely (not just inactive).
-- Only the merged part should be listed here:
SELECT name, active FROM system.parts
WHERE table = 'source' AND partition_id = '202403';

4. Let the commit proceed.

SYSTEM DISABLE FAILPOINT export_partition_commit_always_throw;

5. Watch the export.

SELECT status, exception_count, last_exception_per_replica
FROM system.replicated_partition_exports
WHERE source_table = 'source' AND partition_id = '202403';

Variant — restart instead of cleanup

Same setup, but instead of steps 3–4, restart the server after the data files have been
uploaded and before the commit lands. If the exported parts were merged away before the
restart, they are not there afterwards, and the export that resumes after startup gets
stuck the same way.

---

Expected behavior

The export reaches COMPLETED and the rows are queryable from the Iceberg table:

SELECT count() FROM dest;
-- 2

Everything the commit needs is known before the upload even starts — the export was
already validated and accepted at that point — so a merge on the source table afterwards
should not be able to block it.

---

Actual behavior

The export never leaves PENDING. exception_count climbs on every retry, and after
export_merge_tree_partition_task_timeout_seconds the status becomes KILLED:

status: PENDING          -- exception_count climbing
...
status: KILLED           -- after the task timeout

Every retry fails with the same error:

Code: 232. DB::Exception: No part <part_name> in committed state.
(NO_SUCH_DATA_PART)

The Iceberg table stays empty:

SELECT count() FROM dest;
-- 0

while the exported Parquet files are sitting in the bucket, unreferenced.

[arthurpassos]

Did you manage to reproduce it? This looks like AI gibberish

[Selfeer]

I'm investigating this better - seemed like a real issue at the first glance so I wanted to save it as a ticket.