CAS relink confirm returns `Unknown` after `ATTACH PARTITION FROM`; replicas never receive attached parts

[alsugiliazova]

Describe the situation

After ALTER TABLE … ATTACH PARTITION … FROM on a shared CAS pool, ReplicatedMergeTree replicas fetch the new parts by metadata-only relink. The source answers the confirm handshake with Unknown (not No: the part is still there). The receiver treats every non-Yes as “source unproven”, throws NETWORK_ERROR (Code 210), and is forbidden to fall back to a byte fetch. The queue retries the same handshake. Under attach load the ref ledger stays cold, recovering, or flushing, so confirm stays Unknown past the 300s replica-equality timeout. Destination replicas permanently miss rows.

Discovered in CI: alter_attach_partition_cas part 1 on 26.6.2.20001.altinityantalya (GitHub Actions run 33560831028). No crash. Wrong / incomplete replica data.

This issue:

• Causes silent replica divergence (missing rows after attach), not a crash
• Is CAS-only (fetch-by-relink + zero-I/O confirm). Local-disk attach_partition does not take this path
• Is load-sensitive: ~21k relinks succeeded in the same job; ~170k Code 210 abandons; 31541/31541 confirms on clickhouse1 were unproven (unknown), never (no)

---

How to reproduce the behavior

Environment

• Version: 26.6.2.20001.altinityantalya
• Build type: release (Altinity Antalya)
• Arch: x86_64
• Flags: --use-keeper --with-analyzer
• Storage: --cas (storage_policy = 'cas_policy', shared pool, per-replica namespace alter-cas-{replica})

CI combinatorics ( reliably hits it )

From altinity/clickhouse-regression/alter:
The suite runs many tests concurrently against the same CAS pool. Running only the failed subset does not reproduce the failure, probably because the load is not high enough.

./regression.py --local --with-analyzer --clickhouse-binary-path docker://altinity/clickhouse-server:26.6.2.20001.altinityantalya --cas --only '/alter/attach partition/part 1/*'

---

Expected behavior

1. After ATTACH PARTITION FROM on a same-pool CAS ReplicatedMergeTree, every replica has the same rows as the source partition.
2. If relink confirm cannot be evaluated (Unknown — ledger cold, recovering, or leader_active / pending), the receiver falls back to a byte fetch from a source that still holds the part (gate 0 already found it Active/Outdated).
3. Unknown must not be treated as “the source is in doubt” the way a failed/absent proof is. Unknown means this mount cannot answer without I/O.

---

Actual behavior

On release builds (26.6.2.20001.altinityantalya)

Replica data diverges. First leaf assertion (alter/table/attach_partition/simple_attach_partition_from.py:142-145):

assert (
    destination_data_1.output == destination_data_2.output
    and destination_data_1.output == destination_data_3.output
), error()

clickhouse3 was missing four rows that clickhouse1 had:

-12	18	22	1003		1970-01-01 01:00:00	0	0	1
-12	19	23	1003		1970-01-01 01:00:00	0	0	1
-13	18	22	1004		1970-01-01 01:00:00	0	0	1
-13	19	23	1004		1970-01-01 01:00:00	0	0	1

Partition key (intDiv(a,2), intDiv(b,2)) → (6, 9). Those are exactly the parts that failed relink on clickhouse3 (6-9_1_1_0, 6-9_2_2_0, …). The test retried 28 times over 300s, then AssertionError.

Receiver error (clickhouse3), table destination_e3b71c02_a64b_11f1_acc8_92000910a29a:

2026.09.01 23:28:20.817504 [ 648 ] {} <Error> default.destination_e3b71c02_… :
  processQueueEntry: Code: 210. DB::Exception: Source clickhouse1 did not prove it
  still holds the manifest it offered for part 5-8_3_3_0 by relink; the relink is
  abandoned and the fetch will be retried later. (NETWORK_ERROR)

Stack (trimmed; same shape on every 210):

4. DB::DataPartsExchange::Fetcher::relinkPartToDisk(...)
5. DB::DataPartsExchange::Fetcher::fetchSelectedPart(...)
6. DB::StorageReplicatedMergeTree::executeReplaceRange(...)   # ATTACH FROM is REPLACE_RANGE on the queue
7. DB::StorageReplicatedMergeTree::executeLogEntry(...)
8. … processQueueEntry …
9. DB::ReplicatedMergeTreeQueue::processEntry(...)

Source debug (clickhouse1) — every confirm in this job:

Relink confirm is unproven (unknown) for ref '8_1_1_0' (part 8_1_1_0, manifest 1:332163:1)
  in namespace 'alter-cas-clickhouse1/store/451/4511104e-…@cas@'

Zero unproven (no). Zero source token … is not one this server minted. Gate 0 found the part; gate 1 would not say Yes.

Counts from the failing job

Signal	clickhouse1	clickhouse2	clickhouse3
Code 210 “did not prove … by relink”	45 294	67 280	58 355
Relink confirm is unproven (unknown)	31 541	6 135	(not counted)
Relink confirm is unproven (no)	0
Relink success (finished (no bytes transferred))	9 201	6 344	5 494
S3 ref_catalog errors	468	126	117
S3_ERROR	304	74	70
<Fatal> / sanitizer / cores	0	0	0

Job totals (TestFlows): 78 Fail + 4 Error scenarios, 3068 OK, 4h 44m. No server death.

Secondary (same job, different bug)

One ATTACH PARTITION hit:

Code: 49. DB::Exception: Chosen multipart upload for an empty file. This must not happen. (LOGICAL_ERROR)
  in query: ALTER TABLE destination_bcd4d357_… ATTACH PARTITION 1 FROM source_bcd4d58d_…

Source: src/IO/S3/copyS3File.cpp (calculatePartSize: if (!total_size) throw LOGICAL_ERROR). Track separately if it reproduces; it is not the replica-divergence mechanism.

A few Error leaves are S3 timeouts on CREATE TABLE … SETTINGS storage_policy = 'cas_policy' (Code: 499 / Poco::Exception … Timeout, key data/cas/ref_catalog). That keeps recovery_in_progress true and feeds more Unknown confirms. part level/reset when equal to legacy max level Error'd this way (already xfailed as Fail for >24.8 / ClickHouse#69001, so the xfail did not apply).

---

Root cause analysis

Two design choices collide after ATTACH PARTITION FROM.

Path

1. ATTACH PARTITION FROM on the initiator clones parts onto the destination via MergeTreeData::cloneAndLoadDataPart → DataPartStorageOnDiskBase::freeze / Backup. On a CA disk the clone is one disk transaction and publishes new refs at commit (DataPartStorageOnDiskBase.cpp ~542–606). That is appendRefOps traffic (pending / leader_active).
2. The destination is ReplicatedMergeTree, so other replicas see a REPLACE_RANGE (stack frame 6) and fetch the parts.
3. Same-pool CAS fetch is relink, not bytes (DataPartsExchange.cpp ~391–419): sender calls getRelinkOffer, sends manifest + cas_source_token cookie, no file bodies.
4. Receiver Fetcher::relinkPartToDisk: stages a +1 (prepareAdoptFromManifest), then T2 CONFIRM — HTTP POST with cas_confirm=<token> to the sender. Only cookie yes authorizes promote.
5. Sender Service::resolveContentAddressedConfirm:
   • Gate 0: part must be Active or Outdated on the matched disk. Missing → No.
   • Gate 1: confirmExactRef (ledger). Only source of Yes.
6. CasRefLedger::confirmExactRef is zero object-store I/O by contract (interserver amplification). It returns Unknown when:
   • the ref table is not in cache / evicted (ref_name_slots miss)
   • state_mutex cannot be taken without blocking (try_to_lock fails — recovery in progress)
   • !recovered / recovery_in_progress / catalog invalidated / remounted
   • lane not quiescent: lane_state != Ready or !pending.empty() or leader_active
7. No and Unknown are one wire outcome (unproven). ContentAddressedExchange.h: “ONLY Yes AUTHORIZES ANYTHING.”
8. Taxonomy row 3 (DataPartsExchange.cpp ~1314–1321): unproven confirm THROWS NETWORK_ERROR and must not return nullptr, because a byte re-request to “a source whose state is in doubt” is considered unsound.
9. Caller (fetchSelectedPart ~939–950): nullptr → fall_back_to_byte_fetch(); throw → fallback does not run. Queue stores the exception and retries the same relink handshake.
10. Under attach churn the ledger is often not quiescent (step 1) and/or recovering after S3 ref_catalog timeouts. Retry does not change that within 300s. Parts never arrive.

The Jepsen write-up (cas/docs/CAS-JEPSEN-REPORT-20260811.md) saw the same Code 210 while killing nodes and called it correct fail-closed self-heal. That does not apply here: all three nodes stayed up; confirm was Unknown (cannot evaluate), not No (part gone).

Why Unknown is the wrong class for “do not byte-fetch”

The comment that forbids byte-fetch assumes the source is in doubt. Gate 0 already proved the part object exists on this disk. Unknown from gate 1 is “our in-memory ledger snapshot is incomplete or mid-mutation”, not “the source dropped the part”. Collapsing them on the wire makes a sound recovery (stream bytes from a live part) illegal.

RQ.SRS-048.CAS.Relink.CrossPool.ByteFallback already requires a byte fetch when confirmation is unavailable. The code does that for an empty token (old peer, taxonomy row 1 → return nullptr) and for MechanismFallbackAllowed. It does not do it for Unknown.

---

Suggested fix (for the implementer)

Pick one; (A) is the smallest behavior change that matches the requirement.

A. Treat ledger Unknown as mechanism fallback (recommended)

In Fetcher::relinkPartToDisk, after confirm:

• Cookie yes → promote (unchanged)
• Cookie unproven and the confirm HTTP call succeeded (source answered) → today this is row 3 / throw
• Split the source-side answer so the receiver can tell No from Unknown, or (smaller wire change) treat a successful HTTP confirm that is not yes as fallback only when gate 0 would have been Yes — but the receiver does not see gate 0.

Cleaner: put Unknown vs No back on the wire (third cookie, or a header). Then:

Sender answer	Receiver action
Yes	promote
No	throw retry-later (source no longer holds that binding; do not byte-fetch from this sender)
Unknown	abort() the prepared +1, return nullptr, caller byte-fetches from the same sender (taxonomy row 2/5)

Unknown + byte-fetch is sound because gate 0 found Active/Outdated. The sender can still stream the part.

Update ContentAddressedExchange.h comments that say No and Unknown are one outcome. That coupling exists only because rule 6 (mount fence) is evaluated last and a fenced mount can still answer No. A receiver that byte-fetches on Unknown does not treat No as “part is gone”; it only treats Unknown as “ask for bytes”.

B. Keep the binary wire, but fall back on unproven when the source is reachable

If a third cookie is too much: a completed confirm HTTP response with cookie unproven → return nullptr (byte-fetch). A failed confirm (timeout, no cookie) stays row 3 / throw.

Risk: a true No (part merged away / ref repointed) would byte-fetch from a sender that no longer has that part name. fetchSelectedPart already re-resolves covering parts on retry, so this may still converge — verify against merge-during-fetch.

C. Make confirmExactRef able to answer Yes under attach load

Do not add object-store I/O on the confirm path (the zero-I/O contract is load-bearing). Options that stay in-memory:

• Pin the dest table’s ref runtime for the duration of ATTACH / REPLACE_RANGE so it cannot be evicted mid-handshake
• Do not return Unknown solely because leader_active / pending if the exact row is already in committed and the pending ops are appends of other refs (narrower rule 3). This needs a TLA / spec check — rule 3 is table-scoped on purpose
• After attach commit, wait until lane_state == Ready && pending.empty() before writing the replication log entry (initiator-side). Replicas then confirm against a quiet ledger

(C) alone is not enough if S3 ref_catalog timeouts leave recovery_in_progress. Combine with (A).

Do not

• Xfail /alter/attach partition/… in clickhouse-regression
• Demote Code 210 to Warning and call it fixed (Jepsen rec). Severity is a follow-up; missing rows are the bug
• Only add retries / longer test timeout

Tests that must go green after the fix

1. The first leaf of this job (Option 1).
2. cas/tests/replicated.py replicated_attach_partition_from (already exists; keep it).
3. A new test that forces confirm Unknown then asserts the follower still gets the rows:
   • failpoint cas_relink_receiver_force_mechanism_failure already forces row 2/5 (return nullptr / byte-fetch). Add a failpoint or hook that makes confirmExactRef return Unknown (or skip the in-memory slot) on the sender while the part stays Active.
   • Assert: follower converges; ProfileEvents show a byte fetch (or at least no permanent Code 210 loop).
4. Regression: alter_attach_partition_cas part 1 on a 3-node keeper cluster.

Existing gtests: src/Disks/tests/gtest_cas_confirm_exact_ref.cpp documents that Unknown is the only legal failure direction for some rules. Extend it if you change rule 3.

---

Source map (clone /Volumes/workspace/ClickHouse @ 684161dcc03, 2026-08-05)

The 26.6.2 package is newer than this clone; the protocol and the log strings match the binary. Confirm line numbers on antalya-26.6 / the 26.6.2 tag before editing.

What	File	Function / symbol
Relink offer instead of bytes	src/Storages/MergeTree/DataPartsExchange.cpp ~391–419	Service::processQuery
Mint token	src/Disks/…/ContentAddressed/ContentAddressedMetadataStorage.cpp ~2075–2119	getRelinkOffer
Confirm routing + gate 0	DataPartsExchange.cpp ~212–268	resolveContentAddressedConfirm
Confirm handler / unproven cookie	DataPartsExchange.cpp ~271–299	answerContentAddressedConfirm
Zero-I/O Unknown rules	src/Disks/…/Pool/CasRefLedger.cpp ~400–500	confirmExactRef
Disk lifecycle → Unknown	ContentAddressedMetadataStorage.cpp ~2026–2061	confirmExactRef
No ≡ Unknown on the seam	src/Disks/…/ContentAddressedExchange.h ~15–32	CasConfirmAnswer
Taxonomy row 3 (throw, no byte-fetch)	DataPartsExchange.cpp ~1314–1321, ~1543–1552	relinkPartToDisk
nullptr vs throw at caller	DataPartsExchange.cpp ~939–950	fetchSelectedPart
Empty-token does byte-fetch	DataPartsExchange.cpp ~1411–1421	relinkPartToDisk row 1
CA clone = one txn / new refs	src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp ~542–606	freeze / clone
Attach/replace clone	src/Storages/MergeTree/MergeTreeData.cpp ~9656+	cloneAndLoadDataPart (must_on_same_disk=false for attach)
Queue entry for attach-from	src/Storages/StorageReplicatedMergeTree.cpp	executeReplaceRange (see stack)
Empty multipart LOGICAL_ERROR	src/IO/S3/copyS3File.cpp ~346–349	calculatePartSize
Confirm Unknown gtest	src/Disks/tests/gtest_cas_confirm_exact_ref.cpp
Relink failpoint (mechanism fallback)	DataPartsExchange.cpp + src/Common/FailPoint.cpp	cas_relink_receiver_force_mechanism_failure

Introduced with the publish-then-confirm handshake (git log -S 'did not prove it still holds the manifest' → 834c9517f56 on this clone). Not bisected to a 26.6.2-only commit.

---

Requirements (this repo)

RQ.SRS-048.CAS.Relink.AttachPartitionFrom
  SHALL support ATTACH PARTITION FROM such that a subsequent replication queue
  fetch on a same-pool replica MAY relink … while preserving correct logical
  partition contents.

RQ.SRS-048.CAS.Relink.CrossPool.ByteFallback
  SHALL fall back to ordinary byte fetch when … relink confirmation is
  unavailable, and SHALL not incorrectly claim a metadata-only relink.

RQ.SRS-048.CAS.Relink.VersionMix
  SHALL serve ordinary byte fetch to a peer that cannot participate in the
  relink confirm protocol, instead of leaving that peer unable to obtain the part.

Defined in cas/requirements/requirements.py (RQ_SRS_048_CAS_Relink_*).

---

Additional context

CI failure

• Job: alter_attach_partition_cas part 1 (alter_attach_partition_cas_1)
• Workflow: 🔬 26.6.2.20001.altinityantalya | alter_cas | x86 | --use-keeper --with-analyzer
• Repo / run: Altinity/clickhouse-regression#33560831028
• Branch / SHA (regression repo): main / 1070b7f026923a7c69714cd1b84ab8fb67506fab
• Package: 26.6.2.20001.altinityantalya
• Artifacts: S3 folder
  • raw.log, fails.log.txt, nice-new-fails.log.txt, report.html
  • _instances: alter/_instances
• Local copy of this investigation: /tmp/alter1-attach-partition-cas/ (if still on the analyst machine)

Scope (do not file one issue per leaf)

All unexpected Fail leaves in this job are the same replica-equality assert after attach onto a Replicated destination:

Count	Feature
22	/alter/attach partition/part 1/check simple attach partition/…
17	partition key/attach partition from with id
16	partition key/attach partition from without id
10	rbac/check attach partition from with privileges
6	conditions/order by
6	partition key datetime/attach partition from
4	operations …/freeze partition
1+1+1	replace / update / multiple operations

Ancestor Fail/Error rows are not separate bugs.

Error leaves:

• operations …/drop partition — ExpectTimeoutError (cascade: queue stuck on 210)
• part level/reset when equal to legacy max level — S3 timeout on CREATE TABLE (related load; xfail is Fail-only)

Related tests / docs in clickhouse-regression

• cas/tests/replicated.py — replicated_attach_partition_from (R9), replicated_fetch_relink (R5)
• cas/tests/ref_collision.py — attach/tmp collision (different bug, CAS-020)
• cas/docs/CAS-JEPSEN-REPORT-20260811.md — same Code 210 under node kill (benign there)
• alter/cas_mode.py — already skips filesystem-path attach tests; these replica checks are the ones that should catch this
• alter/table/attach_partition/simple_attach_partition_from.py:129-145 — the failing assert

Related product issue

• No existing Altinity/ClickHouse issue for this confirm-Unknown stall (search 2026-09-03). Open tracking issue CAS — consolidated static analysis audit findings (tracking) #2031 is the static CAS audit, not this runtime failure.

Investigation verdict (triage)

• Category: regression
• Fix goes in: ClickHouse (CAS relink confirm / fetch fallback)
• Not: this repository’s tests, CI, or infrastructure