[tidbx] query report error when injection tiflash wn network partition

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

deploy with tiflash cn (2 replicas) + wn (2 replicas)
1, run ch workload with 2000 warehouse
go-tpc ch run -D tpcc --host tc-tidb.xxx -P4000 --warehouses 2000 -T 32 --acThreads 1 --queries q3 --ignore-error '2013,1213,1205,8022,8028,9004,9007,1062,1105' --time 36000m --user root --interval '10s'
2，injection tiflash wn network partition
wn network fault：2026-07-20 17:58:26 ~ 2026-07-20 18:08:27

2. What did you expect to see? (Required)

ap query can succeed

3. What did you see instead (Required)

query report error when injection fault and after recover fault
[2026/07/20 17:58:58.558 +08:00] [INFO] [collection.go:303] ["[2026-07-20 09:58:58] execute run failed, err execute query q14 failed Error 1105 (HY000): other error for mpp stream: Code: 159, e.displayText() = DB::Exception: Fetch pages timeout for s1027_ks1_t2737_3582_1_114210338273186161_656025, message=Deadline Exceeded, e.what() = DB::Exception,\r\n"]

[2026/07/20 18:09:31.300 +08:00] [INFO] [collection.go:303] ["[2026-07-20 10:09:31] execute run failed, err execute query q14 failed Error 1105 (HY000): other error for mpp stream: Code: 159, e.displayText() = DB::Exception: EstablishDisaggTask timeout exceeded, wn_address=tc-tiflash-0.tc-tiflash-peer.ha-test-tiflash-tps-8175260-1-198.svc.cluster.local:3930, timeout=60s, MPP<gather_id:1, query_ts:1784542111278958720, local_query_id:598, server_id:227, start_ts:467807007212306549,task_id:2>, e.what() = DB::Exception,\r\n"]

4. What is your TiFlash version? (Required)

/tiflash/tiflash version
TiFlash
Release Version: v9.0.0-beta.2.pre-200-g3d986e3de
Edition: Enterprise
Git Commit Hash: 3d986e3
Git Branch: HEAD
UTC Build Time: 2026-07-16 11:26:29
Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto next-gen hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake
Profile: RELWITHDEBINFO
Compiler: clang++ 17.0.6

Raft Proxy
Git Commit Hash: 45187f537fd50287eb46d067960d00d7adbdb11e
Git Commit Branch: HEAD
UTC Build Time: ""
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: "raftstore-proxy/external-jemalloc" portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
2026-07-20T15:22:04.473+0800

[Lily2025]

/assign yongman

[yongman]

Incident Window

• WN network isolation started at 2026-07-20 17:58:26 +08:00
• WN network isolation ended at 2026-07-20 18:08:27 +08:00

Conclusion 1: Fetch pages timeout at 2026-07-20 17:58:58.558 +08:00

Error

execute query q14 failed Error 1105 (HY000): other error for mpp stream:
Code: 159, DB::Exception: Fetch pages timeout for
s1027_ks1_t2737_3582_1_114210338273186161_656025, message=Deadline Exceeded

Analysis

• This failure happened in the FetchDisaggPages stage, not in the EstablishDisaggTask stage.
• At that point, the disaggregated read task had already been established successfully, and the compute node was already reading remote segments.
• The failing object name shows the request was fetching pages from write node store_id=1027 for keyspace=1, table_id=2737, segment_id=3582.
• FetchDisaggPages uses disagg_fetch_pages_timeout, whose default value is 30s.
• The error was reported at 2026-07-20 17:58:58.558 +08:00, about 32s after isolation started. This is consistent with a fetch-pages RPC that started shortly after 2026-07-20 17:58:26 +08:00 and then hit the 30-second gRPC deadline during the isolation window.

Conclusion

• The direct cause of this failure was that the compute node could no longer reach the target write node during FetchDisaggPages after the WN network isolation began.
• This was a network-level timeout on the page-fetch path, not evidence that the write node received the request and then spent 30 seconds processing it.

Conclusion 2: EstablishDisaggTask timeout exceeded at 2026-07-20 18:09:31.300 +08:00

Error

execute query q14 failed Error 1105 (HY000): other error for mpp stream:
Code: 159, DB::Exception: EstablishDisaggTask timeout exceeded,
wn_address=tc-tiflash-0.tc-tiflash-peer.ha-test-tiflash-tps-8175260-1-198.svc.cluster.local:3930,
timeout=60s

Analysis

• This failure happened in the EstablishDisaggTask stage, before the query could build the disaggregated read snapshot on the write node.
• The timeout was reported at 2026-07-20 18:09:31.300 +08:00.
• EstablishDisaggTask uses disagg_build_task_timeout, which was 60s in this case. That means the failing RPC started at approximately 2026-07-20 18:08:31.300 +08:00.
• WN network isolation had already been removed at 2026-07-20 18:08:27 +08:00, so the RPC started about 4s after the isolation rule was lifted.
• However, WN-side logs do not contain the expected Handling EstablishDisaggTask request entry for the failed query identifiers:
  • local_query_id=598
  • query_ts=1784542111278958720
  • start_ts=467807007212306549
• In contrast, a later query (local_query_id=601) was clearly received and processed by the write node at 2026-07-20 18:10:30.667 +08:00, proving that the WN service itself was able to accept EstablishDisaggTask requests again by then.
• Around that time, the WN was busy with snapshot apply / ingest / S3 upload work, but it was not stuck: it was still actively receiving and starting new disaggregated requests.

Conclusion

• The failed EstablishDisaggTask RPC most likely did not successfully reach the target WN service during its 60-second timeout window.
• Therefore, this failure is best explained by incomplete network-path recovery shortly after the isolation rule was removed, such as DNS / Service / Endpoint / TCP / gRPC connectivity not being fully restored yet.
• The evidence does not support the interpretation that the WN received this specific request and then spent 60 seconds processing it internally.

Overall Summary

• The first failure (Fetch pages timeout) was caused by the WN isolation interrupting an already-running query during the delta-page fetch phase.
• The second failure (EstablishDisaggTask timeout exceeded) was caused by incomplete post-isolation recovery on the CN-to-WN request path, even though the isolation rule had already been removed.
• In both cases, the primary issue was connectivity between compute node and write node, not an internal TiFlash execution hang on the WN side.

Explore-logs-2026-07-21 16_09_16.txt