Segmentation fault in RemoteQueryExecutorReadContext async fiber resume during shutdown/timeout (STID: 4348-2010)

[groeneai]

Summary

Recurring Segmentation fault (SIGSEGV, STID: 4348-2010) in the async remote-read fiber, seen only in Stress test (arm_msan). The crash is inside the boost fiber resume() invoked from a suspend_callback() in RemoteQueryExecutorReadContext::Task::run. The faulting address is not mapped to any object (fiber-stack region), which points at a lifetime/teardown race on the fiber context rather than a data bug in the query itself.

Opening this per @alexey-milovidov's request on #110073: the signature is unrelated to that PR (it predates the branch and hits master and many unrelated PRs).

Signature

Received signal Segmentation fault (11)
Address: 0x...  Access: <not available>. Address not mapped to object.

contrib/boost/boost/context/fiber_ucontext.hpp:513: boost::context::fiber::resume() &&
src/Common/Fiber.h:64: Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>::SuspendCallback::operator()()
src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:86: DB::RemoteQueryExecutorReadContext::Task::run(...)
src/Common/AsyncTaskExecutor.cpp:89: DB::AsyncTaskExecutor::Routine::operator()(...)
contrib/boost/boost/context/fiber_ucontext.hpp:313: fiber_capture_record<...>::run()
contrib/boost/boost/context/fiber_ucontext.hpp:81: fiber_entry_func<...>(void*)

Fiber.h:64 is SuspendCallback::operator() -> impl = std::move(impl).resume(); (yielding the fiber back to its sink). RemoteQueryExecutorReadContext.cpp:86 is the trailing suspend_callback(); after reading a packet body.

How to reproduce

Not reliably reproducible outside CI so far. Both crashing queries in the latest occurrence use remote(...) and finish/tear down under load:

SELECT any(total) AS total_distinct_avg
FROM (SELECT number, avgDistinct(number) OVER () AS total FROM remote('127.0.0.{1,2,3}', numbers_mt(100000)));

SELECT count(), toString(number) AS k
FROM remote('127.0.0.{1,2}', numbers(10))
WHERE number > (queryID() = initialQueryID())
GROUP BY GROUPING SETS ((k), (number + 1, k)) ORDER BY k
SETTINGS group_by_two_level_threshold=9;

Both hit the same fiber-resume() fault. The stress harness runs many concurrent queries with the RandomQueryKiller and ZK fault injection, so query cancellation / socket timeout / connection teardown are all firing concurrently with the async read fiber.

Root-cause direction (not yet confirmed)

The unmapped faulting address is the fiber's own stack region. The fiber yields via SuspendCallback (holding a reference Impl & impl to the sink fiber). A crash resuming that sink means the sink fiber context / FiberStack has been moved-from or destroyed while a suspend is still in flight. Suspect interaction between AsyncTaskExecutor::resume() / cancel() / restart() / destroyFiber() (AsyncTaskExecutor.cpp) and the in-flight suspend_callback on the read path (cancel-during-suspend or restart-during-suspend). MSan-only visibility suggests a use-of-uninitialized / freed context surfaced by MSan instrumentation rather than a hard bug that always faults.

Occurrences (CIDB, Stress test (arm_msan))

11 hits over the last month, master + 9 unrelated PRs, none of which touch this code path:

date (UTC)	ref
2026-07-11	PR #110073
2026-07-02	PR #108932
2026-06-29	PR #100173
2026-06-23	PR #108205
2026-06-18	PR #106028
2026-06-11	PRs #107160 #107161 #107074 #107109
2026-06-10	PR #99105, master

CI report

Latest: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=110073&sha=f75f6b4383cd713bb07ad345f879953ae276e50e&name_0=PR&name_1=Stress%20test%20%28arm_msan%29

Version info

• Resolved by: Bump contrib/boost to fix AArch64 MSan fiber-switch false positives (STID 4348-2010) #110152
• Merged into: 26.7.1.1235 (included in 26.7 and later)

[groeneai]

Update after pulling the full fatal.log for the latest occurrence (PR #110073, sha f75f6b4, Stress test (arm_msan), 2026-07-11), plus a wider CIDB sweep. Two corrections to the issue as filed:

1. This is NOT arm_msan-only. The exact 6-frame fiber-resume signature (fiber::resume() -> Fiber.h:64 SuspendCallback -> RemoteQueryExecutorReadContext.cpp:86) has hit multiple arch/sanitizer combos in the last 60 days, so this is a real lifetime/race bug, not an MSan aarch64 artifact:

check	PR	date
Stress test (arm_msan)	master + 10 PRs	2026-06-10 .. 2026-07-11 (11x)
Stress test (amd_tsan)	#108457	2026-06-25
Stress test (amd_msan)	#100272	2026-06-25
Stress test (arm_debug)	#106120	2026-06-23

The amd_tsan hit in particular means a plain x86 TSAN/Debug stress run can reproduce it - it is not gated on ARM hardware.

2. The fault is a resume into an unmapped high thread-stack address, at a shallow stack, not a guard-page stack overflow. In the 2026-07-11 crash:

Address: 0xff338c7127c0. Access: <not available>. Address not mapped to object.

0xff... is a thread-stack region, and the crash stack is only 6 frames deep at RemoteQueryExecutorReadContext.cpp:86 (the trailing suspend_callback() right after a packet body was read). So it is not the deep-recursion guard-page overflow the FiberStack.h comment warns about - it is boost::context::fiber::resume() jumping back into a context whose stack is gone.

Both crashing queries are ordinary remote(...) selects finishing under the RandomQueryKiller / ZK-fault stress harness:

SELECT any(total) FROM (SELECT number, avgDistinct(number) OVER () AS total
  FROM remote('127.0.0.{1,2,3}', numbers_mt(100000)));

SELECT count(), toString(number) AS k FROM remote('127.0.0.{1,2}', numbers(10))
  WHERE number > (queryID() = initialQueryID())
  GROUP BY GROUPING SETS ((k), (number + 1, k)) ORDER BY k;

Root-cause direction (refined, still unconfirmed). The async read fiber is driven from RemoteSource::tryGenerate() -> RemoteQueryExecutor::readAsync() -> read_context->resume(). The boost fiber (BOOST_USE_UCONTEXT under sanitizers/debug) yields back to its sink = the OS thread that resumed it. The PipelineExecutor can run successive tryGenerate() calls for the same processor on different worker threads, and a resume-into-unmapped-thread-stack fault is consistent with the fiber being resumed after the thread whose context it last yielded to has exited (stack unmapped). Next step is an x86 TSAN stress repro of concurrent remote-read + cancel/timeout + worker-thread migration to confirm before any fix. No fix will be pushed without a repro.

[groeneai]

Investigation update (slot follow-up; no repro yet, no fix pushed).

Two distinct crash sites, both consistent with "RemoteQueryExecutor freed while its async read fiber is still live":

• STID 4348-2010 (arm_msan): faults in boost::context::fiber::resume() from SuspendCallback at RemoteQueryExecutorReadContext.cpp:86 (the suspend_callback() right after a packet body read). Fault address is a high thread-stack region, not mapped: resume() jumping into a sink continuation whose stack is gone.
• STID 4231-40cb (amd_tsan/amd_msan/arm_debug, x86-reachable sibling): faults in Poco::Logger::is reading executor.log (addr 0x38 = null/garbage LoggerPtr) inside RemoteQueryExecutor::sendExternalTables() (:911), reached from the FIRST sendQueryUnlocked in Task::run (:49). Reading a destroyed log member => the executor object is gone under a still-running fiber.

The x86 sibling means a plain x86 TSAN/Debug stress run can hit this class; it is not an MSan/aarch64 artifact.

Code lead to audit (unconfirmed): ~RemoteQueryExecutor (RemoteQueryExecutor.cpp:315) only cancels/destroys the read fiber when !established. established is true for the whole sendQuery window (:459 set, :492 cleared), and Task::run calls sendQueryUnlocked inside the fiber. If the executor is destroyed while the fiber is suspended in that window, the dtor takes the else branch (connections->disconnect()), leaving the fiber to be force-unwound only during member destruction of read_context (:260) - which happens AFTER log (:338) is already destroyed (reverse declaration order). That ordering matches the 4231-40cb "read destroyed log" fault.

Local repro attempts (x86 debug): the two crashing queries (avgDistinct OVER window over remote(); count()+GROUPING SETS over remote()) under a KILL-QUERY race, then under thread-fuzzer with heavy worker-thread migration and max_execution_time timeouts - server survived all rounds. Needs CI's full randomized surface to trigger; not yet deterministically reproducible.

[groeneai]

Correction to my two earlier comments. I verified each cited CIDB row by its STID and stack, and the "not arm_msan-only / x86-reachable" claim was wrong: the amd_tsan, amd_msan and arm_debug rows I listed are a DIFFERENT crash (STID 4231-40cb), which is already fixed on master.

Two separate signatures were being conflated:

• STID 4231-40cb: null LoggerPtr dereference in RemoteQueryExecutor::sendExternalTables() (Poco::Logger::is, addr 0x38), on the lazy/delayed remote source path that built the executor without setLogger(). Cross-arch (amd_tsan Fix flaky test 02497_remote_disk_fat_column (Memory limit exceeded) #108457, amd_msan Rewrite in Rust #100272, arm_debug Add ColumnsScatter::scatter: batched column split with type-index dispatch and bytes-budget input buffering #106120, arm_ubsan Match benign mutation parse errors by class in the Upgrade check #108651). This was fixed by 641aa36 ("Fix server crash on lazy remote source with unbuilt materialized CTE", merged 2026-06-25); last CI hit 2026-06-30 on a pre-fix build, none since. NOT this issue.

• STID 4348-2010 (this issue): fault in boost::context::fiber::resume() from SuspendCallback at RemoteQueryExecutorReadContext.cpp:86, resuming into a sink continuation whose thread stack is unmapped (fault addr 0xff..., a high thread-stack region). Verified Stress test (arm_msan) ONLY across all 11 hits in 90 days (master 2026-06-10 plus 10 PRs, 2026-06-10 .. 2026-07-11), zero on any other check. So it IS arm_msan-exclusive, and the x86 TSAN/Debug repro direction was based on the mixed-up 4231-40cb rows.

What this changes:

1. The member-destruction-order lead ("dtor only cancels the fiber when !established, so log is destroyed before the fiber is force-unwound") explained 4231-40cb's "read destroyed log" fault. Since that bug is fixed, that lead no longer has a crash to point at. It may still be a latent lifetime hazard, but it is not what STID 4348-2010 shows.

2. STID 4348-2010 is a resume() into an unmapped sink stack, and it only appears under arm_msan. That is consistent with either (a) a lifetime bug where the async read fiber is resumed after the driving worker thread that last ran it has exited and its stack is unmapped, exposed by MSan's fiber stack tracking on aarch64, or (b) an MSan-on-aarch64 boost fiber switch artifact (__msan_start_switch_fiber/__msan_finish_switch_fiber in contrib/boost/boost/context/fiber_ucontext.hpp). The single-arch/single-sanitizer profile over a full month leans toward the fiber-switch-under-MSan angle, but neither is confirmed.

Both crashing queries are ordinary remote(...) selects finishing under the stress harness (RandomQueryKiller + ZK faults):

SELECT any(total) FROM (SELECT number, avgDistinct(number) OVER () AS total
  FROM remote('127.0.0.{1,2,3}', numbers_mt(100000)));

SELECT count(), toString(number) AS k FROM remote('127.0.0.{1,2}', numbers(10))
  WHERE number > (queryID() = initialQueryID())
  GROUP BY GROUPING SETS ((k), (number + 1, k)) ORDER BY k;

Still no repro and no fix pushed. Next step is an arm_msan build to reproduce directly (the crash has not appeared on any x86 or non-MSan build in 90 days, so x86 repro attempts were structurally unable to hit it), and to determine whether the MSan fiber-switch instrumentation on aarch64 is reporting a real resumed-after-thread-exit lifetime bug or a false positive.

[groeneai]

Root-cause mechanism now confirmed from source (still no repro on x86; the crash is arm_msan-exclusive, so no fix pushed yet).

The (a) real-lifetime vs (b) MSan-artifact fork resolves to a concrete, verifiable mechanism: MSan's boost fiber-switch tracking assumes a fiber is resumed on the same OS thread that suspended it, but ClickHouse's work-stealing executor legitimately resumes the async remote-read fiber on a different worker thread. Verified by reading:

1. boost::context::fiber_activation_record::current() is thread_local (contrib/boost/libs/context/src/fiber.cpp:24,46), so current()->from is the context that was active on the RESUMING thread.
2. On MSan, resume() calls __msan_finish_switch_fiber((const void**)&current()->from->stack_bottom, &current()->from->stack_size) (contrib/boost/boost/context/fiber_ucontext.hpp:173), dereferencing that resuming-thread sink context's stack_bottom.
3. The async remote-read fiber can be resumed on a different thread than it suspended on: RemoteSource::tryGenerate -> RemoteQueryExecutor::readAsync -> read_context->resume(), driven by the pipeline executor. ExecutorTasks::processAsyncTasks only PREFERS the originating thread (fast_task_queue.push(node, task.thread_num) then wakes getAnyThreadWithTasks(task.thread_num), falling back to threads_queue.popAny()), and TaskQueue::pop -> getAnyThreadWithTasks steals across per-thread queues (src/Processors/Executors/TasksQueue.h). So the ready fiber node is dispatched to whichever worker is free.
4. ASan survives this because it saves/restores a per-context fake_stack across every switch (fiber_ucontext.hpp:96,156,168); MSan has no per-context equivalent (only the two __msan_*_switch_fiber calls). That asymmetry matches the arm_msan-exclusivity (0 hits on any ASan/UBSan/TSan/debug lane in 90 days).

So the fault (resume() into a high, unmapped thread-stack region at RemoteQueryExecutorReadContext.cpp:86) is MSan reading a from->stack_bottom for a sink context recorded on a thread whose stack is gone/irrelevant on the resuming thread.

Fix directions (both need an aarch64-MSan repro to demonstrate fails-without/passes-with, per no-speculative-fix; this fleet box is x86_64):

• Pin the async remote-read fiber's resume to the thread that suspended it (make the async task thread-affine so a suspended read fiber is only ever resumed on its originating worker), which removes the cross-thread from staleness; or
• Give MSan a per-context stack save/restore across fiber switches mirroring ASan's fake_stack, or otherwise correct/suppress the MSan fiber-switch tracking for the cross-thread case.

The first is a ClickHouse-side change (executor async-task affinity); the second is a boost/MSan-instrumentation change. Determining which is correct needs the aarch64-MSan repro to confirm whether any real object is used-after-free vs only MSan's shadow tracking being stale.

[groeneai]

Update + one blocking question after confirming the exclusivity is architecture-specific, not a coverage gap.

New fact: the amd_msan Stress test job DOES run in CI (ci/defs/job_configs.py, amd_msan param requiring DEB_AMD_MSAN on the AMD runner), yet STID 4348-2010 has 0 hits on amd_msan in 90 days. All 11 hits are arm_msan only. So this is genuinely aarch64-MSan-specific, not merely "no amd_msan coverage". An x86 MSan build cannot reproduce it, which is why 7 local x86 repro attempts (debug KILL loops, TSAN, TSAN+thread-fuzzer, no-KILL timeout-teardown) all could not fire it.

That leaves the investigation blocked on environment + a fix-direction decision, and the no-speculative-fix policy forbids pushing a fix without a fails-without/passes-with demo. Two questions:

1. Is there an aarch64-MSan lane we can use to reproduce (self-hosted ARM MSan runner, or an approved way to run the arm_msan stress job on a repro)? Without it there is no way to demonstrate a fix from an x86_64 host.

2. Given the confirmed mechanism (work-stealing executor resumes the async remote-read fiber on a different OS thread than it suspended on, and MSan's boost fiber-switch tracking, unlike ASan's per-context fake_stack, has no per-context stack save/restore), do you prefer:
   (A) a ClickHouse-side fix making the async remote-read fiber resume thread-affine (only resumed on its originating worker), or
   (B) a boost/MSan-instrumentation fix (per-context stack save/restore mirroring ASan, or correcting/suppressing the cross-thread MSan fiber-switch tracking)?

Once the repro path is settled I will implement the chosen direction in a separate PR and link it here.

[groeneai]

Fix PR opened: #110152.

The fix for this exact signature already existed on the boost fork (ClickHouse/boost branch fix-msan-fiber-aarch64-padding: a6da369 "Unpoison fiber stack after context switch on AArch64" + 24df31b externc declaration) but was never landed. The pinned contrib/boost (f05991c, on master) is its parent, so the unpoison was absent from every build. #110152 bumps contrib/boost f05991c -> 24df31b (clean fast-forward, 18 lines, all under BOOST_USE_MSAN + aarch64).

This is the mechanism confirmed here: AArch64 loses MSan shadow for struct-padding bytes in return values (llvm/llvm-project#54476); on heap fiber stacks the dirty shadow accumulates across boost::context switches and surfaces at __msan_finish_switch_fiber / resume() as the STID 4348-2010 fault. Unpoisoning the fiber stack after each switch clears it. Verification is via arm_msan CI.

[CheSema]

Reopening: the exact signature (STID 4348-2010) recurred on a build that contains the fix from #110152, so the contrib/boost unpoison-after-switch change is not sufficient.

Post-fix occurrence

• 2026-07-23, PR Skip async insert dedup prewarm when deduplication is disabled; extend prewarm test coverage #111549 (port-prewarm-dedup-tests — touches only InsertDeduplication, unrelated to this code path), Stress test (arm_msan), sha 45f2ca57a09f8627e509b7615cf4f1ea33d74588.
• CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=111549&sha=45f2ca57a09f8627e509b7615cf4f1ea33d74588&name_0=PR&name_1=Stress%20test%20%28arm_msan%29

Proof the fix was in the failing build

1. Submodule containment: git ls-tree 45f2ca57a09f8627e509b7615cf4f1ea33d74588 contrib/boost → 24df31b3e5c13c007efc533ef9f6ccd2e0c1f85d, which is exactly the boost commit Bump contrib/boost to fix AArch64 MSan fiber-switch false positives (STID 4348-2010) #110152 bumped to ("Unpoison fiber stack after context switch on AArch64" + externc declaration).
2. The crash stack itself shows the bumped boost source: the resume() frame moved from contrib/boost/boost/context/fiber_ucontext.hpp:513 (all pre-fix occurrences) to fiber_ucontext.hpp:530, consistent with the ~18 added lines of the fix. The rest of the stack is unchanged:

contrib/boost/boost/context/fiber_ucontext.hpp:530:13: boost::context::fiber::resume() &&
src/Common/Fiber.h:64:40: Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>::SuspendCallback::operator()()
src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:86:12: DB::RemoteQueryExecutorReadContext::Task::run(...)
src/Common/AsyncTaskExecutor.cpp:89:28: DB::AsyncTaskExecutor::Routine::operator()(...)
contrib/boost/boost/context/fiber_ucontext.hpp:330:13: boost::context::detail::fiber_capture_record<...>::run()
contrib/boost/boost/context/fiber_ucontext.hpp:81:11: boost::context::detail::fiber_entry_func<...>(void*)

Note: the other recent hit (2026-07-22, PR #110127, sha 41c0868e0997) was still on pre-fix boost f05991ceab7d (branch not rebased), so it is not evidence either way — the #111549 occurrence is the decisive post-fix one.

Implication

Either the AArch64 struct-padding-shadow diagnosis was incomplete (the unpoison does not cover the path that accumulates the poisoned shadow), or this is after all a real fiber-lifetime/cross-thread-resume bug that MSan happens to surface, per the earlier work-stealing analysis in this thread (the fiber is legitimately resumed on a different worker thread than the one it suspended on, and MSan — unlike ASan with its per-context fake_stack — keeps no per-context stack state across switches).

Since the issue was closed more than 8h ago, CI no longer auto-matches this signature, so every new occurrence lands on unrelated PRs as an unknown failure — reopening also restores the auto-match.

[groeneai]

Agreed, and thanks for the proof. It is conclusive: contrib/boost at 45f2ca57 is 24df31b (exactly what #110152 bumped to), and the resume() frame moving fiber_ucontext.hpp:513 -> :530 confirms the added unpoison lines were compiled into the failing build. So #110152 was necessary but not sufficient.

Reading of it: the boost whole-stack __msan_unpoison cleared the llvm/llvm-project#54476 padding-shadow false positive, but that was the surfaced symptom, not the whole cause. A recurrence with the fix present makes a real fiber-lifetime issue on the RemoteQueryExecutorReadContext teardown path the leading hypothesis now (cancel/restart racing the in-flight async read fiber under RandomQueryKiller + ZK faults + socket timeout), over the pure-MSan-artifact reading we shipped for.

Next: I will investigate AsyncTaskExecutor resume/cancel/restart/destroyFiber against an in-flight suspend_callback on the read path and follow up here with a PR or concrete findings. I won't re-link #110152 for this recurrence since it is proven present in the failing build.