[fix](asan) Restore __asan_handle_no_return on the throw path bypassed by the static link#65366
Open
morningman wants to merge 1 commit into
Open
[fix](asan) Restore __asan_handle_no_return on the throw path bypassed by the static link#65366morningman wants to merge 1 commit into
morningman wants to merge 1 commit into
Conversation
…d by the static link
For years the BE has crashed only in ASAN builds, on the thrift RPC
retry/reopen path (and, before the log guards were added, inside glog
LOG(WARNING)), always with the same signature:
AddressSanitizer: CHECK failed: asan_thread.cpp:36x
"((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
Same failure family as google/glog#978 and google/sanitizers#1010.
It was never root-caused, only worked around (apache#34813 apache#34994 apache#36808
apache#37134 apache#48347 apache#49516 apache#49878).
Root cause: libthrift and the other third-party static libs are built
without ASAN instrumentation. ASAN cleans the stack shadow on exception
unwinding via __asan_handle_no_return(), emitted by the compiler before
instrumented throws and provided as weak __cxa_throw /
_Unwind_RaiseException interceptors for un-instrumented throws. Under
the BE's fully static link (-static-libstdc++/-static-libgcc plus a
static sanitizer runtime) those weak interceptors lose to libstdc++.a's
/ libgcc_eh.a's strong definitions at link time, so the cleanup never
runs when un-instrumented code throws (e.g. TSocket raising
TTransportException on a FE restart). The unwound instrumented frames
keep their stack-shadow poison; the retry path immediately re-descends,
innocent locals land on the stale poison, the first intercepted write
reports a false positive, and ASAN aborts while walking the
already-overwritten frame descriptor. RELEASE/DEBUG builds have no
shadow memory, which is why this never happened outside ASAN.
The fix: ASAN/ASAN_UT Linux links now pass
-Wl,--wrap=__cxa_throw -Wl,--wrap=__cxa_rethrow
-Wl,--wrap=_Unwind_RaiseException
and be/src/common/asan_cxa_throw_wrap.cpp provides wrappers that call
__asan_handle_no_return() before forwarding to the real
implementations -- exactly what the bypassed official interceptors do.
--wrap rewrites every reference from every input object, including the
un-instrumented third-party archives and libstdc++.a itself. The flags
are attached to DORIS_LINK_LIBS so any executable receiving them also
links Common, which carries the wrapper definitions.
_Unwind_RaiseException is wrapped too because std::rethrow_exception
and foreign-language unwinders raise through it without touching
__cxa_throw; its wrapper forwards the return value (it returns when no
handler is found). Production builds are byte-identical.
With the throw path fixed, the ASAN-only forks are removed: 15
"#ifndef ADDRESS_SANITIZER" sites in thrift_rpc_helper.cpp (6),
agent/utils.cpp (6), runtime_query_statistics_mgr.cpp (2, one of which
skipped the whole reopen+retry under ASAN -- the failure pattern
apache#49516 exposed), pipeline_fragment_context.cpp (1), plus the leftover
std::cerr LOG substitutes. ASAN CI now runs the exact production
retry/reopen/logging code.
Verified on the ASAN pipeline via apache#65358 with a deterministic
reproducer: unfixed builds show the interceptor bypass at the link
level, measurable stale stack-shadow poison after an un-instrumented
throw (58/160 probed frames), and a deterministic startup crash with
the historical stack; with this fix the same reproducer measures 0/160
poisoned frames, survives the full crash-site sweep, and the BE runs
the regression normally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Collaborator
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 10th, 2026 6:44 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
TPC-H: Total hot run time: 29219 ms |
Contributor
TPC-DS: Total hot run time: 181501 ms |
Contributor
ClickBench: Total hot run time: 24.91 s |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review completed for PR 65366.
Review opinion: no blocking PR-specific issue found, and I am not submitting inline comments.
Critical checkpoint conclusions:
- Task goal: The PR targets ASAN-only BE crashes caused by uninstrumented/static exception paths bypassing
__asan_handle_no_return(). The added ASAN/ASAN_UT Linux wrap flags andbe/src/common/asan_cxa_throw_wrap.cppaddress that path, and the removed ASAN guards restore the same thrift retry/logging behavior already used by non-ASAN builds. - Scope/focus: The change is focused on ASAN link/runtime exception handling and removal of ASAN-only RPC forks. No unrelated FE/Nereids, storage, transaction, or protocol surface is changed.
- Concurrency/lifecycle: No new threads, locks, shared mutable state, cache ownership, or object lifecycle paths were introduced. The wrapper is process-level link interposition with no persistent state.
- Configuration/compatibility: No new config, session variable, thrift field, persisted format, or rolling-upgrade compatibility concern was introduced. The link flags are ASAN/ASAN_UT plus Linux only; release/debug builds do not receive the new wrap flags.
- Parallel paths: ASAN now follows the same thrift RPC reconnect/retry/report code paths as non-ASAN.
Commonincludes the new wrapper source and BE unit-test links inherit the updatedDORIS_LINK_LIBS. - Error handling and observability: The removed guards restore
LOGcalls and retry/reopen behavior under ASAN. Existing best-effort discardedreopen()cleanup statuses were checked and are pre-existing, unchanged lines rather than new PR behavior. - Tests/validation: I could not run a full local Doris build because this Actions checkout is missing
.worktree_initializedandthirdparty/installed. I did run syntax checks for the wrapper with GCC and Clang, with and withoutADDRESS_SANITIZER, plus clang-format dry-run andgit show --check. Current PR statuses show Linux compile, BE UT, style, P0, non-concurrent, cloud, vault, and performance checks passing. The macOS BE UT failure stops before build due to JDK 25 vs required JDK 17. The external regression failures are external Paimon/Iceberg/Hive/HDFS tests; the detailed failure inspected is ASANMEM_LIMIT_EXCEEDEDin scanner/table-reader materialization, not in the touched wrapper or thrift RPC paths.
Subagent conclusions:
optimizer-rewritefound no optimizer/rewrite, semantic-equivalence, or parallel join/aggregate candidate.tests-session-configfound no regression-output, session/config propagation, compatibility, or basic CI/style candidate.- Convergence round 1 ended with both live subagents replying
NO_NEW_VALUABLE_FINDINGSfor the same ledger/comment set with no inline comments proposed.
User focus: no additional user-provided focus points were present.
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65358 (reproduction & verification branch, do-not-merge), workaround history: #34813 #34994 #36808 #37134 #48347 #49516 #49878
Problem Summary:
For years the BE has crashed only in ASAN builds, on the thrift RPC retry/
reopenpath (and, before the#ifndef ADDRESS_SANITIZERlog guards were added, inside glogLOG(WARNING)), always with the same signature:Same failure family as google/glog#978 and google/sanitizers#1010. It was never root-caused, only worked around with ASAN-only log guards and code-path forks — and #49516 showed those forks hatch real bugs of their own.
Root cause. libthrift and the other third-party static libs are built without ASAN instrumentation. ASAN cleans the stack shadow on exception unwinding via
__asan_handle_no_return(): the compiler emits it before instrumented throws, and the runtime provides weak__cxa_throw/_Unwind_RaiseExceptioninterceptors to cover un-instrumented throws. Under the BE's fully static link (-static-libstdc++/-static-libgccplus a static sanitizer runtime) those weak interceptors lose to libstdc++.a's / libgcc_eh.a's strong definitions at link time, so the cleanup never runs when un-instrumented code throws (e.g.TSocketraisingTTransportExceptionwhen the FE restarts). The unwound instrumented frames keep their stack-shadow poison; the retry path immediately re-descends over the same stack range, innocent locals (thrift'sgetSocketInfo()strings, glog'sstruct tm, ...) land on the stale poison, the first intercepted write reports a false positive, and ASAN aborts while walking the already-overwritten frame descriptor — before it can even print a normal report. RELEASE/DEBUG builds have no shadow memory, which is why this never happened outside ASAN.The fix. ASAN/ASAN_UT Linux links now pass
and
be/src/common/asan_cxa_throw_wrap.cppprovides wrappers that call__asan_handle_no_return()before forwarding to the real implementations — exactly what the bypassed official interceptors do.--wraprewrites every reference from every input object, including the un-instrumented third-party archives and libstdc++.a itself. The flags are attached toDORIS_LINK_LIBS, so any executable receiving them also linksCommon, which carries the wrapper definitions — flags and definitions cannot drift apart._Unwind_RaiseExceptionis wrapped too becausestd::rethrow_exceptionand foreign-language unwinders raise through it without touching__cxa_throw; its wrapper forwards the return value (it returns when no handler is found). Production (RELEASE/DEBUG) builds are byte-identical.Cleanup. With the throw path fixed, the ASAN-only forks are removed — 15
#ifndef ADDRESS_SANITIZERsites inthrift_rpc_helper.cpp(6),agent/utils.cpp(6),runtime_query_statistics_mgr.cpp(2, one of which skipped the whole reopen+retry under ASAN, the failure pattern #49516 exposed),pipeline_fragment_context.cpp(1), plus the leftoverstd::cerrLOG substitutes. ASAN CI now exercises the exact production retry/reopen/logging code.Verification. Done on the ASAN pipeline via the dedicated reproduction branch #65358, which carries a deterministic startup reproducer (not part of this PR):
__cxa_throw ... NOT intercepted (bypassed by static link)WRAPPED by __wrap___cxa_throwreport()fail →close()→flush()→getSocketInfo(), swept over 257 stack offsets)Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)
🤖 Generated with Claude Code