branch-4.0: [Enhancement](BE) Make print stack API more stable #64093#65283
Open
zclllyybb wants to merge 2 commits into
Open
branch-4.0: [Enhancement](BE) Make print stack API more stable #64093#65283zclllyybb wants to merge 2 commits into
zclllyybb wants to merge 2 commits into
Conversation
This PR adds a BE-side diagnostic HTTP API for collecting thread stack
traces from a live BE process:
```text
GET /api/stack_trace
GET /api/stack_trace?thread_id=<tid>[,<tid>...]
GET /api/stack_trace?mode=<DISABLED|FAST|FULL|FULL_WITH_INLINE>
GET /api/stack_trace?dwarf_location_info_mode=<DISABLED|FAST|FULL|FULL_WITH_INLINE>
GET /api/stack_trace?timeout_ms=<ms>
GET /api/stack_trace?skip_blocking_syscalls=<true|false>
```
`thread_id` is the preferred explicit selector. `tid` is kept only as a
legacy alias.
The default full-process collection samples all BE threads without a
signal-attempt cap and does not skip blocking syscalls. This is
intentional: during real incidents, blocked worker stacks are often the
useful part of the dump. `skip_blocking_syscalls=true` remains available
as an explicit conservative mode for isolating EINTR-sensitive paths.
The output is plain text and includes:
- BE pid, diagnostic signal number, thread count, timeout, DWARF mode,
and syscall-skip mode.
- One section per thread, with TID, thread name, capture status, capture
method, frame count, frame-pointer status, and stack bounds.
- A summary line with captured, skipped, timed-out, signal-error, and
exited counts.
Example header:
```text
BE thread stack traces
pid: 96543
service_signal: 40
thread_count: 1168
timeout_ms_per_thread: 3000
dwarf_location_info_mode: fast
skip_blocking_syscalls: false
signal_handler_unwinder: frame_pointer_with_coordinator_signal_context_libunwind_fallback
```
Example execution stack captured under load:
```text
----- thread 12438 (p_normal_simple) status=ok capture_method=frame_pointer frames=32 fp_status=end_of_chain stack_bounds=... -----
doris::CastToStringFunction::execute_impl(...)
doris::VectorizedFnCall::execute_column_impl(...)
doris::AggSinkLocalState::_execute_without_key(...)
doris::AggSinkOperatorX::sink_impl(...)
doris::DataSinkOperatorXBase::sink(...)
doris::PipelineTask::execute(bool*)
doris::TaskScheduler::_do_work(int)
```
Implementation notes:
- The remote-thread capture path uses a signal handler to copy only
lightweight frame-pointer/context data from the target thread.
Symbolization and fallback unwinding run outside the handler.
- The handler publishes through one process-wide slot, so HTTP
collection is serialized. The coordinator waits until the previous
handler has fully released its latch before signaling the next TID;
otherwise back-to-back captures can drop the next signal.
- Stack bounds are collected before sending any signal, because the
handler must not open `/proc`, allocate memory, or take locks while
interrupting an arbitrary BE thread.
- Frame-pointer walking validates stack-range bounds and pointer
alignment before reading frame records.
- The same PCs can be rendered with different DWARF modes, so the
StackTrace cache key includes `dwarf_location_info_mode`. This prevents
a `DISABLED` rendering from poisoning a later `FULL` request and
prevents full file/line detail from leaking into a disabled request.
- Instruction addresses are compared via `uintptr_t` instead of pointer
arithmetic on non-object instruction addresses.
- Threads that block the diagnostic signal are reported as
`status=signal_blocked` instead of being treated as ambiguous timeouts.
- This is not a global stop-the-world snapshot. Each thread is sampled
independently as the coordinator walks the TID list, so stacks are close
in time but not guaranteed to be from the exact same instant.
EINTR handling:
- Sending a diagnostic signal can interrupt blocking syscalls even when
`SA_RESTART` is used, especially for poll/select/accept-style waits.
- Doris Thrift servers previously could treat `Interrupted system call`
from `TServerSocket::acceptImpl()` as a fatal server-loop exception.
- This PR wraps BE Thrift server sockets with `ImprovedServerSocket`,
which retries only the narrow EINTR/`Interrupted system call` case for
accept/read/peek. Other transport exceptions keep the original behavior.
(cherry picked from commit 0141aa2)
Issue Number: close apache#62497 follow up apache#64454 This PR adds a safer BE print stack API based on libunwind signal-context unwinding. Stack unwinding may call `dl_iterate_phdr()` to find DWARF/FDE metadata. In signal-based stack capture, jemalloc profiling, or concurrent `dlopen()` / `dlclose()`, re-entering glibc's loader-lock path may deadlock or become unsafe. Route GNU libunwind's PHDR lookup through Doris' lock-free PHDR cache. The signal handler unwinds from the kernel-provided signal context, while libunwind reads loaded-object metadata from Doris' PHDR snapshot instead of calling glibc `dl_iterate_phdr()` directly. Normal code paths still use the original live `dl_iterate_phdr()` behavior. The research project is at https://github.com/JoverZhang/Doris-BE-Print-Stack-API-Research Co-authored-by: zhaochangle <zhaochangle@selectdb.com> (cherry picked from commit dc70115)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
zclllyybb
changed the title
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Backport the stack trace API stability enhancement to
branch-4.0.This PR includes:
#64454/0141aa2: add the BE stack trace HTTP API dependency#64093/dc70115: make the print stack API more stableValidation
./build.sh --becompleted successfully in thebranch-4.0worktreegit diff --check upstream/branch-4.0..HEADpassed