search: fix trace fragmentation and restructure spans#6587
Conversation
Per-split leaf work was spawned without propagating the current span, starting a new trace instead of nesting under multi_index_leaf_search. Also prunes collinear gRPC-adapter spans, adds spans for distinct phases (plan_splits_for_root_search, list_relevant_splits, fetch_docs_in_split, leaf_search_single_split_wrapper), splits the CPU-pool queue wait out of tantivy_search, and surfaces search_partial_hits_phase at INFO. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ecbd534cac
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // The `grpc-request` span installed by the tonic server's `trace_fn` already | ||
| // extracts the caller's W3C trace context from the request headers and is the | ||
| // active span for each handler, so these methods deliberately add no span of | ||
| // their own to avoid collinear duplicates. |
There was a problem hiding this comment.
For incoming gRPC search calls carrying W3C trace headers, removing set_current_span_parent_from_metadata(request.metadata()) leaves no code path that installs the remote parent: I checked start_grpc_server, and it builds the tonic router with Server::builder().add_service(...) but no TraceLayer/trace_fn, while the generated SearchServiceServer simply invokes these adapter methods. In cross-node searches this makes RootSearch/LeafSearch/FetchDocs/etc. start independent traces instead of being stitched to the caller, so either keep the metadata extraction here or add the actual tonic request span before dropping it.
Useful? React with 👍 / 👎.
1 participant
This PR fixes a few issues with how traces are emitted to Datadog:
multi_index_leaf_searchplan_splits_for_root_search,list_relevant_splits,fetch_docs_in_split, and a per-split wrapper (leaf_search_single_split_wrapper) so each split's wait + warmup + tantivy form one subtreetantivy_searchintowaiting_for_cpu_pool.search_partial_hits_phaseto have all leaves as children🤖 Generated with Claude Code