Add relevant attributes to shard search latency APM metrics #134798
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We already record latency of the query and fetch phase as APM metrics. We'd like to be able to slice such latencies based on some recurring categories of the request: - does it have agg or hit only? - is it sorted by field or by score? - does it have a time range filter? - does it target user data or internal indices? This commit introduces introspection for a shard search request and stores the extracted attributes together with the shard phase latency metrics. This builds on top of elastic#134232 to use the same infra and store the same attributes for shard level latency metrics.
javanna
added
>enhancement
:Search Foundations/Search
|
Pinging @elastic/es-search-foundations (Team:Search Foundations) |
elasticsearchmachine
added
the
Team:Search Foundations
|
Hi @javanna, I've created a changelog YAML for you. |
javanna
commented
Sep 18, 2025
| shardSearchRequest.shardId().getIndexName() | ||
| ); | ||
| boolean isSystem = ((EsExecutors.EsThread) Thread.currentThread()).isSystem(); | ||
| attributes.put(SYSTEM_THREAD_ATTRIBUTE_NAME, isSystem); |
There was a problem hiding this comment.
The isSystem attribute I moved here from the ShardSearchPhaseAPMMetrics class.
javanna
commented
Sep 18, 2025
| // don't track unbounded ranges, they translate to either match_none if the field does not exist | ||
| // or match_all if the field is mapped | ||
| case TIMESTAMP -> { | ||
| if (range.to() != null || range.from() != null) { |
There was a problem hiding this comment.
This is something I discovered as part of working on this chance: rewrite for range queries is a bit funky (I plan on fixing that as a follow-up). A range query may rewrite to one without bounds, if all data is within the range. In that case, it is equivalent to a match_all hence I don't report it as a range. I decided to rather report only "true" range queries at the shard level.
tteofili
approved these changes
Sep 18, 2025
andreidan
approved these changes
Sep 18, 2025
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Sep 26, 2025
We recently added relevant attributes to the existing shard search latency metrics (elastic#134798). This commit introduces an additional attribute that analyzes the parsed time range filter against the @timestamp field and reports back whether it is within the last 15 minutes, last hour, last 12 hours, last day, last three days, last seven days, or last 14 days.
javanna
added a commit
that referenced
this pull request
Sep 26, 2025
#135524) We recently added relevant attributes to the existing shard search latency metrics (#134798). This commit introduces an additional attribute that analyzes the parsed time range filter against the @timestamp field and reports back whether it is within the last 15 minutes, last hour, last 12 hours, last day, last three days, last seven days, or last 14 days.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We already record latency of the query and fetch phase as APM metrics. We'd like to be able to slice such latencies based on some recurring categories of the request:
This commit introduces introspection for a shard search request and stores the extracted attributes together with the shard phase latency metrics.
This builds on top of #134232 to use the same infra and store the same attributes for shard level latency metrics.
Here are some of the differences of the attributes extracted at the shard level compared to those extracted at the coord level, for the same query: