Skip to content

Conversation

@JonasKunz
Copy link
Contributor

@JonasKunz JonasKunz commented Nov 25, 2025

Adds support for querying exponential histograms in TS queries, e.g.:

TS exp_histo_sample 
 | WHERE TRANGE(to_datetime("2025-09-25T00:30:00Z"), to_datetime("2025-09-25T01:00:00Z"))
 | STATS min = MIN(responseTime),
         max = MAX(responseTime),
         median = MEDIAN(responseTime),
         p75 = PERCENTILE(responseTime,75),
         sum = SUM(responseTime),
         avg = AVG(responseTime) 
         BY instance, time=TBUCKET(10m)

The query above aggregates all histogram samples per bucket and yields the corresponding aggregation results.
This PR implements this behaviour by ensuring that we use a hidden merge_over_time inner aggregate function when the queried field is of type exponential_histogram.

This PR does not handle other _over_time functions yet (e.g. last_over_time).
This will be fixed in a follow-up PR.

@elasticsearchmachine elasticsearchmachine added external-contributor Pull request authored by a developer outside the Elasticsearch team v9.3.0 labels Nov 25, 2025
@JonasKunz JonasKunz added >non-issue :Analytics/ES|QL AKA ESQL test-release Trigger CI checks against release build labels Nov 25, 2025
@JonasKunz JonasKunz marked this pull request as ready for review November 25, 2025 11:56
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Nov 25, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@JonasKunz JonasKunz requested review from dnhatn and kkrik-es November 25, 2025 11:59
@JonasKunz
Copy link
Contributor Author

Release tests seem to be failing for an unrelated reason, so I'm removing the tag again.

@JonasKunz JonasKunz removed the test-release Trigger CI checks against release build label Nov 25, 2025
// Then check if there is TimeSeriesAggregateFunction on the path to the outer aggregation (in the chain of parents).
// If not, wrap the field reference with the appropriate TimeSeriesAggregateFunction based on its type
Expression aggregatedExpression = af.field();
if (aggregatedExpression instanceof ExtractHistogramComponent extractHistogramComponent) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the type of the inner field? I wonder if we should generalize this, having a loop that keeps unrolling fields until it finds a real field.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this instanceof case we know that it must be an actual field:
ExtractHistogramComponent is a function which only accepts exponential_histogram-typed values as first parameter. The only way to get one is either a Merge aggregation (which is impossible here because we are alread in an aggregation) or a field reference.

I tried to outline the general solution for this problem in the comment above:

// One possible strategy would be to search for all field references in the expression.
// Then check if there is TimeSeriesAggregateFunction on the path to the outer aggregation (in the chain of parents).
// If not, wrap the field reference with the appropriate TimeSeriesAggregateFunction based on its type

This would be the general solution where we don't need to know all the intermediate expressions and which also works if multiple fields (e.g. SUM(gaugeA + gaugeB) are involved.

Should I create an issue for this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened #138702

Copy link
Contributor

@kkrik-es kkrik-es left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cute! Please wait for Nhat too.

Copy link
Member

@dnhatn dnhatn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Jonas!

@JonasKunz JonasKunz merged commit 26a95cb into elastic:main Nov 27, 2025
34 checks passed
@JonasKunz JonasKunz deleted the exp-histo-ts branch November 27, 2025 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL external-contributor Pull request authored by a developer outside the Elasticsearch team >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants