Support window in more time-series aggregations #138456
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnhatn
added
>non-issue
:StorageEngine/ES|QL
Contributor
ℹ️ Important: Docs version tagging👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version. We use applies_to tags to mark version-specific features and changes. Expand for a quick overviewWhen to use applies_to tags:✅ At the page level to indicate which products/deployments the content applies to (mandatory) What NOT to do:❌ Don't remove or replace information that applies to an older version 🤔 Need help?
|
dnhatn
force-pushed
the
more-window-function-1
branch
from
b6f0988 to
b07352b
Compare
dnhatn
force-pushed
the
more-window-function-1
branch
from
b07352b to
1c43338
Compare
Collaborator
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
dnhatn
commented
Nov 24, 2025
| * 02:00 -> [12:00, 13:59) // bucket=2m | ||
| * 00:04 -> [04:00, 05:59), [06:00, 07:59), [08:00, 09:59), [10:00, 11:59), [12:00, 13:59) // window=10m | ||
| */ | ||
| private void expandWindowBuckets() { |
kkrik-es
reviewed
Nov 24, 2025
| * For example, if the window is 10 minutes and the bucket size is 2 minutes: | ||
| * If we see a bucket at 12:00, we need to ensure buckets exist for the previous 10 minutes. | ||
| * 02:00 -> [12:00, 13:59) // bucket=2m | ||
| * 00:04 -> [04:00, 05:59), [06:00, 07:59), [08:00, 09:59), [10:00, 11:59), [12:00, 13:59) // window=10m |
Contributor
There was a problem hiding this comment.
I'm somewhat confused by these examples - while the text above is clear. Shouldn't we have an example where, at 12:00, we need to include data from 11:50 and on?
Member
Author
There was a problem hiding this comment.
I will update the comment; the PR description may provide more clarity.
Contributor
There was a problem hiding this comment.
Right that example was nice.
kkrik-es
reviewed
Nov 24, 2025
.../compute/src/main/java/org/elasticsearch/compute/operator/TimeSeriesAggregationOperator.java
Show resolved
Hide resolved
kkrik-es
reviewed
Nov 24, 2025
| long endTimestamp = tsBlockHash.getLongKeyFromGroup(groupId); | ||
| long bucket = timeBucket.nextRoundingValue(endTimestamp - timeResolution.convert(largestWindowMillis())); | ||
| bucket = Math.max(bucket, tsBlockHash.getMinLongKey()); | ||
| while (bucket < endTimestamp) { |
Contributor
There was a problem hiding this comment.
Nit: consider adding a comment here, we're filling in missing buckets due to the window expanding outside each bucket.
Member
Author
|
Thanks Kostas! |
afoucret
pushed a commit
to afoucret/elasticsearch
that referenced
this pull request
Nov 26, 2025
This change adds support for window functions for additional time-series
aggregations, including `min_over_time`, `max_over_time`,
`first_over_time`, `count_over_time`, and `sum_over_time`. These changes
are straightforward. The main update in this PR is how the window is
expanded before sliding over the partial results.
For example, given these data points:
```
|_tsid| cluster| host | timestamp | metric |
| t1 | prod | h1 | 2025-04-15T01:12:00Z | 100 |
| t2 | prod | h2 | 2025-04-15T01:14:00Z | 200 |
```
With `bucket=5s` and no window:
```
TS ...
| WHERE TRANGE('2025-04-15T01:10:00Z', '2025-04-15T01:15:00Z')
| STATS sum(sum_over_time(metric)) BY host, TBUCKET(5s)
```
Yields:
```
cluster | bucket | SUM |
prod | 2025-04-15T01:10:00Z | 300 |
```
With a window=5s:
```
TS ...
| WHERE TRANGE('2025-04-15T01:10:00Z', '2025-04-15T01:15:00Z')
| STATS sum(sum_over_time(metric, 5s)) BY host, TBUCKET(1s)
```
Yields:
```
cluster | bucket | SUM |
prod | 2025-04-15T01:12:00Z | 100 |
prod | 2025-04-15T01:14:00Z | 200 |
```
Ideally, all buckets from `2025-04-15T01:10:00Z` to
`2025-04-15T01:14:00Z` should be generated:
```
cluster | bucket | SUM |
prod | 2025-04-15T01:10:00Z | 300 |
prod | 2025-04-15T01:11:00Z | 300 |
prod | 2025-04-15T01:12:00Z | 300 |
prod | 2025-04-15T01:13:00Z | 200 |
prod | 2025-04-15T01:14:00Z | 200 |
```
With this change, buckets are expanded as if sliding over the raw input
before combining for the final results.
ncordon
pushed a commit
to ncordon/elasticsearch
that referenced
this pull request
Nov 26, 2025
This change adds support for window functions for additional time-series
aggregations, including `min_over_time`, `max_over_time`,
`first_over_time`, `count_over_time`, and `sum_over_time`. These changes
are straightforward. The main update in this PR is how the window is
expanded before sliding over the partial results.
For example, given these data points:
```
|_tsid| cluster| host | timestamp | metric |
| t1 | prod | h1 | 2025-04-15T01:12:00Z | 100 |
| t2 | prod | h2 | 2025-04-15T01:14:00Z | 200 |
```
With `bucket=5s` and no window:
```
TS ...
| WHERE TRANGE('2025-04-15T01:10:00Z', '2025-04-15T01:15:00Z')
| STATS sum(sum_over_time(metric)) BY host, TBUCKET(5s)
```
Yields:
```
cluster | bucket | SUM |
prod | 2025-04-15T01:10:00Z | 300 |
```
With a window=5s:
```
TS ...
| WHERE TRANGE('2025-04-15T01:10:00Z', '2025-04-15T01:15:00Z')
| STATS sum(sum_over_time(metric, 5s)) BY host, TBUCKET(1s)
```
Yields:
```
cluster | bucket | SUM |
prod | 2025-04-15T01:12:00Z | 100 |
prod | 2025-04-15T01:14:00Z | 200 |
```
Ideally, all buckets from `2025-04-15T01:10:00Z` to
`2025-04-15T01:14:00Z` should be generated:
```
cluster | bucket | SUM |
prod | 2025-04-15T01:10:00Z | 300 |
prod | 2025-04-15T01:11:00Z | 300 |
prod | 2025-04-15T01:12:00Z | 300 |
prod | 2025-04-15T01:13:00Z | 200 |
prod | 2025-04-15T01:14:00Z | 200 |
```
With this change, buckets are expanded as if sliding over the raw input
before combining for the final results.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change adds support for window functions for additional time-series aggregations, including
min_over_time,max_over_time,first_over_time,count_over_time, andsum_over_time. These changes are straightforward. The main update in this PR is how the window is expanded before sliding over the partial results.For example, given these data points:
With
bucket=5sand no window:Yields:
With a window=5s:
Yields:
Ideally, all buckets from
2025-04-15T01:10:00Zto2025-04-15T01:14:00Zshould be generated:With this change, buckets are expanded as if sliding over the raw input before combining for the final results.