Fix multi fields in downsampling #138869

gmarouli · 2025-12-02T10:11:14Z

Recently, we identified certain edge cases when downsampling multi-fields:

A time series metric should not be downsampled when it's a sub-field because it needs to be downsampled differently than the parent field.
When a field has multiple sub-fields, we only need to use one as the source to retrieve the value, it could be either the parent or one of the sub-fields. If one of the parent or one of the sub-fields is a dimension, we choose the dimension sub-field as a source. Otherwise, if the parent field is not aggregatable, we choose the first aggregatable sub-field we have seen.

…rce.

elasticsearchmachine · 2025-12-02T10:12:00Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

elasticsearchmachine · 2025-12-02T10:12:26Z

Hi @gmarouli, I've created a changelog YAML for you.

martijnvg

LGTM 👍

elasticsearchmachine · 2025-12-04T09:58:12Z

💔 Backport failed

Status	Branch	Result
❌	9.1	Commit could not be cherrypicked due to conflicts
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.2	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 138869

(cherry picked from commit 5918605) # Conflicts: # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/AggregateSubMetricFieldValueFetcher.java # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/DownsampleShardIndexer.java # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/FieldValueFetcher.java # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/RestDownsampleAction.java # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/TransportDownsampleAction.java

(cherry picked from commit 5918605) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.2.csv # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/AggregateSubMetricFieldValueFetcher.java # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/DownsampleShardIndexer.java # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/FieldValueFetcher.java # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/RestDownsampleAction.java # x-pack/plugin/downsample/src/main/java/org/elasticsearch/xpack/downsample/TransportDownsampleAction.java # x-pack/plugin/downsample/src/test/java/org/elasticsearch/xpack/downsample/DownsampleShardPersistentTaskExecutorTests.java

gmarouli · 2025-12-04T12:22:55Z

💔 Some backports could not be created

Status	Branch	Result
✅	9.2
✅	9.1
✅	8.19

Manual backport

To create the backport manually run:

backport --pr 138869

Questions ?

Please refer to the Backport tool documentation

* Fix multi fields in downsampling (#138869) (cherry picked from commit 5918605)

* Fix multi fields in downsampling (#138869)

* [8.19] Fix multi fields in downsampling (#138869) * Add missing capabilities

@timestamp

In #138869 we fixed the handling of the multi-fields; however, we introduced a bug when a mapping contains a alias type. For example, the following mapping: ``` "@timestamp": { "type": "date" }, "timestamp": { "type": "alias", "path": "@timestamp" } ``` The cause of the bug was that in the `FieldValueFetcher` we switch the check wether a field is indexed from the name we retrieved in the label to the name of the field type. In this case, the label is `timestamp` which does not in the indexed but the name of the field type is "@timestamp".

@timestamp

…tic#139104) In elastic#138869 we fixed the handling of the multi-fields; however, we introduced a bug when a mapping contains a alias type. For example, the following mapping: ``` "@timestamp": { "type": "date" }, "timestamp": { "type": "alias", "path": "@timestamp" } ``` The cause of the bug was that in the `FieldValueFetcher` we switch the check wether a field is indexed from the name we retrieved in the label to the name of the field type. In this case, the label is `timestamp` which does not in the indexed but the name of the field type is "@timestamp". (cherry picked from commit 47698be) # Conflicts: # x-pack/plugin/downsample/src/internalClusterTest/java/org/elasticsearch/xpack/downsample/DownsampleIT.java

@timestamp

…tic#139104) In elastic#138869 we fixed the handling of the multi-fields; however, we introduced a bug when a mapping contains a alias type. For example, the following mapping: ``` "@timestamp": { "type": "date" }, "timestamp": { "type": "alias", "path": "@timestamp" } ``` The cause of the bug was that in the `FieldValueFetcher` we switch the check wether a field is indexed from the name we retrieved in the label to the name of the field type. In this case, the label is `timestamp` which does not in the indexed but the name of the field type is "@timestamp". (cherry picked from commit 47698be) # Conflicts: # x-pack/plugin/downsample/src/internalClusterTest/java/org/elasticsearch/xpack/downsample/DownsampleIT.java

@timestamp

…139111) In #138869 we fixed the handling of the multi-fields; however, we introduced a bug when a mapping contains a alias type. For example, the following mapping: ``` "@timestamp": { "type": "date" }, "timestamp": { "type": "alias", "path": "@timestamp" } ``` The cause of the bug was that in the `FieldValueFetcher` we switch the check wether a field is indexed from the name we retrieved in the label to the name of the field type. In this case, the label is `timestamp` which does not in the indexed but the name of the field type is "@timestamp". (cherry picked from commit 47698be) # Conflicts: # x-pack/plugin/downsample/src/internalClusterTest/java/org/elasticsearch/xpack/downsample/DownsampleIT.java

@timestamp

…139114) In #138869 we fixed the handling of the multi-fields; however, we introduced a bug when a mapping contains a alias type. For example, the following mapping: ``` "@timestamp": { "type": "date" }, "timestamp": { "type": "alias", "path": "@timestamp" } ``` The cause of the bug was that in the `FieldValueFetcher` we switch the check wether a field is indexed from the name we retrieved in the label to the name of the field type. In this case, the label is `timestamp` which does not in the indexed but the name of the field type is "@timestamp". (cherry picked from commit 47698be) # Conflicts: # x-pack/plugin/downsample/src/internalClusterTest/java/org/elasticsearch/xpack/downsample/DownsampleIT.java

@timestamp

…139118) In #138869 we fixed the handling of the multi-fields; however, we introduced a bug when a mapping contains a alias type. For example, the following mapping: ``` "@timestamp": { "type": "date" }, "timestamp": { "type": "alias", "path": "@timestamp" } ``` The cause of the bug was that in the `FieldValueFetcher` we switch the check wether a field is indexed from the name we retrieved in the label to the name of the field type. In this case, the label is `timestamp` which does not in the indexed but the name of the field type is "@timestamp". (cherry picked from commit 47698be) # Conflicts: # x-pack/plugin/downsample/src/internalClusterTest/java/org/elasticsearch/xpack/downsample/DownsampleIT.java

@timestamp

…tic#139104) In elastic#138869 we fixed the handling of the multi-fields; however, we introduced a bug when a mapping contains a alias type. For example, the following mapping: ``` "@timestamp": { "type": "date" }, "timestamp": { "type": "alias", "path": "@timestamp" } ``` The cause of the bug was that in the `FieldValueFetcher` we switch the check wether a field is indexed from the name we retrieved in the label to the name of the field type. In this case, the label is `timestamp` which does not in the indexed but the name of the field type is "@timestamp".

gmarouli added 6 commits December 2, 2025 11:55

Add test with multifield edge cases

a8bca08

Specialise the mapping visitor to be able to subobjects and multi-fields

8c15978

Small fixes

c70927e

Time series field are collected once and associated with a single sou…

0a8fe18

…rce.

Expand the assertions of the test

8dd56fd

Add unit tests for TimeSeriesFields

29e6302

elasticsearchmachine added v9.3.0 needs:triage Requires assignment of a team area label labels Dec 2, 2025

gmarouli added the :StorageEngine/Downsampling Downsampling (replacement for rollups) - Turn fine-grained time-based data into coarser-grained data label Dec 2, 2025

elasticsearchmachine added Team:StorageEngine and removed needs:triage Requires assignment of a team area label labels Dec 2, 2025

gmarouli added >bug needs:triage Requires assignment of a team area label and removed Team:StorageEngine labels Dec 2, 2025

elasticsearchmachine added the Team:StorageEngine label Dec 2, 2025

Update docs/changelog/138869.yaml

8214495

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Dec 2, 2025

Remove flakiness from test

cc1dc2b

gmarouli mentioned this pull request Dec 2, 2025

Fix downsampling with disabled subobjects #138715

Merged

gmarouli requested a review from martijnvg December 2, 2025 12:25

gmarouli added 2 commits December 3, 2025 09:59

merge with main and bump version

e11ed11

Merge branch 'main' into fix-multi-fields-in-downsampling

10232f9

gmarouli added auto-backport Automatically create backport pull requests when merged branch:9.2 branch:9.1 branch:8.19 labels Dec 3, 2025

elasticsearchmachine added v9.1.9 v8.19.9 labels Dec 3, 2025

gmarouli and others added 2 commits December 3, 2025 16:17

Rename to multiFieldSources

4fec499

[CI] Update transport version definitions

7f2cce6

gmarouli requested a review from martijnvg December 3, 2025 14:44

martijnvg approved these changes Dec 4, 2025

View reviewed changes

gmarouli added 3 commits December 4, 2025 09:57

Polishing

3416777

Merge branch 'main' into fix-multi-fields-in-downsampling

6f4e2bc

Merge branch 'main' into fix-multi-fields-in-downsampling

5bb7c5b

gmarouli merged commit 5918605 into elastic:main Dec 4, 2025
34 checks passed

gmarouli deleted the fix-multi-fields-in-downsampling branch December 4, 2025 09:56

elasticsearchmachine added the backport pending label Dec 4, 2025

gmarouli mentioned this pull request Dec 4, 2025

[9.2] Fix multi fields in downsampling (#138869) #139049

Merged

gmarouli mentioned this pull request Dec 4, 2025

[9.1] Fix multi fields in downsampling (#138869) #139051

Merged

gmarouli mentioned this pull request Dec 4, 2025

[8.19] Fix multi fields in downsampling (#138869) #139054

Merged

gmarouli added a commit that referenced this pull request Dec 4, 2025

[9.2] Fix multi fields in downsampling (#138869) (#139049)

b8993c3

* Fix multi fields in downsampling (#138869) (cherry picked from commit 5918605)

gmarouli added a commit that referenced this pull request Dec 4, 2025

[9.1] Fix multi fields in downsampling (#138869) (#139051)

15168d0

* Fix multi fields in downsampling (#138869)

elasticsearchmachine pushed a commit that referenced this pull request Dec 4, 2025

[8.19] Fix multi fields in downsampling (#138869) (#139054)

0869145

* [8.19] Fix multi fields in downsampling (#138869) * Add missing capabilities

gmarouli added a commit to gmarouli/elasticsearch that referenced this pull request Dec 5, 2025

Fix bug introduced when we fixed multi-fields in elastic#138869

ac4cc3b

gmarouli mentioned this pull request Dec 5, 2025

Fix bug introduced when we fixed multi-fields in #138869 #139104

Merged

gmarouli removed the backport pending label Dec 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix multi fields in downsampling #138869

Fix multi fields in downsampling #138869

Uh oh!

gmarouli commented Dec 2, 2025

Uh oh!

elasticsearchmachine commented Dec 2, 2025

Uh oh!

elasticsearchmachine commented Dec 2, 2025

Uh oh!

martijnvg left a comment

Uh oh!

Uh oh!

elasticsearchmachine commented Dec 4, 2025

Uh oh!

gmarouli commented Dec 4, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fix multi fields in downsampling #138869

Fix multi fields in downsampling #138869

Uh oh!

Conversation

gmarouli commented Dec 2, 2025

Uh oh!

elasticsearchmachine commented Dec 2, 2025

Uh oh!

elasticsearchmachine commented Dec 2, 2025

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Dec 4, 2025

💔 Backport failed

Uh oh!

gmarouli commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gmarouli commented Dec 4, 2025 •

edited

Loading