From 3fb873b7f6876e562138657e0c5242aa587b02db Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 30 Sep 2025 10:16:18 +0200 Subject: [PATCH 1/7] Add index setting that disables the `index.dimensions` based routing and _tsid creation strategy --- .../index-settings/serverless.md | 19 ++++++----- .../index-settings/time-series.md | 25 +++++++++++++- .../datastreams/TSDBIndexingIT.java | 25 +++++++++++--- .../DataStreamIndexSettingsProvider.java | 4 ++- .../DataStreamIndexSettingsProviderTests.java | 33 ++++++++++++------- .../cluster/metadata/IndexMetadata.java | 17 +++++++++- .../common/settings/IndexScopedSettings.java | 1 + 7 files changed, 96 insertions(+), 28 deletions(-) diff --git a/docs/reference/elasticsearch/index-settings/serverless.md b/docs/reference/elasticsearch/index-settings/serverless.md index 1175bcc84865f..d54ec3fe606a8 100644 --- a/docs/reference/elasticsearch/index-settings/serverless.md +++ b/docs/reference/elasticsearch/index-settings/serverless.md @@ -13,13 +13,13 @@ This page lists the {{es}} index settings available in {{serverless-full}} proje ### General settings * [`index.codec`](./index-modules.md#index-codec) -* [`index.default_pipeline`](./index-modules.md#index-default-pipeline) +* [`index.default_pipeline`](./index-modules.md#index-default-pipeline) * [`index.dense_vector.hnsw_filter_heuristic`](./index-modules.md#index-dense-vector-hnsw-filter-heuristic) -* [`index.final_pipeline`](./index-modules.md#index-final-pipeline) +* [`index.final_pipeline`](./index-modules.md#index-final-pipeline) * [`index.hidden`](./index-modules.md#index-hidden) * [`index.mode`](./index-modules.md#index-mode-setting) -* [`index.query.default_field`](./index-modules.md#index-query-default-field) -* [`index.refresh_interval`](./index-modules.md#index-refresh-interval-setting) +* [`index.query.default_field`](./index-modules.md#index-query-default-field) +* [`index.refresh_interval`](./index-modules.md#index-refresh-interval-setting) ### Index sorting settings @@ -30,10 +30,10 @@ This page lists the {{es}} index settings available in {{serverless-full}} proje ### Index blocks settings -* [`index.blocks.read_only`](./index-block.md#index-blocks-read-only) -* [`index.blocks.read`](./index-block.md#index-blocks-read) -* [`index.blocks.write`](./index-block.md#index-blocks-write) -* [`index.blocks.metadata`](./index-block.md#index-blocks-metadata) +* [`index.blocks.read_only`](./index-block.md#index-blocks-read-only) +* [`index.blocks.read`](./index-block.md#index-blocks-read) +* [`index.blocks.write`](./index-block.md#index-blocks-write) +* [`index.blocks.metadata`](./index-block.md#index-blocks-metadata) ### Field and mapping related settings @@ -56,8 +56,9 @@ This page lists the {{es}} index settings available in {{serverless-full}} proje * [`index.look_ahead_time`](./time-series.md#index-look-ahead-time) * [`index.look_back_time`](./time-series.md#index-look-back-time) * [`index.routing_path`](./time-series.md#index-routing-path) +* [`index.index_dimensions_tsid_strategy_enabled`](./time-series.md#index-dimensions-tsid-strategy-enabled) ### Similarity and analyzers * [`index.similarity.*`](../mapping-reference/similarity.md) -* [`index.analysis.*`](../mapping-reference/analyzer.md) \ No newline at end of file +* [`index.analysis.*`](../mapping-reference/analyzer.md) diff --git a/docs/reference/elasticsearch/index-settings/time-series.md b/docs/reference/elasticsearch/index-settings/time-series.md index 336d7e7730b65..65573dcd9e6e0 100644 --- a/docs/reference/elasticsearch/index-settings/time-series.md +++ b/docs/reference/elasticsearch/index-settings/time-series.md @@ -44,7 +44,30 @@ $$$index-look-back-time$$$ : (Static, [time units](/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) Interval used to calculate the `index.time_series.start_time` for a TSDS’s first backing index when a tsdb data stream is created. Defaults to `2h` (2 hours). Accepts `1m` (one minute) to `7d` (seven days). Only indices with an `index.mode` of `time_series` support this setting. For more information, refer to [Look-back time](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#tsds-look-back-time). $$$index-routing-path$$$ `index.routing_path` {applies_to}`serverless: all` -: (Static, string or array of strings) Plain `keyword` fields used to route documents in a TSDS to index shards. Supports wildcards (`*`). Only indices with an `index.mode` of `time_series` support this setting. Defaults to an empty list, except for data streams then defaults to the list of [dimension fields](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension) with a `time_series_dimension` value of `true` defined in your component and index templates. For more information, refer to [Dimension-based routing](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#dimension-based-routing). +: (Static, string or array of strings) Time series dimension fields used to route documents in a TSDS to index shards. Supports wildcards (`*`). Only indices with an `index.mode` of `time_series` support this setting. Defaults to an empty list, except for data streams then defaults to the list of [dimension fields](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension) with a `time_series_dimension` value of `true` defined in your component and index templates. For more information, refer to [Dimension-based routing](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#dimension-based-routing). + +$$$index-dimensions-tsid-strategy-enabled$$$ + +`index.index_dimensions_tsid_strategy_enabled` {applies_to}`serverless: all` +: (Static, boolean) Controls if the `_tsid` can be created using the `index.dimensions` index setting. +This is an internal setting that will be automatically populated and updated for eligible time series data streams and is not user-configurable. +This strategy offers an improved ingestion performance that avoids processing dimensions multiple times for the purposes of shard routing and creating the `_tsid`. +When used, `index.routing_path` will not be set and shard routing uses the full `_tsid`, +which can help to avoid shard hot-spotting. + +: If set to `false`, +or `index.routing_path` is configured manually, +or in case the index isn't eligible (see below), +shard routing will be based on the `index.routing_path` instead. +The `_tsid` will then be created during document parsing rather than during shard routing. + +: Defaults to `true`. + +: This optimized `_tsid` creation strategy is only available for data streams and if there are no dynamic templates that set `time_series_dimension: true`. +Trying to add such a dynamic template to existing backing indices after the fact will fail the update mapping request with the following message: +```text +Cannot add dynamic templates that define dimension fields on an existing index with index.dimensions. Please change the index template and roll over the data stream instead of modifying the mappings of the backing indices. +``` $$$index-mapping-dimension-fields-limit$$$ diff --git a/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java b/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java index c64421bd3b8a5..28005ac457834 100644 --- a/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java +++ b/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java @@ -333,6 +333,7 @@ public void testTsdbTemplatesNoKeywordFieldType() throws Exception { Settings.builder() .put("index.mode", "time_series") .put("index.routing_path", randomBoolean() ? null : "metricset") + .put("index.index_dimensions_tsid_strategy_enabled", usually()) .build(), new CompressedXContent(mappingTemplate), null @@ -640,12 +641,16 @@ public void testReindexing() throws Exception { public void testAddDimensionToMapping() throws Exception { String dataStreamName = "my-ds"; var putTemplateRequest = new TransportPutComposableIndexTemplateAction.Request("id"); + boolean indexDimensionsTsidStrategyEnabled = randomBoolean(); putTemplateRequest.indexTemplate( ComposableIndexTemplate.builder() .indexPatterns(List.of(dataStreamName)) .template( new Template( - Settings.builder().put("index.mode", "time_series").build(), + Settings.builder() + .put("index.mode", "time_series") + .put("index.index_dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabled) + .build(), new CompressedXContent(MAPPING_TEMPLATE), null ) @@ -662,8 +667,13 @@ public void testAddDimensionToMapping() throws Exception { "my-ds" ); assertAcked(client().execute(CreateDataStreamAction.INSTANCE, createDsRequest)); - assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), equalTo(List.of("metricset"))); - assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), empty()); + if (indexDimensionsTsidStrategyEnabled) { + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), equalTo(List.of("metricset"))); + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), empty()); + } else { + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), empty()); + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), equalTo(List.of("metricset"))); + } // put mapping with k8s.pod.uid as another time series dimension var putMappingRequest = new PutMappingRequest(dataStreamName).source(""" @@ -677,8 +687,13 @@ public void testAddDimensionToMapping() throws Exception { } """, XContentType.JSON); assertAcked(client().execute(TransportPutMappingAction.TYPE, putMappingRequest).actionGet()); - assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), containsInAnyOrder("metricset", "k8s.pod.name")); - assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), empty()); + if (indexDimensionsTsidStrategyEnabled) { + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), containsInAnyOrder("metricset", "k8s.pod.name")); + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), empty()); + } else { + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), empty()); + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), equalTo(List.of("metricset"))); + } // put dynamic template defining time series dimensions // we don't support index.dimensions in that case diff --git a/modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProvider.java b/modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProvider.java index 2da10211d54ed..2f3fc8e3cb66c 100644 --- a/modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProvider.java +++ b/modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProvider.java @@ -130,7 +130,9 @@ public void provideAdditionalSettings( dimensions ); if (dimensions.isEmpty() == false) { - if (matchesAllDimensions && indexVersion.onOrAfter(IndexVersions.TSID_CREATED_DURING_ROUTING)) { + if (matchesAllDimensions + && IndexMetadata.INDEX_DIMENSIONS_TSID_STRATEGY_ENABLED.get(indexTemplateAndCreateRequestSettings) + && indexVersion.onOrAfter(IndexVersions.TSID_CREATED_DURING_ROUTING)) { // Only set index.dimensions if the paths in the dimensions list match all potential dimension fields. // This is not the case e.g. if a dynamic template matches by match_mapping_type instead of path_match additionalSettings.putList(INDEX_DIMENSIONS.getKey(), dimensions); diff --git a/modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProviderTests.java b/modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProviderTests.java index cb636b69c28fe..da4a9c5b2193e 100644 --- a/modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProviderTests.java +++ b/modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProviderTests.java @@ -49,7 +49,8 @@ public class DataStreamIndexSettingsProviderTests extends ESTestCase { private static final TimeValue DEFAULT_LOOK_AHEAD_TIME = TimeValue.timeValueMinutes(30); // default DataStreamIndexSettingsProvider provider; - private boolean indexDimensionsTsidOptimizationEnabled; + private boolean indexDimensionsTsidStrategyEnabledSetting; + private boolean expectedIndexDimensionsTsidOptimizationEnabled; private IndexVersion indexVersion; @Before @@ -60,7 +61,9 @@ public void setup() { indexVersion = randomBoolean() ? IndexVersionUtils.randomPreviousCompatibleVersion(random(), IndexVersions.TSID_CREATED_DURING_ROUTING) : IndexVersionUtils.randomVersionBetween(random(), IndexVersions.TSID_CREATED_DURING_ROUTING, IndexVersion.current()); - indexDimensionsTsidOptimizationEnabled = indexVersion.onOrAfter(IndexVersions.TSID_CREATED_DURING_ROUTING); + indexDimensionsTsidStrategyEnabledSetting = usually(); + expectedIndexDimensionsTsidOptimizationEnabled = indexDimensionsTsidStrategyEnabledSetting + && indexVersion.onOrAfter(IndexVersions.TSID_CREATED_DURING_ROUTING); } public void testGetAdditionalIndexSettings() throws Exception { @@ -114,12 +117,15 @@ public void testGetAdditionalIndexSettings() throws Exception { Settings result = additionalSettings.build(); // The index.time_series.end_time setting requires index.mode to be set to time_series adding it here so that we read this setting: // (in production the index.mode setting is usually provided in an index or component template) - result = builder().put(result).put("index.mode", "time_series").build(); - assertThat(result.size(), equalTo(4)); + result = builder().put(result) + .put("index.mode", "time_series") + .put("index.index_dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) + .build(); + assertThat(result.size(), equalTo(5)); assertThat(IndexSettings.MODE.get(result), equalTo(IndexMode.TIME_SERIES)); assertThat(IndexSettings.TIME_SERIES_START_TIME.get(result), equalTo(now.minusMillis(DEFAULT_LOOK_BACK_TIME.getMillis()))); assertThat(IndexSettings.TIME_SERIES_END_TIME.get(result), equalTo(now.plusMillis(DEFAULT_LOOK_AHEAD_TIME.getMillis()))); - if (indexDimensionsTsidOptimizationEnabled) { + if (expectedIndexDimensionsTsidOptimizationEnabled) { assertThat(IndexMetadata.INDEX_DIMENSIONS.get(result), containsInAnyOrder("field3", "field4", "field5", "field6")); assertThat(IndexMetadata.INDEX_ROUTING_PATH.get(result), empty()); } else { @@ -243,12 +249,15 @@ public void testGetAdditionalIndexSettingsMappingsMerging() throws Exception { Settings result = additionalSettings.build(); // The index.time_series.end_time setting requires index.mode to be set to time_series adding it here so that we read this setting: // (in production the index.mode setting is usually provided in an index or component template) - result = builder().put(result).put("index.mode", "time_series").build(); - assertThat(result.size(), equalTo(4)); + result = builder().put(result) + .put("index.mode", "time_series") + .put("index.index_dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) + .build(); + assertThat(result.size(), equalTo(5)); assertThat(IndexSettings.MODE.get(result), equalTo(IndexMode.TIME_SERIES)); assertThat(IndexSettings.TIME_SERIES_START_TIME.get(result), equalTo(now.minusMillis(DEFAULT_LOOK_BACK_TIME.getMillis()))); assertThat(IndexSettings.TIME_SERIES_END_TIME.get(result), equalTo(now.plusMillis(DEFAULT_LOOK_AHEAD_TIME.getMillis()))); - if (indexDimensionsTsidOptimizationEnabled) { + if (expectedIndexDimensionsTsidOptimizationEnabled) { assertThat(IndexMetadata.INDEX_DIMENSIONS.get(result), containsInAnyOrder("field1", "field3")); assertThat(IndexMetadata.INDEX_ROUTING_PATH.get(result), empty()); } else { @@ -719,7 +728,7 @@ public void testGenerateNonDimensionDynamicTemplate() throws Exception { assertThat(IndexSettings.MODE.get(result), equalTo(IndexMode.TIME_SERIES)); assertThat(IndexSettings.TIME_SERIES_START_TIME.get(result), equalTo(now.minusMillis(DEFAULT_LOOK_BACK_TIME.getMillis()))); assertThat(IndexSettings.TIME_SERIES_END_TIME.get(result), equalTo(now.plusMillis(DEFAULT_LOOK_AHEAD_TIME.getMillis()))); - if (indexDimensionsTsidOptimizationEnabled) { + if (expectedIndexDimensionsTsidOptimizationEnabled) { assertThat(IndexMetadata.INDEX_DIMENSIONS.get(result), containsInAnyOrder("host.id")); assertThat(IndexMetadata.INDEX_ROUTING_PATH.get(result), empty()); } else { @@ -807,7 +816,7 @@ public void testGenerateRoutingPathFromPassThroughObject() throws Exception { assertThat(IndexSettings.MODE.get(result), equalTo(IndexMode.TIME_SERIES)); assertThat(IndexSettings.TIME_SERIES_START_TIME.get(result), equalTo(now.minusMillis(DEFAULT_LOOK_BACK_TIME.getMillis()))); assertThat(IndexSettings.TIME_SERIES_END_TIME.get(result), equalTo(now.plusMillis(DEFAULT_LOOK_AHEAD_TIME.getMillis()))); - if (indexDimensionsTsidOptimizationEnabled) { + if (expectedIndexDimensionsTsidOptimizationEnabled) { assertThat(IndexMetadata.INDEX_DIMENSIONS.get(result), containsInAnyOrder("labels.*")); assertThat(IndexMetadata.INDEX_ROUTING_PATH.get(result), empty()); } else { @@ -971,7 +980,9 @@ public void testAddDynamicTemplate() throws Exception { private Settings generateTsdbSettings(String mapping, Instant now) throws IOException { ProjectMetadata projectMetadata = emptyProject(); String dataStreamName = "logs-app1"; - Settings settings = Settings.EMPTY; + Settings settings = Settings.builder() + .put("index.index_dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) + .build(); Settings.Builder additionalSettings = builder(); provider.provideAdditionalSettings( diff --git a/server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java b/server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java index 25edec3e855e4..21342df54ce2a 100644 --- a/server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java +++ b/server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java @@ -534,7 +534,22 @@ public Iterator> settings() { "index.dimensions", Setting.Property.IndexScope, Property.Dynamic, - Property.PrivateIndex + Property.PrivateIndex, + Property.ServerlessPublic + ); + + /** + * Allows to disable the {@link #INDEX_DIMENSIONS}-based tsid creation strategy on a per-index basis. + * This can help to mitigate potential issues with that strategy. + * For example, when using this strategy, + * it's not allowed to add a dynamic template that defines dimension fields to existing backing indices of a time series data stream. + */ + public static final Setting INDEX_DIMENSIONS_TSID_STRATEGY_ENABLED = Setting.boolSetting( + "index.index_dimensions_tsid_strategy_enabled", + true, + Setting.Property.IndexScope, + Property.Final, + Property.ServerlessPublic ); /** diff --git a/server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java b/server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java index a0a28e9322956..d4fef4e9bb489 100644 --- a/server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java +++ b/server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java @@ -224,6 +224,7 @@ public final class IndexScopedSettings extends AbstractScopedSettings { IndexSettings.MODE, IndexMetadata.INDEX_ROUTING_PATH, IndexMetadata.INDEX_DIMENSIONS, + IndexMetadata.INDEX_DIMENSIONS_TSID_STRATEGY_ENABLED, IndexSettings.TIME_SERIES_START_TIME, IndexSettings.TIME_SERIES_END_TIME, IndexSettings.SEQ_NO_INDEX_OPTIONS_SETTING, From 04f857e205ca501a8c58dbacd6cda942cd8b3edf Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 30 Sep 2025 10:19:27 +0200 Subject: [PATCH 2/7] Update docs/changelog/135673.yaml --- docs/changelog/135673.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 docs/changelog/135673.yaml diff --git a/docs/changelog/135673.yaml b/docs/changelog/135673.yaml new file mode 100644 index 0000000000000..b4607e852a826 --- /dev/null +++ b/docs/changelog/135673.yaml @@ -0,0 +1,6 @@ +pr: 135673 +summary: Add index setting that disables the `index.dimensions` based routing and + `_tsid` creation strategy +area: TSDB +type: enhancement +issues: [] From d985fd3ed2caf2f5f999abd7d150d21653188e72 Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 30 Sep 2025 11:07:02 +0200 Subject: [PATCH 3/7] Fix test --- .../datastreams/TSDBIndexingIT.java | 20 ++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java b/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java index 28005ac457834..a6751597052bf 100644 --- a/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java +++ b/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java @@ -713,13 +713,19 @@ public void testAddDimensionToMapping() throws Exception { } """, XContentType.JSON); ActionFuture putMappingFuture = client().execute(TransportPutMappingAction.TYPE, putMappingRequest); - IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, putMappingFuture::actionGet); - assertThat( - exception.getMessage(), - containsString("Cannot add dynamic templates that define dimension fields on an existing index with index.dimensions") - ); - assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), containsInAnyOrder("metricset", "k8s.pod.name")); - assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), empty()); + if (indexDimensionsTsidStrategyEnabled) { + IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, putMappingFuture::actionGet); + assertThat( + exception.getMessage(), + containsString("Cannot add dynamic templates that define dimension fields on an existing index with index.dimensions") + ); + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), containsInAnyOrder("metricset", "k8s.pod.name")); + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), empty()); + } else { + assertAcked(putMappingFuture.actionGet()); + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_DIMENSIONS), empty()); + assertThat(getSetting(dataStreamName, IndexMetadata.INDEX_ROUTING_PATH), equalTo(List.of("metricset"))); + } indexWithPodNames(dataStreamName, Instant.now(), Map.of(), "dog", "cat"); } From 663a763a1a57444a8b370a0d3bc51205f8108238 Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 30 Sep 2025 11:50:57 +0200 Subject: [PATCH 4/7] Add applies_to for stack 9.2 --- docs/reference/elasticsearch/index-settings/time-series.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/elasticsearch/index-settings/time-series.md b/docs/reference/elasticsearch/index-settings/time-series.md index 65573dcd9e6e0..0909b2fe4696b 100644 --- a/docs/reference/elasticsearch/index-settings/time-series.md +++ b/docs/reference/elasticsearch/index-settings/time-series.md @@ -48,7 +48,7 @@ $$$index-routing-path$$$ `index.routing_path` {applies_to}`serverless: all` $$$index-dimensions-tsid-strategy-enabled$$$ -`index.index_dimensions_tsid_strategy_enabled` {applies_to}`serverless: all` +`index.index_dimensions_tsid_strategy_enabled` {applies_to}`stack: ga 9.2` {applies_to}`serverless: all` : (Static, boolean) Controls if the `_tsid` can be created using the `index.dimensions` index setting. This is an internal setting that will be automatically populated and updated for eligible time series data streams and is not user-configurable. This strategy offers an improved ingestion performance that avoids processing dimensions multiple times for the purposes of shard routing and creating the `_tsid`. From 0f8c9b9c8b577a9ba44d579cb008e6b9a3e8f01c Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 30 Sep 2025 13:18:27 +0200 Subject: [PATCH 5/7] Skip duplicate index prefix --- docs/reference/elasticsearch/index-settings/serverless.md | 2 +- docs/reference/elasticsearch/index-settings/time-series.md | 2 +- .../java/org/elasticsearch/datastreams/TSDBIndexingIT.java | 4 ++-- .../datastreams/DataStreamIndexSettingsProviderTests.java | 6 +++--- .../org/elasticsearch/cluster/metadata/IndexMetadata.java | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/reference/elasticsearch/index-settings/serverless.md b/docs/reference/elasticsearch/index-settings/serverless.md index d54ec3fe606a8..4f5c876771241 100644 --- a/docs/reference/elasticsearch/index-settings/serverless.md +++ b/docs/reference/elasticsearch/index-settings/serverless.md @@ -56,7 +56,7 @@ This page lists the {{es}} index settings available in {{serverless-full}} proje * [`index.look_ahead_time`](./time-series.md#index-look-ahead-time) * [`index.look_back_time`](./time-series.md#index-look-back-time) * [`index.routing_path`](./time-series.md#index-routing-path) -* [`index.index_dimensions_tsid_strategy_enabled`](./time-series.md#index-dimensions-tsid-strategy-enabled) +* [`index.dimensions_tsid_strategy_enabled`](./time-series.md#index-dimensions-tsid-strategy-enabled) ### Similarity and analyzers diff --git a/docs/reference/elasticsearch/index-settings/time-series.md b/docs/reference/elasticsearch/index-settings/time-series.md index 0909b2fe4696b..1ac135e256ad4 100644 --- a/docs/reference/elasticsearch/index-settings/time-series.md +++ b/docs/reference/elasticsearch/index-settings/time-series.md @@ -48,7 +48,7 @@ $$$index-routing-path$$$ `index.routing_path` {applies_to}`serverless: all` $$$index-dimensions-tsid-strategy-enabled$$$ -`index.index_dimensions_tsid_strategy_enabled` {applies_to}`stack: ga 9.2` {applies_to}`serverless: all` +`index.dimensions_tsid_strategy_enabled` {applies_to}`stack: ga 9.2` {applies_to}`serverless: all` : (Static, boolean) Controls if the `_tsid` can be created using the `index.dimensions` index setting. This is an internal setting that will be automatically populated and updated for eligible time series data streams and is not user-configurable. This strategy offers an improved ingestion performance that avoids processing dimensions multiple times for the purposes of shard routing and creating the `_tsid`. diff --git a/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java b/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java index a6751597052bf..1eb4926470314 100644 --- a/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java +++ b/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java @@ -333,7 +333,7 @@ public void testTsdbTemplatesNoKeywordFieldType() throws Exception { Settings.builder() .put("index.mode", "time_series") .put("index.routing_path", randomBoolean() ? null : "metricset") - .put("index.index_dimensions_tsid_strategy_enabled", usually()) + .put("index.dimensions_tsid_strategy_enabled", usually()) .build(), new CompressedXContent(mappingTemplate), null @@ -649,7 +649,7 @@ public void testAddDimensionToMapping() throws Exception { new Template( Settings.builder() .put("index.mode", "time_series") - .put("index.index_dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabled) + .put("index.dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabled) .build(), new CompressedXContent(MAPPING_TEMPLATE), null diff --git a/modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProviderTests.java b/modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProviderTests.java index da4a9c5b2193e..0b0458905526d 100644 --- a/modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProviderTests.java +++ b/modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamIndexSettingsProviderTests.java @@ -119,7 +119,7 @@ public void testGetAdditionalIndexSettings() throws Exception { // (in production the index.mode setting is usually provided in an index or component template) result = builder().put(result) .put("index.mode", "time_series") - .put("index.index_dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) + .put("index.dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) .build(); assertThat(result.size(), equalTo(5)); assertThat(IndexSettings.MODE.get(result), equalTo(IndexMode.TIME_SERIES)); @@ -251,7 +251,7 @@ public void testGetAdditionalIndexSettingsMappingsMerging() throws Exception { // (in production the index.mode setting is usually provided in an index or component template) result = builder().put(result) .put("index.mode", "time_series") - .put("index.index_dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) + .put("index.dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) .build(); assertThat(result.size(), equalTo(5)); assertThat(IndexSettings.MODE.get(result), equalTo(IndexMode.TIME_SERIES)); @@ -981,7 +981,7 @@ private Settings generateTsdbSettings(String mapping, Instant now) throws IOExce ProjectMetadata projectMetadata = emptyProject(); String dataStreamName = "logs-app1"; Settings settings = Settings.builder() - .put("index.index_dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) + .put("index.dimensions_tsid_strategy_enabled", indexDimensionsTsidStrategyEnabledSetting) .build(); Settings.Builder additionalSettings = builder(); diff --git a/server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java b/server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java index 21342df54ce2a..3207f7b9f8b0b 100644 --- a/server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java +++ b/server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java @@ -545,7 +545,7 @@ public Iterator> settings() { * it's not allowed to add a dynamic template that defines dimension fields to existing backing indices of a time series data stream. */ public static final Setting INDEX_DIMENSIONS_TSID_STRATEGY_ENABLED = Setting.boolSetting( - "index.index_dimensions_tsid_strategy_enabled", + "index.dimensions_tsid_strategy_enabled", true, Setting.Property.IndexScope, Property.Final, From dbfd862d6d10721fc3f63750334ed85b17171e3b Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 30 Sep 2025 13:20:37 +0200 Subject: [PATCH 6/7] Use randomDouble() < 0.8 rather than usually --- .../java/org/elasticsearch/datastreams/TSDBIndexingIT.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java b/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java index 1eb4926470314..c22f79b11a597 100644 --- a/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java +++ b/modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBIndexingIT.java @@ -333,7 +333,7 @@ public void testTsdbTemplatesNoKeywordFieldType() throws Exception { Settings.builder() .put("index.mode", "time_series") .put("index.routing_path", randomBoolean() ? null : "metricset") - .put("index.dimensions_tsid_strategy_enabled", usually()) + .put("index.dimensions_tsid_strategy_enabled", randomDouble() < 0.8) .build(), new CompressedXContent(mappingTemplate), null From 5de7b8a4d300b6a6ac9f8e129c8fa18a4664b1f4 Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 30 Sep 2025 14:53:20 +0200 Subject: [PATCH 7/7] Apply review comments --- .../index-settings/time-series.md | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/reference/elasticsearch/index-settings/time-series.md b/docs/reference/elasticsearch/index-settings/time-series.md index 1ac135e256ad4..061889e731f9e 100644 --- a/docs/reference/elasticsearch/index-settings/time-series.md +++ b/docs/reference/elasticsearch/index-settings/time-series.md @@ -44,7 +44,20 @@ $$$index-look-back-time$$$ : (Static, [time units](/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) Interval used to calculate the `index.time_series.start_time` for a TSDS’s first backing index when a tsdb data stream is created. Defaults to `2h` (2 hours). Accepts `1m` (one minute) to `7d` (seven days). Only indices with an `index.mode` of `time_series` support this setting. For more information, refer to [Look-back time](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#tsds-look-back-time). $$$index-routing-path$$$ `index.routing_path` {applies_to}`serverless: all` -: (Static, string or array of strings) Time series dimension fields used to route documents in a TSDS to index shards. Supports wildcards (`*`). Only indices with an `index.mode` of `time_series` support this setting. Defaults to an empty list, except for data streams then defaults to the list of [dimension fields](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension) with a `time_series_dimension` value of `true` defined in your component and index templates. For more information, refer to [Dimension-based routing](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#dimension-based-routing). +: (Static, string or array of strings) Time series dimension fields used to route documents in a TSDS to index shards. +Supports wildcards (`*`). +Only indices with an `index.mode` of `time_series` support this setting. + +: Defaults value: +: Indices that are not part of a time series data stream have no default value and require the routing path to be defined explicitly. +If a time series data stream is used that is eligible for the `index.dimensions`-based routing (see [`index.dimensions_tsid_strategy_enabled`](#index-dimensions-tsid-strategy-enabled)), +the `index.routing_path` will be empty. +For time series data streams where the `index.dimensions`-based routing does not apply, +this defaults to the list of [dimension fields](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension) with a `time_series_dimension` value of `true` as defined in your component and index templates. + +: Manually setting a value disables the `index.dimensions`-based routing strategy (see [`index.dimensions_tsid_strategy_enabled`](#index-dimensions-tsid-strategy-enabled)). +For more information, refer to [Dimension-based routing](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#dimension-based-routing). + $$$index-dimensions-tsid-strategy-enabled$$$ @@ -59,15 +72,11 @@ which can help to avoid shard hot-spotting. or `index.routing_path` is configured manually, or in case the index isn't eligible (see below), shard routing will be based on the `index.routing_path` instead. -The `_tsid` will then be created during document parsing rather than during shard routing. : Defaults to `true`. : This optimized `_tsid` creation strategy is only available for data streams and if there are no dynamic templates that set `time_series_dimension: true`. -Trying to add such a dynamic template to existing backing indices after the fact will fail the update mapping request with the following message: -```text -Cannot add dynamic templates that define dimension fields on an existing index with index.dimensions. Please change the index template and roll over the data stream instead of modifying the mappings of the backing indices. -``` +Trying to add such a dynamic template to existing backing indices after the fact will fail the update mapping request and you will need to roll over the data stream instead. $$$index-mapping-dimension-fields-limit$$$