-
Notifications
You must be signed in to change notification settings - Fork 24.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Autoshard data streams on rollover (#106076)
This enhances our rollover logic to use the data stream autosharding
service to retrieve an autosharding recommendation. If the
recommendation is an INCREASE_SHARDS or an
COOLDOWN_PREVENTED_INCREASE_SHARDS we'll create a rollover condition
that'll capture this recommendation, such that rollover will be
triggered in ourder to increase the number of shards even if other
"regular" conditions are not met (or in the case where cooldown
prevented rollover, display the information as to why in the rollover
response).
All other recommednations are passed to the `MetadataRolloverService`
that'll do the needful to ensure the new write index of the data stream
receives the correct number of shards.
Note that a DECREASE_SHARDS recommendation will reduce the number of
shards for a data stream when one of the other "regular" rollover
conditions match. It will not trigger a rollover itself, only the
INCREASE_SHARDS recommendation will.
Some notes on the `NOT_APPLICABLE` recommendation: N/A results are
switching back a data stream to the sharding configured in the index
template. A data stream can be using auto sharding and later be excluded
from the functionality using the `data_streams.auto_sharding.excludes`
setting. After a data stream is excluded it needs to start using the
number of shards configured in the backing index template.
The new autosharding_condition will look like this in the rollover
response:
```
"acknowledged": true,
"shards_acknowledged": true,
"old_index": ".ds-logs-nginx-2024.03.13-000003",
"new_index": ".ds-logs-nginx-2024.03.13-000004",
"rolled_over": true,
"dry_run": false,
"lazy": false,
"conditions": {
"[optimal_shard_count : 3]": true
}
```
and like so in the `met_conditions` field, part of rollover info in the
cluster state :
```
"rollover_info" : {
"logs-nginx" : {
"met_conditions" : {
"max_docs" : 20000000,
"optimal_shard_count" : 3
},
"time" : 1710421491138
}
},
```- Loading branch information
Showing
28 changed files
with
1,706 additions
and
48 deletions.
There are no files selected for viewing
615 changes: 615 additions & 0 deletions
615
.../src/internalClusterTest/java/org/elasticsearch/datastreams/DataStreamAutoshardingIT.java
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
70 changes: 70 additions & 0 deletions
70
...main/java/org/elasticsearch/action/admin/indices/rollover/OptimalShardCountCondition.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0 and the Server Side Public License, v 1; you may not use this file except | ||
| * in compliance with, at your election, the Elastic License 2.0 or the Server | ||
| * Side Public License, v 1. | ||
| */ | ||
|
|
||
| package org.elasticsearch.action.admin.indices.rollover; | ||
|
|
||
| import org.elasticsearch.TransportVersion; | ||
| import org.elasticsearch.TransportVersions; | ||
| import org.elasticsearch.common.io.stream.StreamInput; | ||
| import org.elasticsearch.common.io.stream.StreamOutput; | ||
| import org.elasticsearch.xcontent.XContentBuilder; | ||
| import org.elasticsearch.xcontent.XContentParser; | ||
|
|
||
| import java.io.IOException; | ||
|
|
||
| /** | ||
| * Condition for automatically increasing the number of shards for a data stream. This indicates the optimum number of shards that was | ||
| * configured for the index abstraction as part of rollover. | ||
| * It's more of a marker condition, when present the condition is met, more than a condition we evaluate against stats. | ||
| */ | ||
| public class OptimalShardCountCondition extends Condition<Integer> { | ||
| public static final String NAME = "optimal_shard_count"; | ||
|
|
||
| public OptimalShardCountCondition(int optimalShards) { | ||
| super(NAME, Type.AUTOMATIC); | ||
| this.value = optimalShards; | ||
| } | ||
|
|
||
| public OptimalShardCountCondition(StreamInput in) throws IOException { | ||
| super(NAME, Type.AUTOMATIC); | ||
| this.value = in.readVInt(); | ||
| } | ||
|
|
||
| @Override | ||
| public Result evaluate(final Stats stats) { | ||
| return new Result(this, true); | ||
| } | ||
|
|
||
| @Override | ||
| public String getWriteableName() { | ||
| return NAME; | ||
| } | ||
|
|
||
| @Override | ||
| public void writeTo(StreamOutput out) throws IOException { | ||
| out.writeVInt(value); | ||
| } | ||
|
|
||
| @Override | ||
| public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { | ||
| return builder.field(NAME, value); | ||
| } | ||
|
|
||
| public static OptimalShardCountCondition fromXContent(XContentParser parser) throws IOException { | ||
| if (parser.nextToken() == XContentParser.Token.VALUE_NUMBER) { | ||
| return new OptimalShardCountCondition(parser.intValue()); | ||
| } else { | ||
| throw new IllegalArgumentException("invalid token when parsing " + NAME + " condition: " + parser.currentToken()); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| boolean includedInVersion(TransportVersion version) { | ||
| return version.onOrAfter(TransportVersions.AUTO_SHARDING_ROLLOVER_CONDITION); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.