Take Shard data path into account in DiskThresholdDecider #13195
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -360,6 +365,7 @@ public void onResponse(IndicesStatsResponse indicesStatsResponse) { | |||
| newShardSizes.put(sid, size); | |||
| } | |||
| shardSizes = Collections.unmodifiableMap(newShardSizes); | |||
| shardRoutingToDataPath = newShardRoutingToDataPath; | |||
There was a problem hiding this comment.
Do we want to wrap this one in Collections.unmodifiableMap(...) as well?
There was a problem hiding this comment.
yeah sure good catch!
|
Left just a few comments |
|
@dakrone pushed a new commit |
|
LGTM |
s1monw
force-pushed
the
shard_path_allocation_decisions
branch
from
92a20db
to
1025e04
Compare
|
@dakrone thx I will give others a chance to look at it and push on monday. |
s1monw
force-pushed
the
shard_path_allocation_decisions
branch
from
1025e04
to
8e0357c
Compare
|
LGTM2 . I do wonder (unrelated to this change), if things will be simpler if we keep the disk usage per path on the cluster info (rather than calc the least free space path) and make the can remain logic check each shard w.r.t it's own path. |
The path that a shard is allocated on is not taken into account when we decide to move a shard away from a node because it passed a watermark. Even worse we potentially moved away (relocated) a shard that was not even allocated on that disk but on another on the node in question. This commit adds a ShardRouting -> dataPath mapping to ClusterInfo that allows to identify on which disk the shards are allocated on. Relates to elastic#13106
s1monw
force-pushed
the
shard_path_allocation_decisions
branch
from
8e0357c
to
a17d750
Compare
lcawl
added
:Distributed/Distributed
clintongormley
added
:Distributed/Allocation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The path that a shard is allocated on is not taken into account when
we decide to move a shard away from a node because it passed a watermark.
Even worse we potentially moved away (relocated) a shard that was not even
allocated on that disk but on another on the node in question. This commit
adds a ShardRouting -> dataPath mapping to ClusterInfo that allows to identify
on which disk the shards are allocated on.
Relates to #13106