Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add merge filters before building cluster filters (#55815) #91582

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

wdongyu
Copy link
Contributor

@wdongyu wdongyu commented Nov 15, 2022

Currently allocation filtering works only on delta settings. For example:

  1. we first set cluster.routing.allocation.exclude.zone to us-east-1a to allocate shards to all nodes not in zone us-east-1a.
  2. Then we update cluster.routing.allocation.exclude.rack to rack-id. At this point, shards will relocate to nodes not in rack rack-id, ignoring the zone tag which we first set.

The reason for this bug is, when updating cluster settings for cluster require/include/exclude filters, AffixMapUpdateConsumer is only passing setting deltas, then FilterAllocationDecider rebuilds new filters using the settings deltas.

To fix this, we can merge the incoming include/exclude/require delta filters with the existing ones, and use the new merge filters to build cluster include/exclude/require filters. In this way, cluster require/include/exclude filters will include all cluster settings we have set instead of the delta ones.

ISSUE: #55815

@elasticsearchmachine elasticsearchmachine added v8.6.0 needs:triage Requires assignment of a team area label external-contributor Pull request authored by a developer outside the Elasticsearch team labels Nov 15, 2022
@kingherc kingherc added v8.7.0 and removed v8.6.0 labels Nov 16, 2022
@nik9000 nik9000 added the :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) label Nov 16, 2022
@elasticsearchmachine elasticsearchmachine added Team:Distributed Meta label for distributed team and removed needs:triage Requires assignment of a team area label labels Nov 16, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@wdongyu
Copy link
Contributor Author

wdongyu commented Jan 20, 2023

Pinging @nik9000 @kingherc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) external-contributor Pull request authored by a developer outside the Elasticsearch team Team:Distributed Meta label for distributed team v8.15.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants