Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shards_availability health indicator turns yellow when creating new index with replicas #99951

Open
idegtiarenko opened this issue Sep 27, 2023 · 1 comment · Fixed by #99995
Assignees
Labels

Comments

@idegtiarenko
Copy link
Contributor

idegtiarenko commented Sep 27, 2023

Description

Unassigned shards caused by create index (or similar operations) are ignored and considered available in order to keep the shards_availability health indicator green:

boolean isNew = isUnassignedDueToNewInitialization(routing);
boolean isRestarting = isUnassignedDueToTimelyRestart(routing, shutdowns);
available &= routing.active() || isRestarting || isNew;

However the isUnassignedDueToNewInitialization check itself is only implemented for primaries:

private static boolean isUnassignedDueToNewInitialization(ShardRouting routing) {
return routing.primary() && routing.active() == false && getInactivePrimaryHealth(routing) == ClusterHealthStatus.YELLOW;
}

as a result:

  • when creating a new index without replicas the health indicator remains green
  • when creating the same index with replicas the health indicator turns yellow until replicas are initialized

The check could be extended for replicas like following (to verify the status of corresponding primary):

    private static boolean isUnassignedDueToNewInitialization(ShardRouting routing, ClusterState state) {
        if (routing.active() == false) {
            return false;
        }
        if (routing.primary()) {
            return getInactivePrimaryHealth(routing) == ClusterHealthStatus.YELLOW;
        } else {
            var primary = state.routingTable().shardRoutingTable(routing.shardId()).primaryShard();
            return getInactivePrimaryHealth(primary) == ClusterHealthStatus.YELLOW;
        }
    }

however overall indicator would still turn yellow when primary is active and replica is still initializing.

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@gmarouli gmarouli self-assigned this Sep 28, 2023
elasticsearchmachine pushed a commit that referenced this issue Sep 28, 2023
When we have a new initialisation of a primary shard we consider the
this indicator `green` based on the idea that this are unexceptional
events in the cluster's lifecycle. However, when we have a replica of
this shard that is inactive we turn to `yellow`. 

With this PR, we change this behaviour to signal that if the primary is
inactive and that is `green`, then an inactive replica of this shard is
also `green`.

Fixes #99951
@idegtiarenko idegtiarenko reopened this Sep 28, 2023
piergm pushed a commit to piergm/elasticsearch that referenced this issue Oct 2, 2023
When we have a new initialisation of a primary shard we consider the
this indicator `green` based on the idea that this are unexceptional
events in the cluster's lifecycle. However, when we have a replica of
this shard that is inactive we turn to `yellow`. 

With this PR, we change this behaviour to signal that if the primary is
inactive and that is `green`, then an inactive replica of this shard is
also `green`.

Fixes elastic#99951
dakrone added a commit to dakrone/elasticsearch that referenced this issue Apr 22, 2024
…quely

Related to the work in elastic#101638 this changes the way we calculate whether all replicas are unassigned
when corresponding to newly created primaries. While this doesn't affect anything in Stateful ES on
its own, it's a building-block used for object-store-based ES (Serverless).

Semi-related to elastic#99951, though it does not solve (and does not strive to solve) that issue.
dakrone added a commit to dakrone/elasticsearch that referenced this issue Apr 23, 2024
…quely

Related to the work in elastic#101638 this changes the way we calculate whether all replicas are unassigned
when corresponding to newly created primaries. While this doesn't affect anything in Stateful ES on
its own, it's a building-block used for object-store-based ES (Serverless).

Semi-related to elastic#99951, though it does not solve (and does not strive to solve) that issue.
elasticsearchmachine pushed a commit that referenced this issue Apr 24, 2024
…quely (#107794)

Related to the work in #101638 this changes the way we calculate whether
all replicas are unassigned when corresponding to newly created
primaries. While this doesn't affect anything in Stateful ES on its own,
it's a building-block used for object-store-based ES (Serverless).

Semi-related to #99951, though it does not solve (and does not strive to
solve) that issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants