Make index creation more user-friendly

[s1monw]

Today when we create an index we return immediately after executing sanity checks and adding metadata to the cluster-state. Yet, we don't wait for any kind of allocations etc. such that an index can be created with more replicas than nodes in the cluster and once it's closed it can't be reopened since reopening an index requires a quorum of the replicas for each shard. If such an index is reopened the shards that have no quorum / not enough replicas are found in the cluster will just not be allocated at all.

Unfortunately not even waiting for yellow will help here since it means waiting for the primary to be allocated which might not be enough in the case of #replicas > 1.

There are a couple of things we can do here to improve the situation:

• Add another wait to the cluster health to wait for quorum
• By default wait for quorum for the index when an index is created
• By default reject closing an index if less than the quorum of shards is allocated

[jpountz]

+1 This will especially make integration testing less trappy!

[bsandvik]

+1 this would help us out a lot

[Mpdreamz]

+1

[synhershko]

Reject sealing an index if no quorum, too?

[nik9000]

Reject sealing an index if no quorum, too?

I don't think that's a good idea. The worst thing that happens if you do a synced_flush when one of the shards isn't around is that it doesn't effect them. Meaning that shard can't be restored quickly. There isn't really anything that can be done to get that shard to restore quickly regardless because its already offline. I wouldn't want to prevent speeding recovery on the other copies of the shard.

And I think its safe because if, by some nasty turn of events, one of the down shards ends up coming back and being the master shard then the synced flush won't have any effect because it won't be on the master.

[s1monw]

@kimchy promised to work on this today

[clintongormley]

This wait-for-quorum should be extended to the open-index API, eg see #12987

[ywelsch]

@s1monw In v3.0 we allocate primary shard based on allocation IDs #14739. This means that reopening an index only requires 1 good copy (not a quorum anymore). With allocation ids, is this a usability issue now (and not resiliency-related anymore)?

[bleskes]

@clintongormley and I discussed this again and came up with a plan.

There are two issues still left with index creation- the first is that an index creation move the cluster health status to RED even if everything is OK. The second is that when an index creation successfully returns, there is no guarantee that a follow up indexing operation will not have to wait (maybe not so bad) and that all operations on that index (for example _analyze, that needs a shard copy) will succeed.

• Index Creation puts status to RED:
  Currently RED means that a shard has a non-active primary. The idea is to change the semantics to exclude primary shards that were never successfully assigned and also didn't experience any shard failure during assignment. If the allocation deciders block the allocation of primary (not throttle it) we will treat it as a failure and make the shard red as well. In other cases the shard is YELLOW.
• Index Creation should wait or enough shard copies to reach started
  The index creation call should add the index to the cluster metadata and wait for enough shard copies (typically only primaries, but this should be based on action.write_consistency) to be started. It will return immediately if the status of one of those shards becomes RED (allocation failure or it can't be assigned to any node), reporting the failure.

[nik9000]

So now that we're testing snippets in the documentation this user-unfriendliness is leaking into the documentation. Which makes the issue pretty obvious. So I'd be pretty excited to have this fixed/make time to do it myself.

[abeyad]

Closed by #19450

[robinst]

Just to check, this was released with 5.0.0, right?

[ywelsch]

yes, the version label is on the linked PR #19450 that was used to close this issue.