Allocate primary shard based on allocation IDs

[ywelsch]

Persist allocation IDs of active shards in cluster state and use them to recover correct shards upon cluster restart (i.e., we can recover with only one copy and we make sure we recover the right ones and not a stale copy left around).

Relates to #14671

Steps

• Persist allocation ID with shard state metadata on nodes (Persist allocation ID with shard state metadata on nodes #14831).
• Persist currently started allocation IDs to index metadata (Persist currently started allocation IDs to index metadata #14964).
• Add allocation IDs to TransportNodesListGatewayStartedShards action. Use the above to assign a primary shard on recovery. In particular, don't assign a stale copy (Allocate primary shards based on allocation IDs #15281).
• Extend reroute with an option to force assign a stale copy. This is only relevant for primaries (Extend reroute with an option to force assign stale primary shard copies #15708).
• Prioritize old primaries when choosing out existing and previously active allocationIds (Prefer nodes that previously held primary shard for primary shard allocation #16096).
• Persist shard state immediately and not only on shard activation. This will help solving a race condition on primary relocation where the master has activated the target shard (removing old copy from the active set list) but the cluster crashes before the target node persisted the shard state (Write shard state metadata as soon as shard is created / initializing #16625)
• Remove version from ShardRouting (Remove version in ShardRouting (now obsolete) #16243).