Add support for replicating closed indices

[tlrx]

This pull request adds the support for replicating closed indices.

Before this change, closed indexes were simply not replicated. It was therefore possible to close an index and then decommission a data node without knowing that this data node contained shards of the closed index, potentially leading to data loss. Shards of closed indices were not completely taken into account when balancing the shards within the cluster, or automatically replicated through shard copies, and they were not easily movable from node A to node B using APIs like Cluster Reroute without being fully reopened and closed again.

This pull request changes the logic executed when closing an index, so that its shards are not just removed and forgotten but are instead reinitialized and reallocated on data nodes using an engine implementation which does not allow searching or indexing, which has a low memory overhead (compared with searchable/indexable opened shards) and which allows shards to be recovered from peer or promoted as primaries when needed.

This new closing logic is built on top of the new Close Index API introduced in 6.7.0 (#37359). Some pre-closing sanity checks are executed on the shards before closing them, and closing an index on a 8.0 cluster will reinitialize the index shards and therefore impact the cluster health.

Some APIs have been adapted to make them work with indices whatever the state they are in:

• Cluster Health API
• Cluster Reroute API
• Cluster Allocation Explain API
• Recovery API
• Cat Indices
• Cat Shards
• Cat Health
• Cat Recovery

Some of these APIs support the expand_wilcards parameter that allows to retrieve information only for closed and/or open indices, like:

GET /_cluster/health?expand_wildcards=closed&level=shards
{
  "cluster_name" : "distribution_run",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0,
  "indices" : {
    "my-closed-index" : {
      "status" : "green",
      "number_of_shards" : 1,
      "number_of_replicas" : 0,
      "active_primary_shards" : 1,
      "active_shards" : 1,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 0,
      "shards" : {
        "0" : {
          "status" : "green",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 0
        }
      }
    }
  }
}

---

This pull request contains all the following changes that have been uniquely reviewed (most recent first):

└─ $ ▶ git log --oneline --no-merges --graph master..replicated-closed-indices

• c6c42a1 Adapt NoOpEngineTests after Do not wait for advancement of checkpoint in recovery #39006
• 3f9993d Wait for shards to be active after closing indices (Wait for shards to be active after closing indices #38854)
• 5e7a428 Adapt the Cluster Health API to closed indices (Adapt the Cluster Health and Cat Indices APIs to closed indices #39364)
• 3e61939 Adapt CloseFollowerIndexIT for replicated closed indices (Adapt CloseFollowerIndexIT for replicated closed indices #38767)
• 71f5c34 Recover closed indices after a full cluster restart (Recover closed indices after a full cluster restart #39249)
• 4db7fd9 Adapt the Recovery API for closed indices (Adapt the Recovery API for closed indices #38421)
• 4fd1bb2 Adapt more tests suites to closed indices (Adapt more tests suites to closed indices #39186)
• 0519016 Add replica to primary promotion test for closed indices (Add replica to primary promotion test for closed indices #39110)
• b756f6c Test the Cluster Shard Allocation Explain API with closed indices (Test the Cluster Shard Allocation Explain API with closed indices #38631)
• c484c66 Remove index routing table of closed indices in mixed versions clusters (Remove index routing table of closed indices in mixed versions clusters #38955)
• 00f1828 Mute CloseFollowerIndexIT.testCloseAndReopenFollowerIndex()
• e845b0a Do not schedule Refresh/Translog/GlobalCheckpoint tasks for closed indices (Do not schedule Refresh/Translog/GlobalCheckpoint tasks for closed indices #38329)
• cf9a015 Adapt testIndexCanChangeCustomDataPath for replicated closed indices (Adapt testIndexCanChangeCustomDataPath for replicated closed indices #38327)
• b9becdd Adapt testPendingTasks() for replicated closed indices (Adapt testPendingTasks() for replicated closed indices #38326)
• 02cc730 Allow shards of closed indices to be replicated as regular shards (Allow shards of closed indices to be replicated as regular shards #38024)
• e53a9be Fix compilation error in IndexShardIT after merge with master
• cae4155 Relax NoOpEngine constraints (Relax NoOpEngine constraints on translog #37413)
• 54d110b [RCI] Adapt NoOpEngine to latest FrozenEngine changes
• c63fd69 [RCI] Add NoOpEngine for closed indices ([RCI] Add NoOpEngine for closed indices #33903)

---

This pull request adds support for 8.0 and should be backported to 7.x.

Relates to #33888

[elasticmachine]

Pinging @elastic/es-distributed

[tlrx]

@ywelsch CI is happy and this is ready for your approval.

Note that some commits were not reviewed as part of pull requests (mostly merge conflicts fixes):
c6c42a1, 00f1828 and e53a9be.

[ywelsch]

LGTM. Thank you

[tlrx]

Thanks @ywelsch