When a Pod is deleted outside of ECK (K8s node maintenance, upgrades, etc.) cold tier data isn't maintained on the node

[SpencerLN]

Bug Report

What did you do?

Our ECK clusters are running on GKE, which frequently automatically applies updates or performs other maintenance, resulting in K8s nodes being replaced, pods being moved, etc. When this occurs for a Hot/Warm tier ES node this process is quick and the data is maintained since the data is on network-attached storage.

For Cold tier ES nodes, the shards are immediately re-allocated to other nodes and the data is re-downloaded from the backing snapshot repository instead of reloaded from the network-attached storage where the data still exists.

What did you expect to see?

Since the data is still on the PV, and the PV is re-attached to the new pod, I expect that the ES cluster should use the local data instead of restoring it from the snapshot repository.

What did you see instead? Under which circumstances?

The cluster immediately reallocated the cold tier shards to other cold tier nodes and they download the data from the snapshot repository, even after the ES node containing the data rejoins the cluster.

This behavior does not occur when ECK recreates/restarts the ES node as part of a rolling upgrade.

Environment

• ECK version:

  2.6.1

• Kubernetes information:

  insert any information about your Kubernetes environment that could help us:

  • Cloud: GKE - 1.22

It isn't clear to me whether this is a bug in ECK or in Elasticsearch. Either way, it doesn't seem to be the expected/desirable behavior.

elastic/elasticsearch#93114

[pebrc]

#6478 should address this if I understand the problem correctly

[pebrc]

Closing this as we have #6544 which should solve the issue.