[enterprise-metrics] update ingester podManagementPolicy to Parallel

[replay]

This is a request for feedback, I'm curious if others in the team agree that the pros outweigh the cons.

I think we should update the ingester podManagementPolicy to Parallel.

Pros:

• This is what we use in our own clusters
• It will help to recover clusters with a broken ingester ring faster, because to do so we'll then simply have to:
  • Scale down Distributors to 0 (to cut off ingestion)
  • Scale down Ingesters to 0
  • Drop Ingester ring
  • Scale up Ingesters to final replica count. This will now be much faster because all Ingesters will replay their WALs concurrently as opposed to sequentially
  • Scale up Distributors to final replica count
• According to the docs the podManagementPolicy is on relevant when scaling the SFS up/down, not when rolling out updates

Cons:

• The upgrade procedure from before this change to after this change is rather complicated. It will require that the SFS is deleted with --cascade=false and then recreated. This is risky, because if a customer fails to specify --cascade=false their write path will go down.

[pracucci]

Change LGTM, but keep in mind you can't change the pod management policy on a statefulset. You have to recreate it, which may be very annoying for customers!

[vlad-diachenko]

Lgtm

[replay]

keep in mind you can't change the pod management policy on a statefulset

Thx for reviewing. Yep, that's the only "con" which I can see with this change, but I guess we'll just have to make sure that our support and solution engineering teams are aware.

[replay]

I had to force-push because I forgot to DCO sign a commit, apparently this has invalidated the approvals which this PR already had (at least I think this is why they are now not counted anymore).