When upgrading istiod, how is community restarting all workloads before removing the "old" control-plane?

[schahal]

Is a "smart" CLI tool worth me investigating/sharing with the community?

This question is specifically tailored for large clusters (not really a problem for small/medium clusters).

Context

https://istio.io/latest/docs/setup/upgrade/canary/

Restarting injected [workloads] will now result in those workloads using the 1-24-2 control plane.

The Problem

Example:

When both 1-23-x and 1-24-2 istio control planes are running on a large cluster (e.g., couple thousand nodes, 10s of thousands of pods), it's not ideal to restart all workloads right away.. e.g.,:

• extra node availability (scheduling concerns)
• some apps not having HA (coordination with non-Istio operators when is a good time)
• it's a manual out-of-band process

Current Solution

We let both control-planes sit for a couple weeks and workloads organically restart (e.g., therefore connecting to v1-24-2).

And, then, after that, we can manually restart the few remaining stragglers remove 1-23-x

Proposed Solution

Have a Job (CLI tool) that does this for us automatically (removing details for brevity, but it would):

1. Check if multiple control planes are installed on the cluster
2. Find all pods connected to the "old" control plane for more than N time since the new control plane has been up
3. Restart (delete) pod (with a "long" wait before going to the next pod, due to us not knowing which kind of app this is, like e.g., a kafka-broker pod, etc).

This can simulate basically a faster, organic-like pod churn (and therefore much quicker removal of the "old" control plane)

[schahal]

The very TL;DR question I'm asking:

For exceptionally large clusters, is a "smart" CLI tool worth me investigating/sharing with the community to restart workloads connected to the old istio-control plane? Or are others going about a different way?

[hercynium]

Thanks for your comment on Slack, @schahal! For anybody else who's got this same requirement, I created a k8s operator that handles restarting workloads after Istio upgrades (and related config changes). It can be found at https://github.com/istio-ecosystem/fortsa

As per your suggestion, I started a new discussion announcing its availability.