Node rollout and upgrade requests for production cluster must be approved by customer

[MartaD]

User Story

• As an administrator of a production cluster, I want to be aware and approve any nodes rollout on my cluster so it doesn't happen unintentionally to avoid possible downtime in case of issues.

• As an administrator of a production cluster, I want to be asked for confirmation before upgrade process is triggered.

Proposal

• After node rollout is triggered it doesn't happen immediately but only after confirmation from a customer.
• Starting an upgrade requires confirmation from a customer

Unknown:

• What happens if the customer doesn't confirm?
• Who can approve these requests?

Tasks

[MartaD]

@othylmann @giantswarm/chapter-se please review and add missing parts.

[pipo02mix]

So it looks we need to separate upgrades that imply node rolling and ones that not. Or do we want to this general? IMO any change in the cluster can affect potentially the customer so it would make sense to have a confirmation.

An implication of the criteria is that all the components on the stack (ignition included) are part of the stack version so nothing can modify a cluster without the customer approval

[MartaD]

so nothing can modify a cluster without the customer approval

Yes, and I think the problems we had with customer were unexpected - we didn't expect any node rollout(last P1) or continuing the upgrade process(previous P1) so the only option would be to block all and expect confirmation. Is this doable and reasonable? I guess many things can break if we don't get that confirmation in time?

Perhaps we should go the other way around and notify customers that they have XX time to stop the upgrade/node rollout and continue if there are no objections within that period?