operator-controller should not uninstall/rollback a failed install/upgrade

[joelanford]

Right now, operator-controller is using helm-operator-plugins, which automatically uninstalls failed installations and rolls back failed upgrades.

This is problematic because the installation or upgrade may have progressed to the point of no return. For example:

• a pre-install/pre-upgrade hook may have run a migration and a subsequent object failed to update.
• after any hooks run, a deployment may have started that ran a migration, but then subsequent objects fail to update.

In my opinion, a better behavior is to just stop, inform a user of the problem, and require human intervention to resolve and progress.

[joelanford]

I have a change in helm-operator-plugins that adds a knob to turn this behavior off. From a PoC standpoint, this looks very straightforward.

[joelanford]

PR up to make it possible for operator-controller to configure this behavior in helm-operator-plugins: operator-framework/helm-operator-plugins#348

[bentito]

I think some high-frequency environments, like particularly those managed by SD, are going to want the simplicity of the default Helm rollback behavior. As long as we leave the knob for them to allow automatic rollback on failed upgrade, that should be enough.

[joelanford]

I'm not sure I agree. I think the important thing for a managed service SRE user is that:

1. The failure is apparent
2. There is a way to centrally apply a fix and have that fix propagate to the affected installations/upgrades.

If OLM automates rollbacks, it could make an SRE's job harder because it could make the problem worse (as described in the description), and it could make troubleshooting more difficult because logs/metrics/object status in the failed release would be wiped out by the rollback.