Discussion: handling backwards incompatible changes for pending resources

[wlynch]

Splitting off from #6436 for more discussion. /cc @lbernick

Something we ran into a few times but I don't know if we have an answer for is how to handle backwards incompatible changes for already existing resources. i.e.

1. Run is created in supported version
2. Run isn't executed for whatever reason (controller isn't healthy, remote resolver is 500ing because upstream dependency is down, etc)
3. Backwards incompatible change is made (feature flag is turned off, update to new pipelines release, etc)
4. Existing resource cannot be modified because new validation admission rejects previously accepted state.
5. Run is stuck in a retry loop because the controllers can't modify the resource, even to mark it as failed.

This affects any controller that acts on pipeline crds (pipelines, chains, results).

Here's a few examples I'm aware of,

• Old resource deletion hangs when alpha flag is off #5203
• Integration tests for changing feature flags #5999
• Revert removal of PipelineResources related fields #6436 (note: this one is speculation, we don't actually know if this would be impacted)

We should figure out the limitations of validating webhooks for backwards incompatible changes (e.g. can we still update status even if the spec is now invalid? annotations/labels?) and document a plan for how to handle these types of changes.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[vdemeester]

/remove-lifecycle rotten

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.