Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: prevent upgrade failures caused by deleting resources #2121

Merged
merged 1 commit into from May 10, 2024

Conversation

l-qing
Copy link
Contributor

@l-qing l-qing commented Apr 20, 2024

When installing resources in TektonInstallerSet, if the original resource is in the process of being deleted, it is necessary to wait for the deletion to complete before proceeding with the installation. Otherwise, it may appear that the TektonInstallerSets have been installed successfully, but in the end, there may be missing resources.

In my scenario, when upgrading the pipeline, since the old RoleBinding tekton-pipelines-webhook was still being deleted, the new tektoninstallersets thought the resource already existed with no hash change, so no action was taken. As a result, when the Pod tekton-pipelines-webhook started up, it found that it did not have permissions for certain resources config-logging, leading to startup failure. Its failure directly caused some configmaps config-registry-cert to not be created due to the presence of the validatingwebhookconfigurations named config.webhook.pipeline.tekton.dev. Ultimately, this led to the entire pipeline upgrade failing. Only by reconstructing the instance could recovery be possible.

Changes

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

See the contribution guide for more details.

Release Notes

  • Bug fixes
fix: prevent upgrade failures caused by deleting resources

@tekton-robot tekton-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Apr 20, 2024
@tekton-robot
Copy link
Contributor

Hi @l-qing. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@tekton-robot tekton-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 20, 2024
@l-qing l-qing force-pushed the fix/upgrade-conflict-resolution branch from c2f97dd to be865eb Compare April 23, 2024 15:07
@jkandasa
Copy link
Member

/ok-to-test

@tekton-robot tekton-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 24, 2024
@tekton-robot
Copy link
Contributor

The following is the coverage report on the affected files.
Say /test pull-tekton-operator-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/kubernetes/tektoninstallerset/install.go 72.5% 73.2% 0.7

@l-qing l-qing force-pushed the fix/upgrade-conflict-resolution branch from be865eb to 552172b Compare April 29, 2024 09:52
@tekton-robot
Copy link
Contributor

The following is the coverage report on the affected files.
Say /test pull-tekton-operator-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/kubernetes/tektoninstallerset/install.go 72.5% 73.2% 0.7

@l-qing l-qing force-pushed the fix/upgrade-conflict-resolution branch from 552172b to 0b45753 Compare May 7, 2024 06:49
@tekton-robot
Copy link
Contributor

The following is the coverage report on the affected files.
Say /test pull-tekton-operator-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/kubernetes/tektoninstallerset/install.go 72.5% 73.1% 0.5

Copy link
Member

@jkandasa jkandasa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tekton-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jkandasa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 8, 2024
@jkandasa
Copy link
Member

jkandasa commented May 8, 2024

Thanks for your contribution to tekon operator @l-qing 🎉

When installing resources in TektonInstallerSet , if the original resource is
in the process of being deleted, it is necessary to wait for the deletion to
complete before proceeding with the installation. Otherwise, it may appear that
the TektonInstallerSets have been installed successfully, but in the end, there
may be missing resources.
@l-qing l-qing force-pushed the fix/upgrade-conflict-resolution branch from 0b45753 to ca18750 Compare May 10, 2024 13:39
@tekton-robot
Copy link
Contributor

The following is the coverage report on the affected files.
Say /test pull-tekton-operator-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/kubernetes/tektoninstallerset/install.go 72.5% 73.1% 0.5

@piyush-garg
Copy link
Contributor

/lgtm

Thanks for the PR 🎉

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label May 10, 2024
@jkandasa
Copy link
Member

/retest-required

@tekton-robot tekton-robot merged commit 40e36e7 into tektoncd:main May 10, 2024
8 checks passed
@jkandasa jkandasa added the kind/bug Categorizes issue or PR as related to a bug. label May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants