Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1822943: return sync error to add to queue #15

Merged
merged 1 commit into from Apr 27, 2020

Conversation

sallyom
Copy link
Contributor

@sallyom sallyom commented Apr 12, 2020

When sync errors encountered, nothing is added to queue, so operator is stuck until operator pod refreshed. This PR ensures resync upon error.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 12, 2020
@sallyom sallyom changed the title WIP: return sync error to add to queue Bug 1822943: return sync error to add to queue Apr 12, 2020
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 12, 2020
@openshift-ci-robot
Copy link
Contributor

@sallyom: This pull request references Bugzilla bug 1822943, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.5.0) matches configured target release for branch (4.5.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1822943: return sync error to add to queue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Apr 12, 2020
@@ -94,12 +94,7 @@ func (c *TargetController) sync() error {
default:
}

forceRequeue, err := c.syncKubeStorageVersionMigrator(spec, status, objectMetaGeneration)
if forceRequeue && err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't just changing && to || fix the problem?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, i can try that instead but will also need to bubble up the errors from syncKubeStorageVersionMigrator - as/is forceRequeue is only true when err is nil, and also, the error from syncKubeStorageVersionMigrator that happens when, for example, the operand ns is deleted, does not get returned, it's buried in the operator status message, while the fn returns nil - so errors like that will never trigger a requeue - that's what I've been observing when I delete the operand ns, the operator gets stuck, only recovers when i refresh the operator pod.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, ptal

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks wrong. Instead we should return the error if non-nil.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, updated, thanks

Comment on lines 34 to 36
manageOperatorStatusDegraded(syncErr, operatorStatus)
manageOperatorStatusProgressing(nil, syncErr, operatorStatus, generation)
return syncErr
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry here that we might of also gone Available==False and we would not be reflecting that if we don't check the deployment also.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i see, yes, i'll look at that in the morning : )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated this PR, made it a much smaller diff-still fixed the issue of operand ns disappearing forever, ptal, thanks!

@sttts
Copy link
Contributor

sttts commented Apr 27, 2020

/approve
lgtm

Leaving final lgtm to @sanchezl.

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 27, 2020
Copy link
Contributor

@sanchezl sanchezl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 27, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sallyom, sanchezl, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit 27e98ad into openshift:master Apr 27, 2020
@openshift-ci-robot
Copy link
Contributor

@sallyom: All pull requests linked via external trackers have merged: openshift/cluster-kube-storage-version-migrator-operator#15. Bugzilla bug 1822943 has been moved to the MODIFIED state.

In response to this:

Bug 1822943: return sync error to add to queue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants