Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1889921: Reporting degraded if not available #644

Merged
merged 4 commits into from Dec 1, 2020
Merged

Bug 1889921: Reporting degraded if not available #644

merged 4 commits into from Dec 1, 2020

Conversation

ricardomaraschini
Copy link
Contributor

Report this operator as Degraded if there is no replica or less replicas
then specified on the configuration for more than one minute.

@openshift-ci-robot openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. labels Nov 24, 2020
@openshift-ci-robot
Copy link
Contributor

@ricardomaraschini: This pull request references Bugzilla bug 1889921, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

WIP - Bug 1889921: Reporting degraded if not available

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Nov 24, 2020
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 24, 2020
@@ -294,6 +295,13 @@ func (c *Controller) syncStatus(cr *imageregistryv1.Config, deploy *appsapi.Depl

updateCondition(cr, operatorapiv1.OperatorStatusTypeProgressing, operatorProgressing)

if isDeploymentStatusAvailable(deploy) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we look at operatorAvailable.Status here, instead of calling out to isDeploymentStatusAvailable again? And do we need .deployUnavailableSince when we should already have LastTransitionTime on the Available status (although maybe not on operatorAvailable until we use some tooling syncs it into the existing Conditions object).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this makes total sense. I will try to adjust according to your input, thanks for that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, PTAL.

if time.Since(c.deployIncompleteSince) > time.Minute {
operatorDegraded.Status = operatorapiv1.ConditionTrue
operatorDegraded.Message = "The registry has minimum availability"
operatorDegraded.Reason = "MinimumAvailability"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reason/message sounds positive. If we are going to set Degraded=True, we should say something negative to motivate the unhappy status. Something like Registry deployment has timed out progressing. But... can we lean on the Deployment controller for this and look at ProgressDeadlineExceeded? See my rotted-due-to-lack-of-interest openshift/library-go#575 for one possible implementation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, PTAL.

Report this operator as Degraded if there is no replica or less replicas
then specified on the configuration.
Covering Controller.syncStatus with tests.
@ricardomaraschini ricardomaraschini changed the title WIP - Bug 1889921: Reporting degraded if not available Bug 1889921: Reporting degraded if not available Nov 26, 2020
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 26, 2020
@ricardomaraschini
Copy link
Contributor Author

/assign @dmage

Comment on lines 314 to 317
} else if deploy == nil {
operatorDegraded.Status = operatorapiv1.ConditionTrue
operatorDegraded.Message = "The deployment does not exist"
operatorDegraded.Reason = "DeploymentNotFound"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we shouldn't immediately react on absence of the deployment. Most likely the operator will fix this problem in a second.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are absolutely right. I have adjusted it by means of removing it, now it will take one minute as well. PTAL.

if cond.Type != appsapi.DeploymentProgressing {
continue
}
if cond.Reason != "ProgressDeadlineExceeded" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to rely on progressDeadlineSeconds of the registry deployment, I'd say we should be more opinionated and set it to 1 minute.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, PTAL.

Setting a default value of 60 seconds before setting the deployment as
failed with ProgressDeadlineExceeded.
If the deployment does not exist we wait one minute before setting the
operator to degraded.
@ricardomaraschini
Copy link
Contributor Author

/retest

@dmage
Copy link
Member

dmage commented Nov 30, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 30, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dmage, ricardomaraschini

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [dmage,ricardomaraschini]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

5 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 10229b9 into openshift:master Dec 1, 2020
@openshift-ci-robot
Copy link
Contributor

@ricardomaraschini: All pull requests linked via external trackers have merged:

Bugzilla bug 1889921 has been moved to the MODIFIED state.

In response to this:

Bug 1889921: Reporting degraded if not available

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants