Bug 1979575: [4.8]: Add timeout to informers cache sync #132

dgrisonnet · 2021-07-07T11:55:52Z

Cherry-pick the following commits to release-4.8 to fix a bug where waiting for caches synchronization never returns:

42e8930 - add 10 minute timeout
80f090b - early exit on error

During heavy apiserver disruptions, the informers cache synchronization done by the operator before running the informers might never return. This result in the operator being stuck forever waiting for its informers caches to be synchronized. To prevent that, we introduce a timeout of 10 minutes in the context passed to client-go to make sure that the sync will always return at some point. Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

If a cache sync times out, there's no need to continue since the operator will exit eventually. Signed-off-by: Simon Pasquier <spasquie@redhat.com>

openshift-ci · 2021-07-07T11:55:53Z

@dgrisonnet: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-07-07T11:56:28Z

@dgrisonnet: This pull request references Bugzilla bug 1977435, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.9.0" instead
expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is VERIFIED instead
expected Bugzilla bug 1977435 to depend on a bug targeting a release in 4.9.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-07-07T11:57:19Z

@dgrisonnet: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

simonpasquier

/lgtm

openshift-ci · 2021-07-07T13:07:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgrisonnet, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dgrisonnet,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-07-07T14:28:59Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-07-07T17:04:58Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-07-08T01:01:01Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-08T01:01:05Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-09T01:00:50Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-09T01:00:54Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-10T01:00:36Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-10T01:00:39Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-11T01:00:50Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-11T01:00:55Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-12T01:00:52Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-12T01:00:55Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-13T01:01:11Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-13T01:01:16Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-14T01:00:31Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-14T01:00:35Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-15T01:01:05Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-15T01:01:09Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-18T01:01:11Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-18T01:01:15Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-19T01:01:13Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-19T01:01:18Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-20T01:01:02Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-20T01:01:06Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-21T01:00:54Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-21T01:00:58Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-22T01:00:57Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-22T01:01:01Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-07-23T01:00:54Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-07-23T01:01:03Z

@openshift-bot: This pull request references Bugzilla bug 1979575, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.z) matches configured target release for branch (4.8.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1977435 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 1977435 targets the "4.9.0" release, which is one of the valid target releases: 4.9.0
bug has dependents

Requesting review from QA contact:
/cc @juzhao

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ecordell · 2021-08-04T19:03:21Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: missed cutoff today, may make it tomorrow

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

ecordell · 2021-08-05T18:54:47Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: missed cutoff today, may make it tomorrow

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

ecordell · 2021-08-05T18:55:26Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: missed cutoff for today, may make tomorrow

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

ecordell · 2021-08-06T16:28:11Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: skipping for now, may still be tagged today

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

ecordell · 2021-08-06T17:55:41Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: missed cutoff this week

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

openshift-ci · 2021-08-11T19:16:03Z

@dgrisonnet: All pull requests linked via external trackers have merged:

openshift/prometheus-operator#132

Bugzilla bug 1979575 has been moved to the MODIFIED state.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2023-11-26T12:29:32Z

[ART PR BUILD NOTIFIER]

This PR has been included in build prometheus-config-reloader-container-v4.8.0-202311261141.p0.g8f4efab.assembly.stream for distgit prometheus-config-reloader.
All builds following this will include this PR.

dgrisonnet and others added 2 commits July 7, 2021 13:52

pkg: exit early if caches can't be synced

a0aa4c8

If a cache sync times out, there's no need to continue since the operator will exit eventually. Signed-off-by: Simon Pasquier <spasquie@redhat.com>

openshift-ci bot requested review from s-urbaniak and squat July 7, 2021 11:56

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 7, 2021

dgrisonnet changed the title ~~Bz 1977435: [4.8]: Add timeout to informers cache sync~~ Bug 1977435: [4.8]: Add timeout to informers cache sync Jul 7, 2021

openshift-ci bot added the bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. label Jul 7, 2021

openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jul 7, 2021

dgrisonnet changed the title ~~Bug 1977435: [4.8]: Add timeout to informers cache sync~~ Bug 1979575: [4.8]: Add timeout to informers cache sync Jul 7, 2021

simonpasquier reviewed Jul 7, 2021

View reviewed changes

openshift-ci bot assigned simonpasquier Jul 7, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 7, 2021

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jul 23, 2021

openshift-ci bot requested a review from juzhao July 23, 2021 01:01

bparees added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Aug 11, 2021

openshift-ci bot merged commit 8f4efab into openshift:release-4.8 Aug 11, 2021

Bug 1979575: [4.8]: Add timeout to informers cache sync #132

Bug 1979575: [4.8]: Add timeout to informers cache sync #132

Conversation

dgrisonnet commented Jul 7, 2021

openshift-ci bot commented Jul 7, 2021

openshift-ci bot commented Jul 7, 2021

openshift-ci bot commented Jul 7, 2021

simonpasquier left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 7, 2021

openshift-bot commented Jul 7, 2021

openshift-bot commented Jul 7, 2021

openshift-bot commented Jul 8, 2021

openshift-ci bot commented Jul 8, 2021

openshift-bot commented Jul 9, 2021

openshift-ci bot commented Jul 9, 2021

openshift-bot commented Jul 10, 2021

openshift-ci bot commented Jul 10, 2021

openshift-bot commented Jul 11, 2021

openshift-ci bot commented Jul 11, 2021

openshift-bot commented Jul 12, 2021

openshift-ci bot commented Jul 12, 2021

openshift-bot commented Jul 13, 2021

openshift-ci bot commented Jul 13, 2021

openshift-bot commented Jul 14, 2021

openshift-ci bot commented Jul 14, 2021

openshift-bot commented Jul 15, 2021

openshift-ci bot commented Jul 15, 2021

openshift-bot commented Jul 18, 2021

openshift-ci bot commented Jul 18, 2021

openshift-bot commented Jul 19, 2021

openshift-ci bot commented Jul 19, 2021

openshift-bot commented Jul 20, 2021

openshift-ci bot commented Jul 20, 2021

openshift-bot commented Jul 21, 2021

openshift-ci bot commented Jul 21, 2021

openshift-bot commented Jul 22, 2021

openshift-ci bot commented Jul 22, 2021

openshift-bot commented Jul 23, 2021

openshift-ci bot commented Jul 23, 2021

ecordell commented Aug 4, 2021

ecordell commented Aug 5, 2021

ecordell commented Aug 5, 2021

ecordell commented Aug 6, 2021

ecordell commented Aug 6, 2021

openshift-ci bot commented Aug 11, 2021

openshift-bot commented Nov 26, 2023