[release-4.13] OCPBUGS-20263: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation #980

openshift-cherrypick-robot · 2023-10-09T17:34:52Z

This is an automated cherry-pick of #976

/assign wking

When: * It has been longer than MinBetweenMatches since our last wrapped evaluation, * There is no alternative key stale enough (MinForCondition) to steal the evaluation, and * The currently requested key is not cached yet, e.g. because it was recently declared as a new risk to a cluster-version operator which had recently run all the other relevent risks from the current Cincinnati response. avoid crashing with: $ grep -B1 -A15 'too fresh' previous.log I0927 12:07:55.594222 1 cincinnati.go:114] Using a root CA pool with 0 root CA subjects to request updates from https://raw.githubusercontent.com/shellyyang1989/upgrade-cincy/master/cincy-conditional-edge-invalid-promql.json?arch=amd64&channel=stable-4.15&id=dc628f75-7778-457a-bb69-6a31a243c3a9&version=4.15.0-0.test-2023-09-27-091926-ci-ln-01zw7kk-latest I0927 12:07:55.726463 1 cache.go:118] {"type":"PromQL","promql":{"promql":"0 * group(cluster_version)"}} is the most stale cached cluster-condition match entry, but it is too fresh (last evaluated on 2023-09-27 11:37:25.876804482 +0000 UTC m=+175.082381015). However, we don't have a cached evaluation for {"type":"PromQL","promql":{"promql":"group(cluster_version_available_updates{channel=buggy})"}}, so attempt to evaluate that now. I0927 12:07:55.726602 1 cache.go:129] {"type":"PromQL","promql":{"promql":"0 * group(cluster_version)"}} is stealing this cluster-condition match call for {"type":"PromQL","promql":{"promql":"group(cluster_version_available_updates{channel=buggy})"}}, because its last evaluation completed 30m29.849594461s ago I0927 12:07:55.758573 1 cvo.go:703] Finished syncing available updates "openshift-cluster-version/version" (170.074319ms) E0927 12:07:55.758847 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) goroutine 194 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1c4df00?, 0x32abc60}) /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001489d40?}) /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75 panic({0x1c4df00, 0x32abc60}) /usr/lib/golang/src/runtime/panic.go:884 +0x213 github.com/openshift/cluster-version-operator/pkg/clusterconditions/promql.(*PromQL).Match(0xc0004860e0, {0x220ded8, 0xc00041e550}, 0x0) /go/src/github.com/openshift/cluster-version-operator/pkg/clusterconditions/promql/promql.go:134 +0x419 github.com/openshift/cluster-version-operator/pkg/clusterconditions/cache.(*Cache).Match(0xc0002d3ae0, {0x220ded8, 0xc00041e550}, 0xc0033948d0) /go/src/github.com/openshift/cluster-version-operator/pkg/clusterconditions/cache/cache.go:132 +0x982 github.com/openshift/cluster-version-operator/pkg/clusterconditions.(*conditionRegistry).Match(0xc000016760, {0x220ded8, 0xc00041e550}, {0xc0033948a0, 0x1, 0x0?})

openshift-ci-robot · 2023-10-09T17:35:01Z

@openshift-cherrypick-robot: Jira Issue OCPBUGS-19921 has been cloned as Jira Issue OCPBUGS-20263. Will retitle bug to link to clone.
/retitle [release-4.13] OCPBUGS-20263: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation

In response to this:

This is an automated cherry-pick of #976

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-10-09T17:35:20Z

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-20263, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.13.z) matches configured target version for branch (4.13.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-19921 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-19921 targets the "4.14.0" version, which is one of the valid target versions: 4.14.0
bug has dependents

Requesting review from QA contact:
/cc @jiajliu

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is an automated cherry-pick of #976

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-10-09T17:35:22Z

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-20263, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.13.z) matches configured target version for branch (4.13.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-19921 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-19921 targets the "4.14.0" version, which is one of the valid target versions: 4.14.0
bug has dependents

Requesting review from QA contact:
/cc @jiajliu

In response to this:

This is an automated cherry-pick of #976

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking

/lgtm

openshift-ci · 2023-10-09T19:51:39Z

@openshift-cherrypick-robot: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

shellyyang1989 · 2023-10-10T09:07:59Z

PR pre-merge test looks good

/label qe-approved

openshift-ci-robot · 2023-10-10T09:08:07Z

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-20263, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.13.z) matches configured target version for branch (4.13.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-19921 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-19921 targets the "4.14.0" version, which is one of the valid target versions: 4.14.0
bug has dependents

Requesting review from QA contact:
/cc @shellyyang1989

In response to this:

This is an automated cherry-pick of #976

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jiajliu · 2023-10-10T09:14:08Z

/label cherry-pick-approved

LalatenduMohanty

/label backport-risk-assessed

openshift-ci · 2023-10-10T19:58:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LalatenduMohanty, openshift-cherrypick-robot, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [LalatenduMohanty,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-10-10T20:01:47Z

@openshift-cherrypick-robot: Jira Issue OCPBUGS-20263: All pull requests linked via external trackers have merged:

openshift/cluster-version-operator#980

Jira Issue OCPBUGS-20263 has been moved to the MODIFIED state.

In response to this:

This is an automated cherry-pick of #976

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot · 2023-10-11T21:44:48Z

Fix included in accepted release 4.13.0-0.nightly-2023-10-11-152221

openshift-cherrypick-robot mentioned this pull request Oct 9, 2023

[release-4.14] OCPBUGS-19921: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation #976

Merged

openshift-cherrypick-robot assigned wking Oct 9, 2023

openshift-ci bot changed the title ~~[release-4.13] OCPBUGS-19921: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation~~ [release-4.13] OCPBUGS-20263: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation Oct 9, 2023

wking mentioned this pull request Oct 9, 2023

[release-4.13] OCPBUGS-19828: pkg/clusterconditions/promql: Warm cache with 1s delay #974

Merged

openshift-ci bot requested review from jiajliu, Davoska and wking October 9, 2023 17:35

wking approved these changes Oct 9, 2023

View reviewed changes

openshift-ci bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 9, 2023

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Oct 10, 2023

openshift-ci bot requested a review from shellyyang1989 October 10, 2023 09:08

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Oct 10, 2023

LalatenduMohanty approved these changes Oct 10, 2023

View reviewed changes

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Oct 10, 2023

openshift-ci bot assigned LalatenduMohanty Oct 10, 2023

openshift-ci bot merged commit 6362a2b into openshift:release-4.13 Oct 10, 2023
9 checks passed

This was referenced Oct 25, 2023

pkg/clusterconditions/promql: Warm cache with 1s delay #987

Closed

OCPBUGS-22408: pkg/clusterconditions/promql: Warm cache with 1s delay #988

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release-4.13] OCPBUGS-20263: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation #980

[release-4.13] OCPBUGS-20263: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation #980

openshift-cherrypick-robot commented Oct 9, 2023

openshift-ci-robot commented Oct 9, 2023

openshift-ci-robot commented Oct 9, 2023

openshift-ci-robot commented Oct 9, 2023

wking left a comment

openshift-ci bot commented Oct 9, 2023

shellyyang1989 commented Oct 10, 2023

openshift-ci-robot commented Oct 10, 2023

jiajliu commented Oct 10, 2023

LalatenduMohanty left a comment

openshift-ci bot commented Oct 10, 2023

openshift-ci-robot commented Oct 10, 2023

openshift-merge-robot commented Oct 11, 2023

[release-4.13] OCPBUGS-20263: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation #980

[release-4.13] OCPBUGS-20263: pkg/clusterconditions/cache: Avoid panic on all-fresh-cache evaluation #980

Conversation

openshift-cherrypick-robot commented Oct 9, 2023

openshift-ci-robot commented Oct 9, 2023

openshift-ci-robot commented Oct 9, 2023

openshift-ci-robot commented Oct 9, 2023

wking left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Oct 9, 2023

shellyyang1989 commented Oct 10, 2023

openshift-ci-robot commented Oct 10, 2023

jiajliu commented Oct 10, 2023

LalatenduMohanty left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Oct 10, 2023

openshift-ci-robot commented Oct 10, 2023

openshift-merge-robot commented Oct 11, 2023