OCPBUGS-25862: CO health: only track current in-progress upgrade start

[petr-muller]

Keeping the original upgrade start time may cause CVO spuriously warn about waiting on a CO for over 40 minutes, in a scenario like this:

1. CVO upgrades etcd / KAS very early in the upgrade, noting the time when it started to do that
2. These two COs upgrade successfully and upgrade proceeds
3. Eventually cluster starts rebooting masters and etcd/KAS go degraded
4. CVO compares current time against the noted time, discovers its more than 40 minutes and starts warning about it.

[openshift-ci-robot]

@petr-muller: This pull request references Jira Issue OCPBUGS-25862, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @jiajliu

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Keeping the original upgrade start time may cause CVO spuriously warn
about waiting on a CO for over 40 minutes, in a scenario like this:

1. CVO upgrades etcd / KAS very early in the upgrade, noting the time when it started to do that
2. These two COs upgrade successfuly and upgrade proceeds
3. Eventually cluster starts rebooting masters and etcd/KAS go degraded
4. CVO compares current time against the noted time, discovers its more than 40 minutes and starts warning about it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[petr-muller]

/test unit

time="2023-12-22T16:20:20Z" level=fatal msg="Error pushing manifest list/index to registry: sha256:d08090c120f7e6d771b2deeacb7af4444bdb7c608598cae9ad0299085ebbbbf3: failed commit on ref \"index-image-registry.openshift-image-registry.svc:5000/ci-op-0z6krxrn/pipeline:src@sha256:d08090c120f7e6d771b2deeacb7af4444bdb7c608598cae9ad0299085ebbbbf3\": unexpected status from PUT request to https://image-registry.openshift-image-registry.svc:5000/v2/ci-op-0z6krxrn/pipeline/manifests/src: 500 Internal Server Error"

[wking]

/lgtm

[petr-muller]

/retest

[openshift-bot]

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

[openshift-ci-robot]

@openshift-bot: This pull request references Jira Issue OCPBUGS-25862, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @jiajliu

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@petr-muller: This pull request references Jira Issue OCPBUGS-25862, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (jianl@redhat.com), skipping review request.

In response to this:

Keeping the original upgrade start time may cause CVO spuriously warn about waiting on a CO for over 40 minutes, in a scenario like this:

1. CVO upgrades etcd / KAS very early in the upgrade, noting the time when it started to do that
2. These two COs upgrade successfully and upgrade proceeds
3. Eventually cluster starts rebooting masters and etcd/KAS go degraded
4. CVO compares current time against the noted time, discovers its more than 40 minutes and starts warning about it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[petr-muller]

/label tide/merge-method-squash

[petr-muller]

/retest

[petr-muller]

/remove-label tide/merge-method-squash

[JianLi-RH]

/label qe-approved

[petr-muller]

/test e2e-agnostic-ovn

[openshift-ci]

@petr-muller: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[LalatenduMohanty]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LalatenduMohanty, petr-muller, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [LalatenduMohanty,petr-muller,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@petr-muller: Jira Issue OCPBUGS-25862: All pull requests linked via external trackers have merged:

• openshift/cluster-version-operator#1011

Jira Issue OCPBUGS-25862 has been moved to the MODIFIED state.

In response to this:

Keeping the original upgrade start time may cause CVO spuriously warn about waiting on a CO for over 40 minutes, in a scenario like this:

1. CVO upgrades etcd / KAS very early in the upgrade, noting the time when it started to do that
2. These two COs upgrade successfully and upgrade proceeds
3. Eventually cluster starts rebooting masters and etcd/KAS go degraded
4. CVO compares current time against the noted time, discovers its more than 40 minutes and starts warning about it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build cluster-version-operator-container-v4.16.0-202401162251.p0.g384e326.assembly.stream for distgit cluster-version-operator.
All builds following this will include this PR.

[petr-muller]

/cherry-pick release-4.15

[openshift-cherrypick-robot]

@petr-muller: new pull request created: #1023

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.