OTA-1160: pkg/cvo/reconciliation_issues: Publish ClusterOperator transitionStart

[wking]

Allow ClusterVersion status consumers to make their own decisions about how long to wait before complaining about a slow-to-update ClusterOperator.

[openshift-ci-robot]

@wking: This pull request references OTA-1160 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Allow ClusterVersion status consumers to make their own decisions about how long to wait before complaining about a slow-to-update ClusterOperator.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[wking]

Testing in Cluster Bot via launch 4.16,openshift/cluster-version-operator#1044 aws,techpreview (logs), I sent the cluster back to 4.16.0-ec.5 with:

$ oc adm upgrade --force --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:f5c9cf5a461434e775af2b946fc2a8aee7240f5b05d4141a588e527adcad7af9

Maybe I didn't need to --force that? Anyhow, I let it run for a bit, until etcd completed:

$ oc adm upgrade
info: An upgrade is in progress. Working towards 4.16.0-ec.5: 113 of 952 done (11% complete), waiting on kube-apiserver
...

Then I turned around and went back to my original release images (I had to --force this time, because CI releases are unsigned):

$ oc adm upgrade --force --allow-upgrade-with-warnings --allow-explicit-upgrade --to-image registry.build03.ci.openshift.org/ci-ln-cm7dfnt/release@sha256:370d219ae7612d27ac08ab6de38e452c71bb4f770c692577fdd4493242ff3efd

And shortly after, here's the CVO waiting on etcd and kube-apiserver giving us start-times:

$ date --utc --iso=m
2024-04-25T04:50+00:00
$ oc get -o json clusterversion version | jq -r '.status.conditions[] | select(.type == "ReconciliationIssues") | .message | fromjson'
{
  "message": "Cluster operators etcd, kube-apiserver are updating versions",
  "children": [
    {
      "message": "[Cluster operator etcd is updating versions, Cluster operator kube-apiserver is updating versions]",
      "children": [
        {
          "message": "Cluster operator etcd is updating versions",
          "children": [
            {
              "message": "cluster operator etcd is available and not degraded but has not finished updating to target version"
            }
          ],
          "effect": "None",
          "manifest": {
            "originalFilename": "0000_20_etcd-operator_07_clusteroperator.yaml",
            "group": "config.openshift.io",
            "kind": "ClusterOperator",
            "name": "etcd"
          },
          "transitionStart": "2024-04-25T04:50:25.572524285Z"
        },
        {
          "message": "Cluster operator kube-apiserver is updating versions",
          "children": [
            {
              "message": "cluster operator kube-apiserver is available and not degraded but has not finished updating to target version"
            }
          ],
          "effect": "None",
          "manifest": {
            "originalFilename": "0000_20_kube-apiserver-operator_07_clusteroperator.yaml",
            "group": "config.openshift.io",
            "kind": "ClusterOperator",
            "name": "kube-apiserver"
          },
          "transitionStart": "2024-04-25T04:50:25.572311511Z"
        }
      ]
    }
  ],
  "effect": "None"
}

And a bit later, when kube-apiserver (which had never gotten all the way through to 4.16.0-ec.5) had completed, just waiting on etcd:

$ oc get -o json clusterversion version | jq -r '.status.conditions[] | select(.type == "ReconciliationIssues") | .message | fromjson'
{
  "message": "Cluster operator etcd is updating versions",
  "children": [
    {
      "message": "cluster operator etcd is available and not degraded but has not finished updating to target version"
    }
  ],
  "effect": "None",
  "manifest": {
    "originalFilename": "0000_20_etcd-operator_07_clusteroperator.yaml",
    "group": "config.openshift.io",
    "kind": "ClusterOperator",
    "name": "etcd"
  },
  "transitionStart": "2024-04-25T04:50:25.572524285Z"
}

[petr-muller]

We're definitely not breaking Hypershift with this feature-gated change
/override ci/prow/e2e-hypershift ci/prow/e2e-hypershift-conformance

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [petr-muller,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@petr-muller: Overrode contexts on behalf of petr-muller: ci/prow/e2e-hypershift, ci/prow/e2e-hypershift-conformance

In response to this:

We're definitely not breaking Hypershift with this feature-gated change
/override ci/prow/e2e-hypershift ci/prow/e2e-hypershift-conformance

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wking]

Waiving pre-merge QE for this tech-preview, currently unconsumed content.

/label no-qe

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 53a8fbf and 2 for PR HEAD a804d97 in total

[wking]

/retest-required

[wking]

not sure what's up with HyperShift, but that job won't care about this tech-preview property.

/retest-required

[wking]

well, e2e-hypershift-conformance passed. Still dunno what's going on with the other.

/retest-required

[wking]

wait, I'm a root approver for this repo 🤦 :

/override ci/prow/e2e-hypershift

[openshift-ci]

@wking: Overrode contexts on behalf of wking: ci/prow/e2e-hypershift

In response to this:

wait, I'm a root approver for this repo 🤦 :

/override ci/prow/e2e-hypershift

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@wking: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build cluster-version-operator-container-v4.16.0-202404251943.p0.g0b3f507.assembly.stream.el9 for distgit cluster-version-operator.
All builds following this will include this PR.