OCPBUGS-18131: Block upgrade to 4.13 via admin ack #171

gnufied · 2023-09-22T19:00:33Z

Fixes https://issues.redhat.com/browse/OCPBUGS-18131

Also address other review comments

openshift-ci · 2023-09-22T19:01:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gnufied

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [gnufied]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-09-22T19:02:38Z

@gnufied: This pull request references Jira Issue OCPBUGS-18131, which is invalid:

expected the bug to target the "4.12.z" version, but no target version was set
expected Jira Issue OCPBUGS-18131 to depend on a bug targeting a version in 4.13.0, 4.13.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Fixes https://issues.redhat.com/browse/OCPBUGS-18131

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

gnufied · 2023-09-22T19:04:45Z

/jira refresh

openshift-ci-robot · 2023-09-22T19:04:48Z

@gnufied: This pull request references Jira Issue OCPBUGS-18131, which is invalid:

expected dependent Jira Issue OCPBUGS-18132 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Also tweak wording for 7 and 8 series

jsafrane

/lgtm

jsafrane · 2023-09-27T12:33:23Z

pkg/operator/vspherecontroller/checks/check_error.go

+
+	pvs, err := apiDependencies.ListPersistentVolumes()
+	if err != nil {
+		reason := fmt.Errorf("vsphere csi driver installed failed with %s, unable to list pvs: %v", result.Reason, err)


This error is IMO not related to CSI driver installation
(this applies also to the original PR, so I don't mind if we leave it as it is...)

jsafrane · 2023-09-27T12:41:55Z

/label backport-risk-assessed
/retest-required

duanwei33 · 2023-09-28T09:21:47Z

Pre-merge verified with PR: #171 and openshift/cluster-storage-operator#402.

Create cluster in vsphere 8, while ESXi version is 8.0.1.0-22088125(doesn't match the requirement) and vsphere version is 8.0.1.0-22088981 (matches the requirement)

oc adm upgrade shows the CSI migration warning as below which is expected, not starting with a new line however:

$ oc adm upgrade
Cluster version is 4.12.0-0.test-2023-09-28-022513-ci-ln-58nnh9k-latest

Upgradeable=False

  Reason: AdminAckRequired
  Message: Kubernetes 1.26 and therefore OpenShift 4.13 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6958394 for details and instructions. vSphere CSI migration will be enabled in Openshift-4.14. Your cluster appears to be using in-tree vSphere volumes and is on a vSphere version that has CSI migration related bugs. See - https://access.redhat.com/node/7011683 for more information, before upgrading to 4.14.

Check the driver operator log which is expected:

I0928 07:16:18.276872       1 vmware.go:273] Found existing profile with same name: openshift-storage-policy-wduan-0928b-vs-k6pft
I0928 07:16:19.178787       1 check_nodes.go:174] checking for patched version of ESXi for CSI migration: 8.0.1.0-22088125
I0928 07:16:19.178913       1 check_nodes.go:174] checking for patched version of ESXi for CSI migration: 8.0.1.0-22088125
I0928 07:16:19.186219       1 check_nodes.go:174] checking for patched version of ESXi for CSI migration: 8.0.1.0-22088125
I0928 07:16:19.186276       1 check_patched_vcenter.go:29] checking for patched version of vSphere for CSI migration: 8.0.1.0-22088981
I0928 07:16:19.186289       1 vsphere_environment_checker.go:90] Overall check result: UpgradeRequiresAdminAck: host host-1013 is on ESXI version 8.0.1.0-22088125, minimum required for CSI migration: > 8.0.1-22088125
I0928 07:16:19.186296       1 vspherecontroller.go:261] Scheduled the next check in 1h0m0s
I0928 07:16:19.186310       1 check_error.go:217] found vSphere in-tree persistent volumes in the cluster
I0928 07:16:19.186316       1 vspherecontroller.go:361] check status is: pass
I0928 07:16:19.186333       1 vspherecontroller.go:516] Updating admin-gates to require admin-ack for vSphere CSI migration
W0928 07:16:19.199366       1 vspherecontroller.go:496] host host-1013 is on ESXI version 8.0.1.0-22088125, minimum required for CSI migration: > 8.0.1-22088125
I0928 07:16:19.199788       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"vmware-vsphere-csi-driver-operator", UID:"4a86e840-d6e9-4af9-b1a5-8eac2ad524ea", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/admin-gates -n openshift-config-managed:

Check admin-gates, we use ack-4.12-kube-126-vsphere-migration-in-4.14 as it has impact when upgrading to 4.14 which is expected:

$ oc get ConfigMap/admin-gates -n openshift-config-managed -o yaml
apiVersion: v1
data:
  ......
  ack-4.12-kube-126-vsphere-migration-in-4.14: vSphere CSI migration will be enabled
    in Openshift-4.14. Your cluster appears to be using in-tree vSphere volumes and
    is on a vSphere version that has CSI migration related bugs. See - https://access.redhat.com/node/7011683
    for more information, before upgrading to 4.14.

duanwei33 · 2023-09-28T09:25:59Z

@gnufied Just one strange observation, the message "Scheduled the next check in 1h0m0s" which stands for the end actually came before all check finished:
(But I don't think it blocks the PER merge, we can have another PRs to fix 4.12/4.13 if needed.)

I0928 07:16:18.276872       1 vmware.go:273] Found existing profile with same name: openshift-storage-policy-wduan-0928b-vs-k6pft
I0928 07:16:19.178787       1 check_nodes.go:174] checking for patched version of ESXi for CSI migration: 8.0.1.0-22088125
I0928 07:16:19.178913       1 check_nodes.go:174] checking for patched version of ESXi for CSI migration: 8.0.1.0-22088125
I0928 07:16:19.186219       1 check_nodes.go:174] checking for patched version of ESXi for CSI migration: 8.0.1.0-22088125
I0928 07:16:19.186276       1 check_patched_vcenter.go:29] checking for patched version of vSphere for CSI migration: 8.0.1.0-22088981
I0928 07:16:19.186289       1 vsphere_environment_checker.go:90] Overall check result: UpgradeRequiresAdminAck: host host-1013 is on ESXI version 8.0.1.0-22088125, minimum required for CSI migration: > 8.0.1-22088125
I0928 07:16:19.186296       1 vspherecontroller.go:261] Scheduled the next check in 1h0m0s
I0928 07:16:19.186310       1 check_error.go:217] found vSphere in-tree persistent volumes in the cluster
I0928 07:16:19.186316       1 vspherecontroller.go:361] check status is: pass
I0928 07:16:19.186333       1 vspherecontroller.go:516] Updating admin-gates to require admin-ack for vSphere CSI migration
W0928 07:16:19.199366       1 vspherecontroller.go:496] host host-1013 is on ESXI version 8.0.1.0-22088125, minimum required for CSI migration: > 8.0.1-22088125
I0928 07:16:19.199788       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"vmware-vsphere-csi-driver-operator", UID:"4a86e840-d6e9-4af9-b1a5-8eac2ad524ea", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/admin-gates -n openshift-config-managed:

duanwei33 · 2023-09-28T09:30:54Z

/label qe-approved
/label cherry-pick-approved

jsafrane · 2023-10-02T12:38:31Z

/jira refresh

openshift-ci-robot · 2023-10-02T12:38:36Z

@jsafrane: This pull request references Jira Issue OCPBUGS-18131, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.12.z) matches configured target version for branch (4.12.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-18132 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-18132 targets the "4.13.z" version, which is one of the valid target versions: 4.13.0, 4.13.z
bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (wduan@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-10-02T13:09:21Z

/retest-required

Remaining retests: 0 against base HEAD 64cb29d and 2 for PR HEAD 77ab37f in total

openshift-ci-robot · 2023-10-02T17:40:24Z

/retest-required

Remaining retests: 0 against base HEAD 2dff285 and 1 for PR HEAD 77ab37f in total

gnufied · 2023-10-02T17:45:20Z

@duanwei33 that should be okay. So we run all the expensive checks and when they are finished we run another check which takes result of previously finished checks and applies further logic. For example - if expensive checks resulted in determining an older version of vSphere then the next check verifies if cluster has intree PVs.

gnufied · 2023-10-04T16:34:02Z

/retest

openshift-ci-robot · 2023-10-04T18:33:31Z

@gnufied: Jira Issue OCPBUGS-18131: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-18131 has been moved to the MODIFIED state.

In response to this:

Fixes https://issues.redhat.com/browse/OCPBUGS-18131

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot · 2023-10-05T00:10:03Z

Fix included in accepted release 4.12.0-0.nightly-2023-10-04-200342

gnufied added 5 commits September 22, 2023 12:19

Block 4.14 upgrades with admin ack

0bfcee0

Add placeholder checks for vSphere 8

6eacf76

Add tests for adding and remove admin-ack keys

3608ea4

Add last known release for vSphere 8

7e3134a

Also address other review comments

Make fixes for vSphere 4.12

eb414fa

openshift-ci bot requested review from bertinatto and jsafrane September 22, 2023 19:00

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 22, 2023

gnufied changed the title ~~Block upgrade 413 admin ack~~ OCPBUGS-18131: Block upgrade 413 admin ack Sep 22, 2023

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Sep 22, 2023

gnufied changed the title ~~OCPBUGS-18131: Block upgrade 413 admin ack~~ OCPBUGS-18131: Block upgrade to 4.13 via admin ack Sep 22, 2023

gnufied added 2 commits September 26, 2023 11:12

Use message from check for adding to conditions

6d282b7

Also tweak wording for 7 and 8 series

Change admin-ack key for upgrading to 4.13

77ab37f

gnufied force-pushed the block-upgrade-413-admin-ack branch from 928c41e to 77ab37f Compare September 26, 2023 16:30

jsafrane reviewed Sep 27, 2023

View reviewed changes

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Sep 27, 2023

openshift-ci bot assigned chao007, duanwei33, Phaow, radeore, ropatil010 and jsafrane Sep 27, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 27, 2023

duanwei33 mentioned this pull request Sep 28, 2023

[release-4.12] OCPBUGS-18131: Add patch for allowing configmap updates via clusterrole openshift/cluster-storage-operator#402

Merged

openshift-ci bot added qe-approved Signifies that QE has signed off on this PR cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. labels Sep 28, 2023

openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Oct 2, 2023

openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Oct 2, 2023

openshift-ci bot merged commit 1271ebf into openshift:release-4.12 Oct 4, 2023
7 of 8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-18131: Block upgrade to 4.13 via admin ack #171

OCPBUGS-18131: Block upgrade to 4.13 via admin ack #171

gnufied commented Sep 22, 2023 •

edited

openshift-ci bot commented Sep 22, 2023

openshift-ci-robot commented Sep 22, 2023

gnufied commented Sep 22, 2023

openshift-ci-robot commented Sep 22, 2023

jsafrane left a comment

jsafrane Sep 27, 2023

jsafrane commented Sep 27, 2023

duanwei33 commented Sep 28, 2023

duanwei33 commented Sep 28, 2023 •

edited

duanwei33 commented Sep 28, 2023

jsafrane commented Oct 2, 2023

openshift-ci-robot commented Oct 2, 2023

openshift-ci-robot commented Oct 2, 2023

openshift-ci-robot commented Oct 2, 2023

gnufied commented Oct 2, 2023

gnufied commented Oct 4, 2023

openshift-ci-robot commented Oct 4, 2023

openshift-merge-robot commented Oct 5, 2023

OCPBUGS-18131: Block upgrade to 4.13 via admin ack #171

OCPBUGS-18131: Block upgrade to 4.13 via admin ack #171

Conversation

gnufied commented Sep 22, 2023 • edited

openshift-ci bot commented Sep 22, 2023

openshift-ci-robot commented Sep 22, 2023

gnufied commented Sep 22, 2023

openshift-ci-robot commented Sep 22, 2023

jsafrane left a comment

Choose a reason for hiding this comment

jsafrane Sep 27, 2023

Choose a reason for hiding this comment

jsafrane commented Sep 27, 2023

duanwei33 commented Sep 28, 2023

duanwei33 commented Sep 28, 2023 • edited

duanwei33 commented Sep 28, 2023

jsafrane commented Oct 2, 2023

openshift-ci-robot commented Oct 2, 2023

openshift-ci-robot commented Oct 2, 2023

openshift-ci-robot commented Oct 2, 2023

gnufied commented Oct 2, 2023

gnufied commented Oct 4, 2023

openshift-ci-robot commented Oct 4, 2023

openshift-merge-robot commented Oct 5, 2023

gnufied commented Sep 22, 2023 •

edited

duanwei33 commented Sep 28, 2023 •

edited