[release-4.13] OCPBUGS-23016: Introduce upgrading label to block concurrent upgrades #1931

jrvaldes · 2023-11-07T16:18:25Z

This is a manual cherry-pick of #1901

openshift-ci · 2023-11-07T16:18:31Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

jrvaldes · 2023-11-07T16:19:17Z

/jira cherrypick OCPBUGS-22984

openshift-ci-robot · 2023-11-07T16:19:25Z

@jrvaldes: Jira Issue OCPBUGS-22984 has been cloned as Jira Issue OCPBUGS-23016. Will retitle bug to link to clone.
/retitle OCPBUGS-23016: [release-4.13] Introduce upgrading label to block concurrent upgrades

In response to this:

/jira cherrypick OCPBUGS-22984

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-11-07T16:19:39Z

@jrvaldes: This pull request references Jira Issue OCPBUGS-23016, which is invalid:

expected dependent Jira Issue OCPBUGS-22984 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is a manual cherry-pick of #1901

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jrvaldes · 2023-11-07T16:20:53Z

/test azure-e2e-upgrade

jrvaldes · 2023-11-08T14:00:31Z

azure-e2e-upgrade failed with timeout while waiting for Windows nodes to come up as shown in WMCO logs

2023-11-07T18:51:19Z	INFO	controller.windowsmachine	node upgrade has been blocked, as it can not be ensured that workloads will not be disrupted. If an in-tree persistent storage volume is in use, please ensure the CSI drivers for the given node have been deployed. This block must be overriden by applying the label windowsmachineconfig.openshift.io/allow-upgrade=true to the node. It is recommended to unblock Nodes individually, and to wait for the upgrade to complete sucessfully before unblocking another Node.
2023-11-07T18:51:19Z	DEBUG	events	node upgrade has been blocked, as it can not be ensured that workloads will not be disrupted. If an in-tree persistent storage volume is in use, please ensure the CSI drivers for the given node have been deployed. This block must be overriden by applying the label windowsmachineconfig.openshift.io/allow-upgrade=true to the node. It is recommended to unblock Nodes individually, and to wait for the upgrade to complete sucessfully before unblocking another Node.	{"type": "Warning", "object": {"kind":"Node","name":"e2e-wm-jtz7v","uid":"eeec953c-189a-4ad3-b38e-74671b509657","apiVersion":"v1","resourceVersion":"51048"}, "reason": "UpgradeBlocked"}

jrvaldes · 2023-11-08T14:44:52Z

/test azure-e2e-upgrade

mtnbikenc · 2023-12-01T15:25:26Z

/hold
Until after 8.1.1

This change introduces the concept of maximum number of parallel upgrades that takes place concurrently for Windows nodes during reconciliation. The `windowsmachineconfig.openshift.io/upgrading` label is proposed as the locking mechanism among the Windows nodes to account for how many instances can perform an upgrade under following a threshold i.e. MaxParallelUpgrades which is fixed to 1. (cherry picked from commit f1dd8f5) (cherry picked from commit f7dd118)

This commit introduces a test to check the maximum allowed numbers of Windows nodes upgrading in parallel. The test is divided in two phases, 1) setup and 2) test, where the setup phase deploys a job with a fixed name that constantly fetch the number of Windows nodes with the `windowsmachineconfig.openshift.io/upgrading` label and fail if is greater than the maximum allowed. The polling frequency is set to 5 seconds. The latter test, checks the number of failed pods for the checker job and require no failures, otherwise fails the e2e test. A new service account is proposed in the test namespace to hold the RBAC required by the checker job to list the nodes in the test cluster. The test is designed to run as a separate job due to the structure of the new upgrade test in vSphere (vsphere-e2e-upgrade) that is scattered between the steps in the release repo and code in the WMCO test suite. (cherry picked from commit e03c792) (cherry picked from commit 48f08a8)

This changes aggregates the RBAC resources required by the test runner job in the proposed function ensureTestRunnerRBAC() to avoid duplication of intended functionality. (cherry picked from commit a2a6f6b) (cherry picked from commit a367e19)

jrvaldes · 2023-12-19T17:09:59Z

/jira refresh

openshift-ci-robot · 2023-12-19T17:10:02Z

@jrvaldes: This pull request references Jira Issue OCPBUGS-23016, which is invalid:

expected dependent Jira Issue OCPBUGS-22984 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mansikulkarni96 · 2023-12-19T17:18:32Z

/approve

openshift-ci · 2023-12-19T17:20:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jrvaldes, mansikulkarni96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mansikulkarni96]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

saifshaikh48

/lgtm

jrvaldes · 2023-12-21T14:08:14Z

/retest-required

jrvaldes · 2023-12-21T14:08:34Z

/jira refresh

openshift-ci-robot · 2023-12-21T14:08:40Z

@jrvaldes: This pull request references Jira Issue OCPBUGS-23016, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.13.z) matches configured target version for branch (4.13.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-22984 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-22984 targets the "4.14.z" version, which is one of the valid target versions: 4.14.0, 4.14.z
bug has dependents

Requesting review from QA contact:
/cc @rrasouli

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jrvaldes · 2023-12-21T14:10:21Z

/hold cancel

openshift-ci-robot · 2023-12-21T17:19:23Z

/retest-required

Remaining retests: 0 against base HEAD 3a68e91 and 2 for PR HEAD d10c7fb in total

jrvaldes · 2023-12-21T17:58:20Z

/test remaining-required

aravindhp · 2023-12-21T18:32:27Z

/unassign

openshift-ci · 2023-12-21T21:12:59Z

@jrvaldes: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2023-12-21T21:17:22Z

@jrvaldes: Jira Issue OCPBUGS-23016: All pull requests linked via external trackers have merged:

openshift/windows-machine-config-operator#1931

Jira Issue OCPBUGS-23016 has been moved to the MODIFIED state.

In response to this:

This is a manual cherry-pick of #1901

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 7, 2023

openshift-ci bot changed the title ~~[release-4.13] Introduce upgrading label to block concurrent upgrades~~ OCPBUGS-23016: [release-4.13] Introduce upgrading label to block concurrent upgrades Nov 7, 2023

jrvaldes changed the title ~~OCPBUGS-23016: [release-4.13] Introduce upgrading label to block concurrent upgrades~~ [release-4.13] OCPBUGS-23016: Introduce upgrading label to block concurrent upgrades Nov 7, 2023

jrvaldes force-pushed the release-4.13-backport-1901 branch 2 times, most recently from 06667a3 to 6187bb9 Compare November 8, 2023 14:44

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 1, 2023

jrvaldes marked this pull request as ready for review December 8, 2023 20:04

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 8, 2023

jrvaldes marked this pull request as draft December 8, 2023 20:05

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 8, 2023

openshift-ci bot requested review from aravindhp and mansikulkarni96 December 8, 2023 20:07

jrvaldes added 3 commits December 19, 2023 12:09

jrvaldes force-pushed the release-4.13-backport-1901 branch from fc57982 to d10c7fb Compare December 19, 2023 17:09

jrvaldes marked this pull request as ready for review December 19, 2023 17:09

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 19, 2023

openshift-ci bot requested a review from sebsoto December 19, 2023 17:13

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 19, 2023

saifshaikh48 reviewed Dec 19, 2023

View reviewed changes

openshift-ci bot assigned saifshaikh48 Dec 19, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 19, 2023

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Dec 21, 2023

openshift-ci bot requested a review from rrasouli December 21, 2023 14:08

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 21, 2023

openshift-merge-bot bot merged commit 72dd116 into openshift:release-4.13 Dec 21, 2023
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release-4.13] OCPBUGS-23016: Introduce upgrading label to block concurrent upgrades #1931

[release-4.13] OCPBUGS-23016: Introduce upgrading label to block concurrent upgrades #1931

jrvaldes commented Nov 7, 2023

openshift-ci bot commented Nov 7, 2023

jrvaldes commented Nov 7, 2023

openshift-ci-robot commented Nov 7, 2023

openshift-ci-robot commented Nov 7, 2023

jrvaldes commented Nov 7, 2023

jrvaldes commented Nov 8, 2023

jrvaldes commented Nov 8, 2023

mtnbikenc commented Dec 1, 2023 •

edited

jrvaldes commented Dec 19, 2023

openshift-ci-robot commented Dec 19, 2023

mansikulkarni96 commented Dec 19, 2023

openshift-ci bot commented Dec 19, 2023

saifshaikh48 left a comment

jrvaldes commented Dec 21, 2023

jrvaldes commented Dec 21, 2023

openshift-ci-robot commented Dec 21, 2023

jrvaldes commented Dec 21, 2023

openshift-ci-robot commented Dec 21, 2023

jrvaldes commented Dec 21, 2023

aravindhp commented Dec 21, 2023

openshift-ci bot commented Dec 21, 2023

openshift-ci-robot commented Dec 21, 2023

[release-4.13] OCPBUGS-23016: Introduce upgrading label to block concurrent upgrades #1931

[release-4.13] OCPBUGS-23016: Introduce upgrading label to block concurrent upgrades #1931

Conversation

jrvaldes commented Nov 7, 2023

openshift-ci bot commented Nov 7, 2023

jrvaldes commented Nov 7, 2023

openshift-ci-robot commented Nov 7, 2023

openshift-ci-robot commented Nov 7, 2023

jrvaldes commented Nov 7, 2023

jrvaldes commented Nov 8, 2023

jrvaldes commented Nov 8, 2023

mtnbikenc commented Dec 1, 2023 • edited

jrvaldes commented Dec 19, 2023

openshift-ci-robot commented Dec 19, 2023

mansikulkarni96 commented Dec 19, 2023

openshift-ci bot commented Dec 19, 2023

saifshaikh48 left a comment

Choose a reason for hiding this comment

jrvaldes commented Dec 21, 2023

jrvaldes commented Dec 21, 2023

openshift-ci-robot commented Dec 21, 2023

jrvaldes commented Dec 21, 2023

openshift-ci-robot commented Dec 21, 2023

jrvaldes commented Dec 21, 2023

aravindhp commented Dec 21, 2023

openshift-ci bot commented Dec 21, 2023

openshift-ci-robot commented Dec 21, 2023

mtnbikenc commented Dec 1, 2023 •

edited