Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release-4.14] OCPBUGS-22984: Introduce upgrading label to block concurrent upgrades #1926

Merged

Conversation

jrvaldes
Copy link
Contributor

@jrvaldes jrvaldes commented Nov 6, 2023

This is a manual cherry-pick of #1901

This change introduces the concept of maximum number of parallel upgrades
that takes place concurrently for Windows nodes during reconciliation.

The `windowsmachineconfig.openshift.io/upgrading` label is proposed
as the locking mechanism among the Windows nodes to account for how
many instances can perform an upgrade under following a threshold
i.e. MaxParallelUpgrades which is fixed to 1.

(cherry picked from commit f1dd8f5)
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 6, 2023
Copy link
Contributor

openshift-ci bot commented Nov 6, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 6, 2023
@openshift-ci-robot
Copy link

@jrvaldes: This pull request references Jira Issue OCPBUGS-22984, which is invalid:

  • expected the bug to target only the "4.14.z" version, but multiple target versions were set
  • expected Jira Issue OCPBUGS-22984 to depend on a bug targeting a version in 4.15.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is a manual cherry-pick of #1901

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jrvaldes
Copy link
Contributor Author

jrvaldes commented Nov 6, 2023

/jira refresh

@openshift-ci-robot
Copy link

@jrvaldes: This pull request references Jira Issue OCPBUGS-22984, which is invalid:

  • expected dependent Jira Issue OCPBUGS-8996 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jrvaldes
Copy link
Contributor Author

jrvaldes commented Nov 6, 2023

/test ?

Copy link
Contributor

openshift-ci bot commented Nov 6, 2023

@jrvaldes: The following commands are available to trigger required jobs:

  • /test aws-e2e-operator
  • /test aws-e2e-upgrade
  • /test azure-e2e-operator
  • /test build
  • /test ci-bundle-wmco-bundle
  • /test gcp-e2e-operator
  • /test images
  • /test lint
  • /test nutanix-e2e-operator
  • /test platform-none-vsphere-e2e-operator
  • /test security
  • /test unit
  • /test vsphere-e2e-operator
  • /test vsphere-proxy-e2e-operator
  • /test wicd-unit-vsphere

The following commands are available to trigger optional jobs:

  • /test vsphere-e2e-upgrade

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-windows-machine-config-operator-release-4.14-aws-e2e-operator
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-aws-e2e-upgrade
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-azure-e2e-operator
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-build
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-ci-bundle-wmco-bundle
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-gcp-e2e-operator
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-images
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-lint
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-nutanix-e2e-operator
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-platform-none-vsphere-e2e-operator
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-security
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-unit
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-vsphere-e2e-operator
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-vsphere-proxy-e2e-operator
  • pull-ci-openshift-windows-machine-config-operator-release-4.14-wicd-unit-vsphere

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jrvaldes
Copy link
Contributor Author

jrvaldes commented Nov 6, 2023

/jira ?

@jrvaldes
Copy link
Contributor Author

jrvaldes commented Nov 7, 2023

/test vsphere-e2e-upgrade

@jrvaldes
Copy link
Contributor Author

jrvaldes commented Nov 7, 2023

vsphere-e2e-upgrade failed with

2023-11-07T16:19:23Z	INFO	controller.windowsmachine	node upgrade has been blocked, as it can not be ensured that workloads will not be disrupted. If an in-tree persistent storage volume is in use, please ensure the CSI drivers for the given node have been deployed. This block must be overriden by applying the label windowsmachineconfig.openshift.io/allow-upgrade=true to the node. It is recommended to unblock Nodes individually, and to wait for the upgrade to complete sucessfully before unblocking another Node.

This commit introduces a test to check the maximum allowed numbers
of Windows nodes upgrading in parallel. The test is divided in two
phases, 1) setup and 2) test, where the setup phase deploys a job
with a fixed name that constantly fetch the number of Windows nodes
with the `windowsmachineconfig.openshift.io/upgrading` label
and fail if is greater than the maximum allowed. The polling
frequency is set to 5 seconds.

The latter test, checks the number of failed pods for the
checker job and require no failures, otherwise fails the e2e test.

A new service account is proposed in the test namespace to hold the
RBAC required by the checker job to list the nodes in the test cluster.

The test is designed to run as a separate job due to the structure of
the new upgrade test in vSphere (vsphere-e2e-upgrade) that is scattered
between the steps in the release repo and code in the WMCO test suite.

(cherry picked from commit e03c792)
This changes aggregates the RBAC resources required by the test runner job
in the proposed function ensureTestRunnerRBAC() to avoid duplication of
intended functionality.

(cherry picked from commit a2a6f6b)
@jrvaldes
Copy link
Contributor Author

jrvaldes commented Nov 8, 2023

/test vsphere-e2e-upgrade

@jrvaldes
Copy link
Contributor Author

jrvaldes commented Nov 9, 2023

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 9, 2023
@openshift-ci-robot
Copy link

@jrvaldes: This pull request references Jira Issue OCPBUGS-22984, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.14.z) matches configured target version for branch (4.14.z)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
  • dependent bug Jira Issue OCPBUGS-8996 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-8996 targets the "4.15.0" version, which is one of the valid target versions: 4.15.0
  • bug has dependents

Requesting review from QA contact:
/cc @rrasouli

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested a review from rrasouli November 9, 2023 14:52
@jrvaldes
Copy link
Contributor Author

/jira refresh

@openshift-ci-robot
Copy link

@jrvaldes: This pull request references Jira Issue OCPBUGS-22984, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.14.z) matches configured target version for branch (4.14.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • dependent bug Jira Issue OCPBUGS-8996 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-8996 targets the "4.15.0" version, which is one of the valid target versions: 4.15.0
  • bug has dependents

Requesting review from QA contact:
/cc @rrasouli

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jrvaldes jrvaldes marked this pull request as ready for review November 20, 2023 16:55
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 20, 2023
@aravindhp
Copy link
Contributor

/unassign

@aravindhp aravindhp removed their request for review November 24, 2023 16:47
@mtnbikenc
Copy link
Member

mtnbikenc commented Dec 1, 2023

/hold
Until after #1950

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 1, 2023
@mtnbikenc
Copy link
Member

/hold cancel

@jrvaldes
Copy link
Contributor Author

jrvaldes commented Dec 5, 2023

/lgtm

Copy link
Contributor

openshift-ci bot commented Dec 5, 2023

@jrvaldes: you cannot LGTM your own PR.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jrvaldes
Copy link
Contributor Author

jrvaldes commented Dec 8, 2023

/hold cancel

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 8, 2023
@jrvaldes
Copy link
Contributor Author

jrvaldes commented Dec 8, 2023

/retest-required

@alinaryan
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 8, 2023
@sebsoto
Copy link
Contributor

sebsoto commented Dec 11, 2023

/approve

Copy link
Contributor

openshift-ci bot commented Dec 11, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jrvaldes, sebsoto

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 11, 2023
Copy link
Contributor

openshift-ci bot commented Dec 11, 2023

@jrvaldes: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/security 1fffa07 link false /test security

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 6e82dfc and 2 for PR HEAD 1fffa07 in total

@openshift-merge-bot openshift-merge-bot bot merged commit 124be2f into openshift:release-4.14 Dec 12, 2023
16 of 17 checks passed
@openshift-ci-robot
Copy link

@jrvaldes: Jira Issue OCPBUGS-22984: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-22984 has been moved to the MODIFIED state.

In response to this:

This is a manual cherry-pick of #1901

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants