Add AWS UPI Windows Containers CI jobs by rrasouli · Pull Request #75983 · openshift/release

rrasouli · 2026-03-10T10:07:09Z

Add AWS UPI Windows Containers CI jobs for OCP 4.18-4.22

Summary

This PR adds periodic and presubmit debug CI jobs for Windows Containers testing on AWS UPI (User-Provisioned Infrastructure) for OpenShift versions 4.18 through 4.22.

Changes

CI Job Configurations

Release 4.18-4.21:

Added debug presubmit jobs (debug-winc-aws-upi)
Triggered on changes to test/extended/winc/ directory
Uses cucushift-installer-rehearse-aws-upi-ovn-winc workflow

Release 4.22 (active release):

Added periodic job (aws-upi-ovn-winc-f7) - runs on schedule (every 7 days)
Generated cron schedule: 0 23 4,11,18,27 * *
Added debug presubmit job - triggered by code changes
Provides comprehensive coverage for the current development release

New Workflow

Created cucushift-installer-rehearse-aws-upi-ovn-winc workflow:

Provisions AWS UPI cluster with standard platform: aws configuration
Uses OVN hybrid overlay networking required for Windows Containers
Uses BYOH (Bring Your Own Host) for Windows nodes via terraform-windows-provisioner
UPI deployment (no Machine API / MachineSets for Windows nodes)
Deploys WMCO (Windows Machine Config Operator) from Konflux
Uses cucushift-installer-check-upi chain to skip controlplanemachinesets validation

New Step Registry Components

1. AWS Windows AMI Discovery

Files: ci-operator/step-registry/aws/windows/ami-discover/

Discovers the latest Windows Server AMI for the configured version (2019/2022):

Queries AWS SSM Parameter Store for official Microsoft Windows AMIs
Supports both Windows Server 2019 and 2022
Saves AMI ID and region to SHARED_DIR for use by BYOH provisioner

2. BYOH Windows Workload Preparation

Files: ci-operator/step-registry/cucushift/winc/prepare/byoh/

Prepares Windows Container test workloads for BYOH clusters:

Sets up ImageTagMirrorSet to redirect Windows images to CI registry mirror
Waits for Windows nodes to become Ready and schedulable
Removes CCM uninitialized taint from Windows nodes
Creates test workloads (Windows and Linux deployments, services)
Validates workloads are running before tests begin

Windows Node Taints

Windows nodes are configured with the following taints by WMCO:

os=Windows:NoSchedule - Primary Windows node taint
os=windows:NoSchedule - Alternate casing support

The CCM taint node.cloudprovider.kubernetes.io/uninitialized:NoSchedule is removed by the prepare step before creating workloads.

Test workloads include tolerations for all three taints as a safety net.

Technical Details

UPI vs IPI Architecture

IPI (Infrastructure Provider Install):

OpenShift manages all infrastructure
Windows nodes via MachineSets
Automatic scaling and management
Platform type: aws, azure, gcp, etc.

UPI (User-Provisioned Infrastructure) - AWS:

Platform aws with manually provisioned infrastructure
Windows nodes via BYOH (terraform-windows-provisioner)
Manual node provisioning and management
No MachineSets for Windows nodes (control plane may use Machine API)

OVN Hybrid Networking

Windows Containers require OVN hybrid overlay networking:

Linux nodes: Standard OVN-Kubernetes
Windows nodes: Hybrid overlay network
Configuration via Network CR: spec.defaultNetwork.ovnKubernetesConfig.hybridOverlayConfig
Manifests created by ovn-conf-hybrid-manifest step

Workflow Architecture

ipi-install-rbac
  ↓
ipi-conf-aws (creates install-config with platform: aws)
  ↓
ovn-conf + ovn-conf-hybrid-manifest (create OVN hybrid overlay manifests)
  ↓
upi-install-aws (creates cluster)
  ↓
enable-qe-catalogsource + ssh-bastion
  ↓
openshift-windows-setup-wmco-konflux (deploy WMCO)
  ↓
operatorhub-subscribe + windows-conf-operator
  ↓
aws-windows-ami-discover (find Windows AMI)
  ↓
windows-byoh-provision (provision Windows nodes via terraform)
  ↓
cucushift-winc-prepare-byoh (mirror images, create test workloads)
  ↓
cucushift-installer-check-upi (validates cluster health, excludes CPMS check)

Testing

Rehearsal Commands

4.22 Periodic:

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

Debug Presubmits (triggered by changing test/extended/winc/):

4.18: debug-winc-aws-upi on release-4.18 branch
4.19: debug-winc-aws-upi on release-4.19 branch
4.20: debug-winc-aws-upi on release-4.20 branch
4.21: debug-winc-aws-upi on release-4.21 branch
4.22: debug-winc-aws-upi on release-4.22 branch

Test Configuration

Test scenarios: Windows_Containers (Smokerun only, no upgrades/connected-only)
Test timeout: 50 minutes
Cluster profile: aws-qe
Base domain: qe.devcluster.openshift.com

Related Work

Addresses feedback from @JiaLiu regarding periodic job frequency for older releases
Only 4.22 has periodic jobs; 4.18-4.21 have debug-only to reduce CI load
Aligns with existing IPI Windows Container workflows
Based on working workflow from PR Add AWS UPI Windows Containers CI jobs #74742

Jira: https://issues.redhat.com/browse/WINC-1482

Replaces: #74742 (clean rebase from main)

openshift-ci-robot · 2026-03-10T10:09:57Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-10T11:45:24Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-10T11:45:28Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-10T14:25:15Z

/retest

rrasouli · 2026-03-10T14:45:47Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-10T14:45:50Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-10T17:27:11Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-10T17:27:14Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-10T17:47:34Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-10T17:47:37Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-10T18:00:08Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-10T18:00:12Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-10T19:56:30Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-10T19:56:34Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T00:01:36Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-11T00:01:39Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T03:03:42Z

/pj-rehearse abort

openshift-ci-robot · 2026-03-11T03:03:45Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T03:20:48Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-11T03:20:51Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T15:30:54Z

/pj-rehearse abort

openshift-ci-robot · 2026-03-11T15:30:58Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T15:41:51Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-11T15:41:56Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T16:53:26Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-11T16:53:30Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T17:04:56Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-11T17:05:00Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T21:23:29Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-11T21:23:32Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-03-11T21:30:49Z

/pj-rehearse abort

openshift-ci-robot · 2026-03-11T21:30:51Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Add debug and periodic (f7) jobs for Windows Containers testing on AWS UPI clusters for releases 4.18-4.22. ## Changes - Add debug job: aws-upi-ovn-winc-debug (run_if_changed: test/extended/winc/) - Add periodic job: aws-upi-ovn-winc-f7 (every 7 days) - Use cucushift-installer-rehearse-aws-upi-ovn-winc workflow - Generated cron schedule: 0 23 4,11,18,27 * * (every 7 days) ## Workflow Details The workflow provisions Windows nodes via AWS terraform (BYOH) and includes: 1. Cluster provisioning: upi-aws-provisioner-pre (platform=aws UPI) 2. Windows node provisioning: windows-byoh-provision (terraform + wait for Ready) 3. Test environment setup: cucushift-winc-prepare-byoh 4. UPI validation: cucushift-installer-check-upi (excludes Machine API checks) ## Windows Node Taints Windows nodes are configured with the following taints by WMCO: - `os=Windows:NoSchedule` - Primary Windows node taint - `os=windows:NoSchedule` - Alternate casing support The CCM taint `node.cloudprovider.kubernetes.io/uninitialized:NoSchedule` is removed by the prepare step before creating workloads. Test workloads include tolerations for all three taints as a safety net. ## Test Configuration - Test scenarios: Windows_Containers (Smokerun only, no upgrades/connected-only) - Test timeout: 50 minutes - Cluster profile: aws-qe - Base domain: qe.devcluster.openshift.com

openshift-ci · 2026-03-11T22:19:59Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rrasouli
Once this PR has been reviewed and has the lgtm label, please assign jianlinliu, liangxia for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rrasouli · 2026-03-11T22:21:30Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-upi-ovn-winc-f7

openshift-ci-robot · 2026-03-11T22:21:32Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci-robot · 2026-03-11T22:21:34Z

[REHEARSALNOTIFIER]
@rrasouli: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-openshift-tests-private-release-4.18-debug-winc-aws-upi	openshift/openshift-tests-private	presubmit	Presubmit changed
pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-aws-upi	openshift/openshift-tests-private	presubmit	Presubmit changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-aws-upi	openshift/openshift-tests-private	presubmit	Presubmit changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-aws-upi	openshift/openshift-tests-private	presubmit	Presubmit changed
pull-ci-openshift-openshift-tests-private-release-4.22-debug-winc-aws-upi	openshift/openshift-tests-private	presubmit	Presubmit changed
pull-ci-openshift-openshift-tests-private-main-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-5.0-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.23-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.22-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.18-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.17-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.16-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.13-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-main-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-5.0-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.23-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.22-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.18-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.16-debug-winc-vsphere-ipi	openshift/openshift-tests-private	presubmit	Registry content changed

A total of 211 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

rrasouli mentioned this pull request Mar 10, 2026

Add AWS UPI Windows Containers CI jobs #74742

Closed

4 tasks

openshift-ci bot requested review from gpei and jianlinliu March 10, 2026 10:09

rrasouli force-pushed the aws-upi-winc-clean branch 5 times, most recently from df3068d to f2a8788 Compare March 10, 2026 11:43

rrasouli force-pushed the aws-upi-winc-clean branch from f2a8788 to f0985ef Compare March 10, 2026 17:35

rrasouli force-pushed the aws-upi-winc-clean branch from f0985ef to 210703d Compare March 11, 2026 09:40

rrasouli force-pushed the aws-upi-winc-clean branch from d9b0872 to 1a8bda8 Compare March 11, 2026 15:18

rrasouli mentioned this pull request Mar 11, 2026

Add vSphere platform=none Windows Containers CI jobs #76098

Open

rrasouli force-pushed the aws-upi-winc-clean branch 2 times, most recently from 829d97c to 1a8bda8 Compare March 11, 2026 17:02

rrasouli force-pushed the aws-upi-winc-clean branch 2 times, most recently from 80705b8 to d7407a6 Compare March 11, 2026 21:16

rrasouli force-pushed the aws-upi-winc-clean branch 2 times, most recently from d0c59bd to 180b7f6 Compare March 11, 2026 22:00

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 11, 2026

openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Mar 11, 2026

rrasouli force-pushed the aws-upi-winc-clean branch from 180b7f6 to 843d7bc Compare March 11, 2026 22:18

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 11, 2026

openshift-ci-robot removed the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Mar 11, 2026

Conversation

rrasouli commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add AWS UPI Windows Containers CI jobs for OCP 4.18-4.22

Summary

Changes

CI Job Configurations

New Workflow

New Step Registry Components

1. AWS Windows AMI Discovery

2. BYOH Windows Workload Preparation

Windows Node Taints

Technical Details

UPI vs IPI Architecture

OVN Hybrid Networking

Workflow Architecture

Testing

Rehearsal Commands

Test Configuration

Related Work

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

rrasouli commented Mar 10, 2026

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

rrasouli commented Mar 10, 2026

Uh oh!

rrasouli commented Mar 10, 2026

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

rrasouli commented Mar 10, 2026

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

rrasouli commented Mar 10, 2026

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

rrasouli commented Mar 10, 2026

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

rrasouli commented Mar 10, 2026

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

rrasouli commented Mar 11, 2026

Uh oh!

openshift-ci-robot commented Mar 11, 2026

Uh oh!

rrasouli commented Mar 11, 2026

Uh oh!

openshift-ci-robot commented Mar 11, 2026

Uh oh!

rrasouli commented Mar 11, 2026

Uh oh!

openshift-ci-robot commented Mar 11, 2026

Uh oh!

rrasouli commented Mar 11, 2026

Uh oh!

openshift-ci-robot commented Mar 11, 2026

Uh oh!

rrasouli commented Mar 11, 2026

Uh oh!

openshift-ci-robot commented Mar 11, 2026

Uh oh!

rrasouli commented Mar 11, 2026

Uh oh!

openshift-ci-robot commented Mar 11, 2026

Uh oh!

rrasouli commented Mar 11, 2026

Uh oh!

openshift-ci-robot commented Mar 11, 2026

Uh oh!

rrasouli commented Mar 11, 2026

Uh oh!

openshift-ci-robot commented Mar 11, 2026

rrasouli commented Mar 10, 2026 •

edited

Loading