OCPBUGS-82060: Fix GCP ARM capacity exhaustion in multi-arch jobs

[jianlinliu]

Summary

This PR addresses Component Readiness regression 37839 where GCP multi-arch jobs were failing due to ARM instance capacity exhaustion through zone randomization, instance sizing optimization, balanced worker configuration, and schedule distribution.

Root Causes

1. GCP T2A ARM capacity exhaustion in us-central1-a zone causing machines to get stuck in PROVISIONING state
2. ipi-install-heterogeneous always used first machineset, concentrating load in single zone
3. Larger instances have lower availability - standard-4 instances more likely to hit capacity limits
4. Unbalanced worker configuration - heterogeneous jobs used 3 AMD64 + 2 ARM64 workers (5 total)
5. Schedule contention - 18 GCP ARM jobs all scheduled at Sunday 11:00 UTC, creating massive resource contention

Solutions

1. Zone Randomization

Random machineset selection in ipi-install-heterogeneous distributes ARM instances across zones to reduce capacity pressure.

2. Instance Sizing Optimization

Use smaller instances (standard-2) for additional workers and migration infra to improve availability and reduce resource consumption.

3. Balanced Worker Configuration

Set COMPUTE_NODE_REPLICAS: "2" for GCP heterogeneous jobs to create balanced 2+2 worker layout instead of unbalanced 3+2 configuration.

Before:

• 3 AMD64 workers (default COMPUTE_NODE_REPLICAS: 3)
• 2 ARM64 additional workers (default ADDITIONAL_WORKERS: 2)
• Total: 5 workers

After:

• 2 AMD64 workers (COMPUTE_NODE_REPLICAS: "2")
• 2 ARM64 additional workers (ADDITIONAL_WORKERS: 2)
• Total: 4 workers

Benefits:

• ✅ More balanced heterogeneous cluster (2+2 vs 3+2)
• ✅ Reduced total resource consumption (4 workers vs 5)
• ✅ Faster cluster installation (one less worker to provision)
• ✅ Lower cluster overhead (less kubelet/container runtime load)

4. Schedule Distribution

Change job scheduling from cron to interval-based (168h) for releases 4.19-5.0 to prevent simultaneous execution.

Before:

• All 18 jobs ran at Sunday 11:00 UTC (cron: 0 11 * * 0)
• 3 jobs per release × 6 releases = 18 concurrent jobs
• Massive resource contention spike every Sunday

After:

• All releases 4.19-5.0: Changed to interval: 168h (weekly interval)
• Jobs naturally distribute based on completion times
• Prevents simultaneous resource contention

Benefits:

• ✅ Eliminates weekly resource contention spike
• ✅ Zone randomization more effective when jobs don't overlap
• ✅ Better capacity distribution throughout the week
• ✅ Reduced likelihood of hitting zone capacity limits

5. Why T2A (not C4A/N4A)?

C4A and N4A Compatibility Issue:

• C4A and N4A only support Hyperdisk, NOT Persistent Disk
• OpenShift monitoring, logging, and registry use pd-standard PVCs by default
• Rehearse testing with C4A failed with:

  AttachVolume.Attach failed for volume "pvc-0ad4471f-65fe-40c6-8850-8768b0a91e07"
  rpc error: code = InvalidArgument desc = Failed to Attach: failed cloud service
  attach disk call: googleapi: Error 400: pd-standard disk type cannot be used by
  c4a-standard-4 machine type., badRequest
  

T2A Advantages:

• ✅ Supports BOTH Persistent Disk (pd-standard, pd-balanced, pd-ssd) AND Hyperdisk
• ✅ Full OpenShift compatibility with default storage classes
• ✅ No breaking changes to monitoring, logging, or registry
• ✅ Proven reliability and stability

GCP ARM Instance Disk Support Comparison:

Instance Type	Persistent Disk	Hyperdisk	OpenShift Compatible
T2A (Tau)	✅ Yes (pd-standard, pd-balanced, pd-ssd)	✅ Yes	✅ Yes
C4A (Axion Compute)	❌ No	✅ Yes (hyperdisk-balanced, hyperdisk-extreme)	❌ No
N4A (Axion General)	❌ No	✅ Yes (hyperdisk-balanced, hyperdisk-throughput)	❌ No

Changes

ipi-install-heterogeneous Step

• Random machineset selection for zone distribution
• New ADDITIONAL_WORKER_DISK_TYPE parameter for GCP disk type configuration

Multi-arch Configs (4.17-5.0)

All Releases 4.17-5.0 (T2A processor with optimized sizing):

• COMPUTE_NODE_TYPE: t2a-standard-4 (unchanged, 4 vCPU)
• COMPUTE_NODE_REPLICAS: "2" (for GCP heterogeneous jobs, balanced layout)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → t2a-standard-2 (optimized to 2 vCPU)

MIGRATION jobs:

• MIGRATION_CP_MACHINE_TYPE: t2a-standard-4 (unchanged)
• MIGRATION_INFRA_MACHINE_TYPE: t2a-standard-4 → t2a-standard-2 (4.20-5.0, optimized to 2 vCPU)

Job Schedules:

• 4.19-5.0: Changed to interval: 168h (weekly interval, distributed execution)
• Eliminates simultaneous execution and resource contention

Modified Files (27 total)

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.17*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.18*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.19*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.20*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0*.yaml (3 files)
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-ref.yaml
ci-operator/jobs/openshift/multiarch/openshift-multiarch-main-periodics.yaml

Benefits

1. Reduces capacity exhaustion by:

   • Spreading load across multiple GCP zones (random machineset selection)
   • Using smaller instances (standard-2 vs standard-4) for additional workers and migration infra
   • Smaller instances have exponentially better availability
   • Balanced worker configuration reduces total resource consumption
   • Distributed job execution prevents simultaneous resource spikes

2. Full OpenShift compatibility:

   • T2A supports both Persistent Disk and Hyperdisk
   • Works with default pd-standard storage class
   • No changes required to monitoring, logging, or registry

3. Optimal resource allocation:

   • Compute nodes use larger instances (t2a-standard-4, 4 vCPU)
   • Additional workers use smaller instances (t2a-standard-2, 2 vCPU)
   • Migration infra uses smaller instances (t2a-standard-2, 2 vCPU)
   • GCP heterogeneous jobs: balanced 2+2 worker layout

4. Improved scheduling efficiency:

   • Interval-based scheduling (168h) prevents simultaneous execution
   • Jobs naturally distribute throughout the week
   • Zone randomization more effective without overlapping demand
   • Eliminates weekly resource contention spike

5. Better cluster efficiency:

   • Faster cluster installation (4 workers vs 5)
   • Lower cluster overhead
   • More balanced heterogeneous cluster architecture

6. Future flexibility:

   • ADDITIONAL_WORKER_DISK_TYPE parameter allows easy migration to Hyperdisk if needed
   • If OpenShift defaults change to hyperdisk-balanced, can reconsider C4A/N4A

Related Issues

• Fixes Component Readiness regression: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/regressions/37839
• JIRA Bug: https://issues.redhat.com/browse/OCPBUGS-82060

[openshift-ci-robot]

@jianlinliu: This pull request references Jira Issue OCPBUGS-82060, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

This PR addresses Component Readiness regression 37839 where GCP multi-arch jobs were failing with "verify all machines should be in Running state" due to T2A ARM instance capacity exhaustion.

The test was failing across multiple GCP multi-arch variants (4.22, 4.21, and ARM64/multi architectures) with machines getting stuck in PROVISIONING state and then disappearing with "Instance not found on provider" errors. Root cause analysis showed GCP T2A ARM capacity exhaustion in us-central1-a zone.

Changes

1. Randomize zone selection in heterogeneous install step

• Modified ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh to randomly select a worker machineset instead of always using the first one
• Each machineset is in a different GCP zone, distributing load across zones to avoid zone-specific capacity issues
• Added logging to show which machineset index was selected

2. Distribute GCP ARM instance types across releases

Updated all nightly config files for multiarch jobs to use different ARM instance types per release:

• Release 4.21: t2a-standard-4 → t2a-standard-2 (smaller for better availability, known Tau processor)
• Release 4.22: t2a-standard-4 → c4a-standard-4 (newer Axion generation, addresses regression 37839)
• Release 4.23: t2a-standard-4 → c4a-standard-2 (newest Axion + smallest for best availability)
• Release 5.0: t2a-standard-4 → c4a-standard-2 (newest Axion + smallest for best availability)
• Older releases (≤4.20): unchanged (t2a-standard-4)

Modified files:

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0*.yaml (3 files)

Benefits

1. Reduces capacity exhaustion by spreading load across:

• Multiple GCP zones (via random machineset selection)
• Multiple ARM instance types (t2a-standard-2, t2a-standard-4, c4a-standard-2, c4a-standard-4)

2. Improved availability:

• C4A (Axion) instances generally have better availability than T2A (Tau)
• Smaller instance types (standard-2) have exponentially better availability than standard-4

3. Cost savings: Using smaller instances where appropriate (standard-2 vs standard-4)

4. Forward compatibility: Newer releases use newer Axion processors with better availability

Test Plan

• Monitor multi-arch GCP jobs across all affected releases (4.21, 4.22, 4.23, 5.0)
• Verify heterogeneous worker provisioning succeeds across different zone selections
• Watch for reduced "verify all machines should be in Running state" failures

Related Issues

• Fixes Component Readiness regression: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/regressions/37839

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@jianlinliu: This pull request references Jira Issue OCPBUGS-82060, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Summary

This PR addresses Component Readiness regression 37839 where GCP multi-arch jobs were failing with "verify all machines should be in Running state" due to T2A ARM instance capacity exhaustion.

The test was failing across multiple GCP multi-arch variants (4.22, 4.21, and ARM64/multi architectures) with machines getting stuck in PROVISIONING state and then disappearing with "Instance not found on provider" errors. Root cause analysis showed GCP T2A ARM capacity exhaustion in us-central1-a zone.

Changes

1. Randomize zone selection in heterogeneous install step

• Modified ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh to randomly select a worker machineset instead of always using the first one
• Each machineset is in a different GCP zone, distributing load across zones to avoid zone-specific capacity issues
• Added logging to show which machineset index was selected

2. Distribute GCP ARM instance types across releases

Strategy: Use bigger instances (standard-4) for COMPUTE_NODE_TYPE and smaller instances (standard-2) for ADDITIONAL_WORKER_VM_TYPE to reduce capacity pressure while maintaining sufficient resources.

Release 4.21 (Tau processor):

• COMPUTE_NODE_TYPE: t2a-standard-4 (unchanged, 4 vCPU)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → t2a-standard-2 (2 vCPU, smaller)
• MIGRATION_CP/INFRA_MACHINE_TYPE: t2a-standard-4 (unchanged, 4 vCPU)

Release 4.22 (newer Axion processor):

• COMPUTE_NODE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU, newer generation)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → c4a-standard-2 (2 vCPU, smaller)
• MIGRATION_CP/INFRA_MACHINE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU)

Release 4.23 (newest Axion):

• COMPUTE_NODE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → c4a-standard-2 (2 vCPU, smaller)
• MIGRATION_CP/INFRA_MACHINE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU)

Release 5.0 (same as 4.23):

• COMPUTE_NODE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → c4a-standard-2 (2 vCPU, smaller)
• MIGRATION_CP/INFRA_MACHINE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU)

Older releases (≤4.20): unchanged (t2a-standard-4)

Modified files:

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0*.yaml (3 files)
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh (1 file)

Benefits

1. Reduces capacity exhaustion by:

• Spreading load across multiple GCP zones (via random machineset selection)
• Using smaller instances for additional heterogeneous workers
• Distributing across multiple ARM instance types (T2A Tau and C4A Axion)

2. Improved availability:

• C4A (Axion) instances generally have better availability than T2A (Tau)
• Smaller instance types (standard-2) have exponentially better availability than standard-4
• Additional workers sized appropriately to reduce capacity pressure

3. Optimal resource allocation:

• Compute nodes maintain full resources (standard-4, 4 vCPU)
• Additional heterogeneous workers use smaller instances (standard-2, 2 vCPU) sufficient for testing

4. Cost optimization:

• Smaller instances for additional workers reduce costs
• Progressive migration to newer Axion processors

5. Forward compatibility:

• Newer releases (4.22, 4.23, 5.0) use newest Axion processors with best availability

Test Plan

• Monitor multi-arch GCP jobs across all affected releases (4.21, 4.22, 4.23, 5.0)
• Verify heterogeneous worker provisioning succeeds across different zone selections and instance types
• Watch for reduced "verify all machines should be in Running state" failures
• Confirm jobs complete successfully with the new instance type distribution

Related Issues

• Fixes Component Readiness regression: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/regressions/37839
• JIRA Bug filed: https://issues.redhat.com/browse/OCPBUGS-82060

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@jianlinliu: This pull request references Jira Issue OCPBUGS-82060, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Summary

This PR addresses Component Readiness regression 37839 where GCP multi-arch jobs were failing with "verify all machines should be in Running state" due to T2A ARM instance capacity exhaustion.

The test was failing across multiple GCP multi-arch variants (4.22, 4.21, and ARM64/multi architectures) with machines getting stuck in PROVISIONING state and then disappearing with "Instance not found on provider" errors. Root cause analysis showed GCP T2A ARM capacity exhaustion in us-central1-a zone.

Changes

1. Randomize zone selection in heterogeneous install step

• Modified ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh to randomly select a worker machineset instead of always using the first one
• Each machineset is in a different GCP zone, distributing load across zones to avoid zone-specific capacity issues
• Added logging to show which machineset index was selected

2. Distribute GCP ARM instance types across releases

Strategy: Use bigger instances (standard-4) for COMPUTE_NODE_TYPE and smaller instances (standard-2) for ADDITIONAL_WORKER_VM_TYPE to reduce capacity pressure while maintaining sufficient resources.

Release 4.21 (Tau processor):

• COMPUTE_NODE_TYPE: t2a-standard-4 (unchanged, 4 vCPU, Tau)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → t2a-standard-2 (2 vCPU, Tau, smaller)
• MIGRATION_CP/INFRA_MACHINE_TYPE: t2a-standard-4 (unchanged, 4 vCPU, Tau)

Release 4.22 (Axion processor):

• COMPUTE_NODE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU, Axion)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → c4a-standard-2 (2 vCPU, Axion, smaller)
• MIGRATION_CP/INFRA_MACHINE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU, Axion)

Release 4.23 (Axion processor):

• COMPUTE_NODE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU, Axion)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → c4a-standard-2 (2 vCPU, Axion, smaller)
• MIGRATION_CP/INFRA_MACHINE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU, Axion)

Release 5.0 (Axion processor):

• COMPUTE_NODE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU, Axion)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → c4a-standard-2 (2 vCPU, Axion, smaller)
• MIGRATION_CP/INFRA_MACHINE_TYPE: t2a-standard-4 → c4a-standard-4 (4 vCPU, Axion)

Older releases (≤4.20): Tau processor (t2a-standard-4, unchanged)

Modified files:

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0*.yaml (3 files)
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh (1 file)

Processor Distribution

• Tau (T2A): Releases 4.21 and older (≤4.20)
• Axion (C4A): Releases 4.22, 4.23, 5.0

Benefits

1. Reduces capacity exhaustion by:

• Spreading load across multiple GCP zones (via random machineset selection)
• Using smaller instances for additional heterogeneous workers
• Distributing across multiple ARM instance types (T2A Tau and C4A Axion)

2. Improved availability:

• C4A (Axion) instances generally have better availability than T2A (Tau)
• Smaller instance types (standard-2) have exponentially better availability than standard-4
• Additional workers sized appropriately to reduce capacity pressure

3. Optimal resource allocation:

• Compute nodes maintain full resources (standard-4, 4 vCPU)
• Additional heterogeneous workers use smaller instances (standard-2, 2 vCPU) sufficient for testing

4. Cost optimization:

• Smaller instances for additional workers reduce costs
• Progressive migration to newer Axion processors

5. Forward compatibility:

• Newer releases (4.22, 4.23, 5.0) use newest Axion processors with best availability

Test Plan

• Monitor multi-arch GCP jobs across all affected releases (4.21, 4.22, 4.23, 5.0)
• Verify heterogeneous worker provisioning succeeds across different zone selections and instance types
• Watch for reduced "verify all machines should be in Running state" failures
• Confirm jobs complete successfully with the new instance type distribution

Related Issues

• Fixes Component Readiness regression: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/regressions/37839
• JIRA Bug filed: https://issues.redhat.com/browse/OCPBUGS-82060

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jianlinliu]

/pj-rehearse periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-upgrade-gcp-ovn-multi-x-ax periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-gcp-ovn-multi-x-x-to-a-x

[openshift-ci-robot]

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[jianlinliu]

/pj-rehearse periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-upgrade-gcp-ovn-multi-x-ax periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-gcp-ovn-multi-x-x-to-a-x periodic-ci-openshift-multiarch-main-nightly-4.22-upgrade-from-stable-4.21-ocp-e2e-upgrade-gcp-ovn-multi-a-a periodic-ci-openshift-multiarch-main-nightly-4.17-upgrade-from-nightly-4.16-ocp-e2e-upgrade-gcp-ovn-multi-x-ax

[openshift-ci-robot]

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci-robot]

@jianlinliu: This pull request references Jira Issue OCPBUGS-82060, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Summary

This PR addresses Component Readiness regression 37839 where GCP multi-arch jobs were failing due to ARM instance capacity exhaustion and C4A disk type compatibility issues.

Root Causes

1. GCP T2A ARM capacity exhaustion in us-central1-a zone causing machines to get stuck in PROVISIONING state
2. C4A instances incompatible with pd-ssd disk type (require hyperdisk-balanced)
3. ipi-install-heterogeneous always used first machineset, concentrating load in single zone

Solutions

1. Zone Randomization

Random machineset selection in ipi-install-heterogeneous distributes ARM instances across zones to reduce capacity pressure.

2. Instance Type Distribution

Spread load across processor families:

• Releases 4.17-4.21: T2A (Tau) processor
• Releases 4.22-5.0: C4A (Axion) processor

3. Disk Type Compatibility

New parameters ensure C4A instances use hyperdisk-balanced:

• COMPUTE_DISK_TYPE: Applied to main workers via install-config.yaml
• ADDITIONAL_WORKER_DISK_TYPE: Applied to heterogeneous additional workers via machineset

Changes

ipi-install-heterogeneous Step

• Random machineset selection for zone distribution
• New ADDITIONAL_WORKER_DISK_TYPE parameter for GCP disk type configuration

ipi-conf-gcp Chain

• Include ipi-conf-gcp-osdisk-disktype step
• Remove duplicate refs from chains that already include ipi-conf-gcp

Multi-arch Configs (4.17-5.0)

Releases 4.17-4.21 (Tau processor, smaller additional workers):

• COMPUTE_NODE_TYPE: t2a-standard-4 (unchanged)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → t2a-standard-2 (smaller)

Releases 4.22-5.0 (Axion processor, smaller additional workers):

• COMPUTE_NODE_TYPE: t2a-standard-4 → c4a-standard-4
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → c4a-standard-2 (smaller)
• COMPUTE_DISK_TYPE: hyperdisk-balanced (for main workers)
• ADDITIONAL_WORKER_DISK_TYPE: hyperdisk-balanced (for heterogeneous workers)

MIGRATION jobs (unchanged):

• MIGRATION_CP_MACHINE_TYPE: t2a-standard-4 (unchanged)
• MIGRATION_INFRA_MACHINE_TYPE: t2a-standard-4 (unchanged)

Modified Files (32 total)

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.17*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.18*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.19*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.20*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0*.yaml (3 files)
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-ref.yaml
ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-chain.yaml
ci-operator/step-registry/cucushift/installer/rehearse/gcp/ipi/*-provision-chain.yaml (4 files)
ci-operator/step-registry/openshift/e2e/gcp/csi/custom-worker/openshift-e2e-gcp-csi-custom-worker-workflow.yaml

Benefits

1. Reduces capacity exhaustion by:

• Spreading load across multiple GCP zones (random machineset selection)
• Using smaller instances for additional heterogeneous workers
• Distributing across T2A (4.17-4.21) and C4A (4.22-5.0) processor families

2. Fixes C4A compatibility:

• Ensures C4A instances use hyperdisk-balanced disk type
• Prevents pd-ssd incompatibility errors

3. Optimal resource allocation:

• Compute nodes use larger instances (standard-4, 4 vCPU)
• Additional workers use smaller instances (standard-2, 2 vCPU)

4. Better availability:

• C4A (Axion) instances have better availability than T2A
• Smaller instance types have exponentially better availability

Related Issues

• Fixes Component Readiness regression: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/regressions/37839
• JIRA Bug: https://issues.redhat.com/browse/OCPBUGS-82060

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jianlinliu]

/pj-rehearse periodic-ci-openshift-multiarch-main-nightly-4.22-upgrade-from-stable-4.21-ocp-e2e-upgrade-gcp-ovn-multi-a-a

[openshift-ci-robot]

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci-robot]

@jianlinliu: This pull request references Jira Issue OCPBUGS-82060, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Summary

This PR addresses Component Readiness regression 37839 where GCP multi-arch jobs were failing due to ARM instance capacity exhaustion and C4A disk type compatibility issues.

Root Causes

1. GCP T2A ARM capacity exhaustion in us-central1-a zone causing machines to get stuck in PROVISIONING state
2. C4A instances incompatible with pd-ssd disk type (require hyperdisk-balanced)
3. ipi-install-heterogeneous always used first machineset, concentrating load in single zone

Solutions

1. Zone Randomization

Random machineset selection in ipi-install-heterogeneous distributes ARM instances across zones to reduce capacity pressure.

2. Instance Type Migration

Migrate all releases (4.17-5.0) to C4A (Axion) processor to spread load across processor families and improve availability.

3. Disk Type Compatibility

New parameters ensure C4A instances use hyperdisk-balanced:

• COMPUTE_DISK_TYPE: Applied to main workers via install-config.yaml
• ADDITIONAL_WORKER_DISK_TYPE: Applied to heterogeneous additional workers via machineset

4. Instance Sizing Optimization

Use smaller instances for additional workers and migration infra to reduce resource consumption and improve availability.

Changes

ipi-install-heterogeneous Step

• Random machineset selection for zone distribution
• New ADDITIONAL_WORKER_DISK_TYPE parameter for GCP disk type configuration

ipi-conf-gcp Chain

• Include ipi-conf-gcp-osdisk-disktype step
• Remove duplicate refs from chains that already include ipi-conf-gcp

Multi-arch Configs (4.17-5.0)

All Releases 4.17-5.0 (Axion processor with optimized sizing):

• COMPUTE_NODE_TYPE: t2a-standard-4 → c4a-standard-4
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → c4a-standard-2 (smaller instance)
• COMPUTE_DISK_TYPE: hyperdisk-balanced (for main workers)
• ADDITIONAL_WORKER_DISK_TYPE: hyperdisk-balanced (for heterogeneous workers)

MIGRATION jobs:

• MIGRATION_CP_MACHINE_TYPE: t2a-standard-4 (unchanged)
• MIGRATION_INFRA_MACHINE_TYPE: t2a-standard-4 → t2a-standard-2 (4.20-5.0, optimized)

Modified Files (32 total)

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.17*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.18*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.19*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.20*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0*.yaml (3 files)
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-ref.yaml
ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-chain.yaml
ci-operator/step-registry/cucushift/installer/rehearse/gcp/ipi/*-provision-chain.yaml (4 files)
ci-operator/step-registry/openshift/e2e/gcp/csi/custom-worker/openshift-e2e-gcp-csi-custom-worker-workflow.yaml

Benefits

1. Reduces capacity exhaustion by:

• Spreading load across multiple GCP zones (random machineset selection)
• Using smaller instances for additional heterogeneous workers and migration infra
• Migrating all releases to C4A (Axion) processor

2. Fixes C4A compatibility:

• Ensures C4A instances use hyperdisk-balanced disk type
• Prevents pd-ssd incompatibility errors

3. Optimal resource allocation:

• Compute nodes use larger instances (standard-4, 4 vCPU)
• Additional workers use smaller instances (standard-2, 2 vCPU)
• Migration infra uses smaller instances (standard-2, 2 vCPU)

4. Better availability:

• C4A (Axion) instances have better availability than T2A
• Smaller instance types have exponentially better availability
• Unified processor family (C4A) simplifies capacity planning

Related Issues

• Fixes Component Readiness regression: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/regressions/37839
• JIRA Bug: https://issues.redhat.com/browse/OCPBUGS-82060

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jianlinliu]

/pj-rehearse periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-gcp-ovn-multi-x-x-to-a-x periodic-ci-openshift-multiarch-main-nightly-4.17-upgrade-from-stable-4.16-ocp-e2e-upgrade-gcp-ovn-multi-a-a periodic-ci-openshift-multiarch-main-nightly-4.22-upgrade-from-stable-4.21-ocp-e2e-upgrade-gcp-ovn-multi-a-a

[openshift-ci-robot]

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci-robot]

@jianlinliu: job(s): periodic-ci-openshift-multiarch-main-nightly-4.17-upgrade-from-stable-4.16-ocp-e2e-upgrade-gcp-ovn-multi-a-a, periodic-ci-openshift-multiarch-main-nightly-4.22-upgrade-from-stable-4.21-ocp-e2e-upgrade-gcp-ovn-multi-a-a either don't exist or were not found to be affected, and cannot be rehearsed

[jianlinliu]

/pj-rehearse abort

[openshift-ci-robot]

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci-robot]

@jianlinliu: This pull request references Jira Issue OCPBUGS-82060, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Summary

This PR addresses Component Readiness regression 37839 where GCP multi-arch jobs were failing due to ARM instance capacity exhaustion through zone randomization, instance sizing optimization, balanced worker configuration, and schedule distribution.

Root Causes

1. GCP T2A ARM capacity exhaustion in us-central1-a zone causing machines to get stuck in PROVISIONING state
2. ipi-install-heterogeneous always used first machineset, concentrating load in single zone
3. Larger instances have lower availability - standard-4 instances more likely to hit capacity limits
4. Unbalanced worker configuration - heterogeneous jobs used 3 AMD64 + 2 ARM64 workers (5 total)
5. Schedule contention - 18 GCP ARM jobs all scheduled at Sunday 11:00 UTC, creating massive resource contention

Solutions

1. Zone Randomization

Random machineset selection in ipi-install-heterogeneous distributes ARM instances across zones to reduce capacity pressure.

2. Instance Sizing Optimization

Use smaller instances (standard-2) for additional workers and migration infra to improve availability and reduce resource consumption.

3. Balanced Worker Configuration

Set COMPUTE_NODE_REPLICAS: "2" for GCP heterogeneous jobs to create balanced 2+2 worker layout instead of unbalanced 3+2 configuration.

Before:

• 3 AMD64 workers (default COMPUTE_NODE_REPLICAS: 3)
• 2 ARM64 additional workers (default ADDITIONAL_WORKERS: 2)
• Total: 5 workers

After:

• 2 AMD64 workers (COMPUTE_NODE_REPLICAS: "2")
• 2 ARM64 additional workers (ADDITIONAL_WORKERS: 2)
• Total: 4 workers

Benefits:

• ✅ More balanced heterogeneous cluster (2+2 vs 3+2)
• ✅ Reduced total resource consumption (4 workers vs 5)
• ✅ Faster cluster installation (one less worker to provision)
• ✅ Lower cluster overhead (less kubelet/container runtime load)

4. Schedule Distribution

Change job scheduling from cron to interval-based (168h) for releases 4.20-5.0 to prevent simultaneous execution.

Before (Releases 4.19-5.0):

• All 18 jobs ran at Sunday 11:00 UTC (cron: 0 11 * * 0)
• 3 jobs per release × 6 releases = 18 concurrent jobs
• Massive resource contention spike every Sunday

After:

• 4.19: Keeps cron schedule (Sunday 11:00 UTC)
• 4.20-5.0: Changed to interval: 168h (weekly interval)
• Jobs naturally distribute based on completion times
• Prevents simultaneous resource contention

Benefits:

• ✅ Eliminates weekly resource contention spike
• ✅ Zone randomization more effective when jobs don't overlap
• ✅ Better capacity distribution throughout the week
• ✅ Reduced likelihood of hitting zone capacity limits

5. Why T2A (not C4A/N4A)?

C4A and N4A Compatibility Issue:

• C4A and N4A only support Hyperdisk, NOT Persistent Disk
• OpenShift monitoring, logging, and registry use pd-standard PVCs by default
• Rehearse testing with C4A failed with:

AttachVolume.Attach failed for volume "pvc-0ad4471f-65fe-40c6-8850-8768b0a91e07"
rpc error: code = InvalidArgument desc = Failed to Attach: failed cloud service
attach disk call: googleapi: Error 400: pd-standard disk type cannot be used by
c4a-standard-4 machine type., badRequest

T2A Advantages:

• ✅ Supports BOTH Persistent Disk (pd-standard, pd-balanced, pd-ssd) AND Hyperdisk
• ✅ Full OpenShift compatibility with default storage classes
• ✅ No breaking changes to monitoring, logging, or registry
• ✅ Proven reliability and stability

GCP ARM Instance Disk Support Comparison:

Instance Type	Persistent Disk	Hyperdisk	OpenShift Compatible
T2A (Tau)	✅ Yes (pd-standard, pd-balanced, pd-ssd)	✅ Yes	✅ Yes
C4A (Axion Compute)	❌ No	✅ Yes (hyperdisk-balanced, hyperdisk-extreme)	❌ No
N4A (Axion General)	❌ No	✅ Yes (hyperdisk-balanced, hyperdisk-throughput)	❌ No

Changes

ipi-install-heterogeneous Step

• Random machineset selection for zone distribution
• New ADDITIONAL_WORKER_DISK_TYPE parameter for GCP disk type configuration

Multi-arch Configs (4.17-5.0)

All Releases 4.17-5.0 (T2A processor with optimized sizing):

• COMPUTE_NODE_TYPE: t2a-standard-4 (unchanged, 4 vCPU)
• COMPUTE_NODE_REPLICAS: "2" (for GCP heterogeneous jobs, balanced layout)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → t2a-standard-2 (optimized to 2 vCPU)

MIGRATION jobs:

• MIGRATION_CP_MACHINE_TYPE: t2a-standard-4 (unchanged)
• MIGRATION_INFRA_MACHINE_TYPE: t2a-standard-4 → t2a-standard-2 (4.20-5.0, optimized to 2 vCPU)

Job Schedules:

• 4.19: cron: 0 11 * * 0 (Sunday 11:00 UTC, unchanged)
• 4.20-5.0: interval: 168h (weekly interval, distributed execution)

Modified Files (26 total)

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.17*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.18*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.19*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.20*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0*.yaml (3 files)
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-ref.yaml

Benefits

1. Reduces capacity exhaustion by:

• Spreading load across multiple GCP zones (random machineset selection)
• Using smaller instances (standard-2 vs standard-4) for additional workers and migration infra
• Smaller instances have exponentially better availability
• Balanced worker configuration reduces total resource consumption
• Distributed job execution prevents simultaneous resource spikes

2. Full OpenShift compatibility:

• T2A supports both Persistent Disk and Hyperdisk
• Works with default pd-standard storage class
• No changes required to monitoring, logging, or registry

3. Optimal resource allocation:

• Compute nodes use larger instances (t2a-standard-4, 4 vCPU)
• Additional workers use smaller instances (t2a-standard-2, 2 vCPU)
• Migration infra uses smaller instances (t2a-standard-2, 2 vCPU)
• GCP heterogeneous jobs: balanced 2+2 worker layout

4. Improved scheduling efficiency:

• Interval-based scheduling (168h) prevents simultaneous execution
• Jobs naturally distribute throughout the week
• Zone randomization more effective without overlapping demand
• Eliminates weekly resource contention spike

5. Better cluster efficiency:

• Faster cluster installation (4 workers vs 5)
• Lower cluster overhead
• More balanced heterogeneous cluster architecture

6. Future flexibility:

• ADDITIONAL_WORKER_DISK_TYPE parameter allows easy migration to Hyperdisk if needed
• If OpenShift defaults change to hyperdisk-balanced, can reconsider C4A/N4A

Related Issues

• Fixes Component Readiness regression: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/regressions/37839
• JIRA Bug: https://issues.redhat.com/browse/OCPBUGS-82060

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jianlinliu
Once this PR has been reviewed and has the lgtm label, please assign tvardema for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/multiarch/OWNERS
• ci-operator/jobs/openshift/multiarch/OWNERS
• ci-operator/step-registry/ipi/install/heterogeneous/OWNERS [jianlinliu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

[REHEARSALNOTIFIER]
@jianlinliu: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp420-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp419-e2e-azure	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp417-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp417-azure-mto-heterogeneous-perfscale	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp417-azure-heterogeneous-perfscale	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp416-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp418-e2e-azure	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp422-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-main-ocp421-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v1.x-ocp416-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v1.x-ocp420-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v1.x-ocp419-e2e-azure	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v1.x-ocp417-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v1.x-ocp421-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v0.0.1-e2e-gcp-multi-operator-olm	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v0.0.1-e2e-gcp-multi-operator	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v1.x-ocp422-e2e-gcp	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v1.x-ocp418-e2e-azure	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v0.9-e2e-gcp-multi-operator-olm	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-outrigger-project-multiarch-tuning-operator-v0.9-e2e-gcp-multi-operator	outrigger-project/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-openshift-multiarch-tuning-operator-main-ocp416-e2e-gcp	openshift/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-openshift-multiarch-tuning-operator-main-ocp420-e2e-gcp	openshift/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-openshift-multiarch-tuning-operator-main-ocp417-e2e-gcp	openshift/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-openshift-multiarch-tuning-operator-main-ocp417-azure-mto-heterogeneous-perfscale	openshift/multiarch-tuning-operator	presubmit	Registry content changed
pull-ci-openshift-multiarch-tuning-operator-main-ocp417-azure-heterogeneous-perfscale	openshift/multiarch-tuning-operator	presubmit	Registry content changed

A total of 773 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[jianlinliu]

/pj-rehearse periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-gcp-ovn-multi-x-x-to-a-x periodic-ci-openshift-multiarch-main-nightly-4.17-upgrade-from-stable-4.16-ocp-e2e-upgrade-gcp-ovn-multi-a-a periodic-ci-openshift-multiarch-main-nightly-4.22-upgrade-from-stable-4.21-ocp-e2e-upgrade-gcp-ovn-multi-a-a periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-upgrade-gcp-ovn-multi-x-ax

[openshift-ci-robot]

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci-robot]

@jianlinliu: job(s): periodic-ci-openshift-multiarch-main-nightly-4.17-upgrade-from-stable-4.16-ocp-e2e-upgrade-gcp-ovn-multi-a-a, periodic-ci-openshift-multiarch-main-nightly-4.22-upgrade-from-stable-4.21-ocp-e2e-upgrade-gcp-ovn-multi-a-a either don't exist or were not found to be affected, and cannot be rehearsed

[openshift-ci]

@jianlinliu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/periodic-ci-openshift-multiarch-main-nightly-4.22-upgrade-from-stable-4.21-ocp-e2e-upgrade-gcp-ovn-multi-a-a	28d3298	link	unknown	/pj-rehearse periodic-ci-openshift-multiarch-main-nightly-4.22-upgrade-from-stable-4.21-ocp-e2e-upgrade-gcp-ovn-multi-a-a
ci/rehearse/periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-gcp-ovn-multi-x-x-to-a-x	a289177	link	unknown	/pj-rehearse periodic-ci-openshift-multiarch-main-nightly-4.22-ocp-e2e-gcp-ovn-multi-x-x-to-a-x

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-bot]

Closing, please open a separate request if this work is still required.

[openshift-ci-robot]

@jianlinliu: This pull request references Jira Issue OCPBUGS-82060. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state.

Details

In response to this:

Summary

This PR addresses Component Readiness regression 37839 where GCP multi-arch jobs were failing due to ARM instance capacity exhaustion through zone randomization, instance sizing optimization, balanced worker configuration, and schedule distribution.

Root Causes

1. GCP T2A ARM capacity exhaustion in us-central1-a zone causing machines to get stuck in PROVISIONING state
2. ipi-install-heterogeneous always used first machineset, concentrating load in single zone
3. Larger instances have lower availability - standard-4 instances more likely to hit capacity limits
4. Unbalanced worker configuration - heterogeneous jobs used 3 AMD64 + 2 ARM64 workers (5 total)
5. Schedule contention - 18 GCP ARM jobs all scheduled at Sunday 11:00 UTC, creating massive resource contention

Solutions

1. Zone Randomization

Random machineset selection in ipi-install-heterogeneous distributes ARM instances across zones to reduce capacity pressure.

2. Instance Sizing Optimization

Use smaller instances (standard-2) for additional workers and migration infra to improve availability and reduce resource consumption.

3. Balanced Worker Configuration

Set COMPUTE_NODE_REPLICAS: "2" for GCP heterogeneous jobs to create balanced 2+2 worker layout instead of unbalanced 3+2 configuration.

Before:

• 3 AMD64 workers (default COMPUTE_NODE_REPLICAS: 3)
• 2 ARM64 additional workers (default ADDITIONAL_WORKERS: 2)
• Total: 5 workers

After:

• 2 AMD64 workers (COMPUTE_NODE_REPLICAS: "2")
• 2 ARM64 additional workers (ADDITIONAL_WORKERS: 2)
• Total: 4 workers

Benefits:

• ✅ More balanced heterogeneous cluster (2+2 vs 3+2)
• ✅ Reduced total resource consumption (4 workers vs 5)
• ✅ Faster cluster installation (one less worker to provision)
• ✅ Lower cluster overhead (less kubelet/container runtime load)

4. Schedule Distribution

Change job scheduling from cron to interval-based (168h) for releases 4.19-5.0 to prevent simultaneous execution.

Before:

• All 18 jobs ran at Sunday 11:00 UTC (cron: 0 11 * * 0)
• 3 jobs per release × 6 releases = 18 concurrent jobs
• Massive resource contention spike every Sunday

After:

• All releases 4.19-5.0: Changed to interval: 168h (weekly interval)
• Jobs naturally distribute based on completion times
• Prevents simultaneous resource contention

Benefits:

• ✅ Eliminates weekly resource contention spike
• ✅ Zone randomization more effective when jobs don't overlap
• ✅ Better capacity distribution throughout the week
• ✅ Reduced likelihood of hitting zone capacity limits

5. Why T2A (not C4A/N4A)?

C4A and N4A Compatibility Issue:

• C4A and N4A only support Hyperdisk, NOT Persistent Disk
• OpenShift monitoring, logging, and registry use pd-standard PVCs by default
• Rehearse testing with C4A failed with:

AttachVolume.Attach failed for volume "pvc-0ad4471f-65fe-40c6-8850-8768b0a91e07"
rpc error: code = InvalidArgument desc = Failed to Attach: failed cloud service
attach disk call: googleapi: Error 400: pd-standard disk type cannot be used by
c4a-standard-4 machine type., badRequest

T2A Advantages:

• ✅ Supports BOTH Persistent Disk (pd-standard, pd-balanced, pd-ssd) AND Hyperdisk
• ✅ Full OpenShift compatibility with default storage classes
• ✅ No breaking changes to monitoring, logging, or registry
• ✅ Proven reliability and stability

GCP ARM Instance Disk Support Comparison:

Instance Type	Persistent Disk	Hyperdisk	OpenShift Compatible
T2A (Tau)	✅ Yes (pd-standard, pd-balanced, pd-ssd)	✅ Yes	✅ Yes
C4A (Axion Compute)	❌ No	✅ Yes (hyperdisk-balanced, hyperdisk-extreme)	❌ No
N4A (Axion General)	❌ No	✅ Yes (hyperdisk-balanced, hyperdisk-throughput)	❌ No

Changes

ipi-install-heterogeneous Step

• Random machineset selection for zone distribution
• New ADDITIONAL_WORKER_DISK_TYPE parameter for GCP disk type configuration

Multi-arch Configs (4.17-5.0)

All Releases 4.17-5.0 (T2A processor with optimized sizing):

• COMPUTE_NODE_TYPE: t2a-standard-4 (unchanged, 4 vCPU)
• COMPUTE_NODE_REPLICAS: "2" (for GCP heterogeneous jobs, balanced layout)
• ADDITIONAL_WORKER_VM_TYPE: t2a-standard-4 → t2a-standard-2 (optimized to 2 vCPU)

MIGRATION jobs:

• MIGRATION_CP_MACHINE_TYPE: t2a-standard-4 (unchanged)
• MIGRATION_INFRA_MACHINE_TYPE: t2a-standard-4 → t2a-standard-2 (4.20-5.0, optimized to 2 vCPU)

Job Schedules:

• 4.19-5.0: Changed to interval: 168h (weekly interval, distributed execution)
• Eliminates simultaneous execution and resource contention

Modified Files (27 total)

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.17*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.18*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.19*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.20*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23*.yaml (3 files)
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0*.yaml (3 files)
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-commands.sh
ci-operator/step-registry/ipi/install/heterogeneous/ipi-install-heterogeneous-ref.yaml
ci-operator/jobs/openshift/multiarch/openshift-multiarch-main-periodics.yaml

Benefits

1. Reduces capacity exhaustion by:

• Spreading load across multiple GCP zones (random machineset selection)
• Using smaller instances (standard-2 vs standard-4) for additional workers and migration infra
• Smaller instances have exponentially better availability
• Balanced worker configuration reduces total resource consumption
• Distributed job execution prevents simultaneous resource spikes

2. Full OpenShift compatibility:

• T2A supports both Persistent Disk and Hyperdisk
• Works with default pd-standard storage class
• No changes required to monitoring, logging, or registry

3. Optimal resource allocation:

• Compute nodes use larger instances (t2a-standard-4, 4 vCPU)
• Additional workers use smaller instances (t2a-standard-2, 2 vCPU)
• Migration infra uses smaller instances (t2a-standard-2, 2 vCPU)
• GCP heterogeneous jobs: balanced 2+2 worker layout

4. Improved scheduling efficiency:

• Interval-based scheduling (168h) prevents simultaneous execution
• Jobs naturally distribute throughout the week
• Zone randomization more effective without overlapping demand
• Eliminates weekly resource contention spike

5. Better cluster efficiency:

• Faster cluster installation (4 workers vs 5)
• Lower cluster overhead
• More balanced heterogeneous cluster architecture

6. Future flexibility:

• ADDITIONAL_WORKER_DISK_TYPE parameter allows easy migration to Hyperdisk if needed
• If OpenShift defaults change to hyperdisk-balanced, can reconsider C4A/N4A

Related Issues

• Fixes Component Readiness regression: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/regressions/37839
• JIRA Bug: https://issues.redhat.com/browse/OCPBUGS-82060

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.