feat(aro-hcp): add dev-ci Unprivileged postsubmit and fix pipeline concurrency

[roivaz]

What

Add a Prow postsubmit (dev-ci-pipeline-postsubmit) that rolls out the Microsoft.Azure.ARO.HCP.DevCI.Unprivileged entrypoint of the ARO-HCP dev-ci topology via a new aro-hcp/provision/dev-ci registry step.

Also fixes two pre-existing postsubmit concurrency bugs.

Why

New dev-ci job. ARO-HCP now has a dev-ci topology split into two entrypoints: Unprivileged (unattended, no subscription Owner needed) and Privileged (Owner-only, on-demand). This job continuously reconciles the Unprivileged shared dev/CI support infrastructure (opstool AKS, gateway, cert-manager, tenant-quota, CIHealth, and the non-privileged CI bot identities). The Owner-only Privileged entrypoint is intentionally not run here — its service principal is not a subscription Owner — and is applied on demand by an OWNERS member.

• max_concurrency: 1 with no Boskos lease: concurrency=1 alone serializes executions, so no quota slice is needed.
• run_if_changed scoped to the entrypoint's inputs: config/config-dev-ci.yaml, config/config-dev-ci.schema.json, topology-dev-ci.yaml, dev-infrastructure/.*, tooling/.* (covers all shared bicep templates/configurations/scripts/observability and tenant-quota that the entrypoint deploys).

Concurrency fixes. #78576 set max_concurrency: 4 on every postsubmit. Both ARO-HCP pipeline postsubmits must stay serialized, so they are restored to max_concurrency: 1. Since concurrency=1 already provides the serialization their Boskos leases did, both leases and their now-unused quota slices are dropped from the boskos generator:

• global-pipeline-postsubmit: max_concurrency 4 → 1; remove lease + aro-hcp-dev-global-pipeline-quota-slice.
• cspr-pipeline-postsubmit: max_concurrency 4 → 1; remove lease + aro-hcp-dev-cspr-pipeline-quota-slice. At 4, fast-landing commits on ARO-HCP main triggered concurrent CSPR pipeline executions.

Dependencies

Depends on Azure/ARO-HCP#6148 (introduces topology-dev-ci.yaml and the two entrypoints). Holding until it lands.

/hold
/cc @geoberle

Summary by CodeRabbit

• Adds a dev-ci-pipeline-postsubmit Prow job (with max_concurrency: 1) that deploys the ARO-HCP Microsoft.Azure.ARO.HCP.DevCI.Unprivileged entrypoint via a new aro-hcp/provision/dev-ci registry step, and runs only when dev-ci configuration/topology/infrastructure/tooling inputs change; the owner-only Microsoft.Azure.ARO.HCP.DevCI.Privileged entrypoint is explicitly excluded. Held pending feat: bump CIHealth and split dev-ci e2e RBAC pipeline by privilege Azure/ARO-HCP#6148.
• Serializes existing postsubmit jobs by setting max_concurrency: 1 for both global-pipeline-postsubmit and cspr-pipeline-postsubmit, removing their Boskos lease usage.
• Updates Boskos quota slices for ARO-HCP dev by removing the global and CSPR pipeline quota slices and adding an aro-hcp-dev-hypershift-westus3-slot quota slice.
• Introduces the new step-registry implementation for DevCI provisioning, including ownership metadata, the aro-hcp-provision-dev-ci reference YAML, and aro-hcp-provision-dev-ci-commands.sh (Azure credentialed az login + dev-ci-local-run execution).

[coderabbitai]

Walkthrough

Adds an ARO-HCP dev-ci provisioning step, connects it to a new postsubmit job, limits selected postsubmit concurrency, removes related leases, and replaces Boskos quota resources with a Hypershift westus3 slot.

Changes

ARO-HCP dev-ci integration

Layer / File(s)	Summary
Postsubmit concurrency and Boskos resources ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main.yaml, core-services/prow/02_config/_boskos.yaml, core-services/prow/02_config/generate-boskos.py	Limits global and CSPR postsubmit concurrency to one, removes their lease configurations, removes the global pipeline quota definition, and adds a Hypershift westus3 slot resource.
Dev-ci provisioning step ci-operator/step-registry/aro-hcp/provision/dev-ci/*	Adds ownership and step metadata, mounts credentials, authenticates with Azure service-principal values, and runs the unprivileged dev-ci entrypoint.
Dev-ci postsubmit wiring ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main.yaml	Adds a filtered postsubmit job that runs aro-hcp-provision-dev-ci with single-job concurrency.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PostsubmitJob
  participant ProvisionStep
  participant ProvisionCommands
  participant Azure
  participant DevCI
  PostsubmitJob->>ProvisionStep: invoke aro-hcp-provision-dev-ci
  ProvisionStep->>ProvisionCommands: run command script
  ProvisionCommands->>Azure: authenticate with service principal
  ProvisionCommands->>DevCI: run dev-ci-local-run

Loading

Suggested reviewers: geoberle

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	No Ginkgo test titles were added; touched files are config/shell/OWNERS only, with no Go test files or It/Describe/Context/When matches.
Test Structure And Quality	✅ Passed	The PR only changes CI config, step-registry, and Boskos files; no Ginkgo test code or test blocks were modified.
Microshift Test Compatibility	✅ Passed	No Ginkgo e2e tests were added or changed; only CI config, registry, and scripts were modified, so MicroShift compatibility is not applicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; this PR only changes CI config and step-registry scripts, so SNO compatibility review is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	Only Prow/step-registry/Boskos config changed; no deployment manifests, operators, controllers, or pod-scheduling constraints were introduced.
Ote Binary Stdout Contract	✅ Passed	No OTE binary entrypoint code changed; the diff is CI config plus a shell wrapper with no stdout prints in process-level code.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo tests were added; the PR only adds CI wiring and a shell step using cluster creds/internal tooling, with no IPv4-only or public-host references.
No-Weak-Crypto	✅ Passed	No MD5/SHA1/DES/RC4/3DES/Blowfish/ECB or custom secret comparisons appear in the changed files; the diff is CI config and a shell wrapper.
Container-Privileges	✅ Passed	The PR adds the dev-ci job/step registry entry, but the diff shows no privileged, hostPID/Network/IPC, SYS_ADMIN, or allowPrivilegeEscalation settings in manifests.
No-Sensitive-Data-In-Logs	✅ Passed	No secret-bearing logging found: the new script uses no echo/printf/set -x and runs az login with --output none after reading creds into env vars.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: a new dev-ci Unprivileged postsubmit and concurrency fixes for existing pipelines.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@roivaz: no rehearsable tests are affected by this change

Note: If this PR includes changes to step registry files (ci-operator/step-registry/) and you expected jobs to be found, try rebasing your PR onto the base branch. This helps pj-rehearse accurately detect changes when the base branch has moved forward.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[roivaz]

/hold cancel

[janboll]

/lgtm
/approve

[danilo-gemoli]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danilo-gemoli, janboll, roivaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/Azure/ARO-HCP/OWNERS [danilo-gemoli,janboll,roivaz]
• ci-operator/jobs/Azure/ARO-HCP/OWNERS [danilo-gemoli,janboll,roivaz]
• ci-operator/step-registry/aro-hcp/provision/OWNERS [danilo-gemoli,janboll,roivaz]
• core-services/prow/02_config/OWNERS [danilo-gemoli]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[roivaz]

/retest

[openshift-ci]

@roivaz: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

@roivaz: Updated the resources configmap in namespace ci at cluster app.ci using the following files:

• key boskos.yaml using file core-services/prow/02_config/_boskos.yaml

Details

In response to this:

What

Add a Prow postsubmit (dev-ci-pipeline-postsubmit) that rolls out the Microsoft.Azure.ARO.HCP.DevCI.Unprivileged entrypoint of the ARO-HCP dev-ci topology via a new aro-hcp/provision/dev-ci registry step.

Also fixes two pre-existing postsubmit concurrency bugs.

Why

New dev-ci job. ARO-HCP now has a dev-ci topology split into two entrypoints: Unprivileged (unattended, no subscription Owner needed) and Privileged (Owner-only, on-demand). This job continuously reconciles the Unprivileged shared dev/CI support infrastructure (opstool AKS, gateway, cert-manager, tenant-quota, CIHealth, and the non-privileged CI bot identities). The Owner-only Privileged entrypoint is intentionally not run here — its service principal is not a subscription Owner — and is applied on demand by an OWNERS member.

• max_concurrency: 1 with no Boskos lease: concurrency=1 alone serializes executions, so no quota slice is needed.
• run_if_changed scoped to the entrypoint's inputs: config/config-dev-ci.yaml, config/config-dev-ci.schema.json, topology-dev-ci.yaml, dev-infrastructure/.*, tooling/.* (covers all shared bicep templates/configurations/scripts/observability and tenant-quota that the entrypoint deploys).

Concurrency fixes. #78576 set max_concurrency: 4 on every postsubmit. Both ARO-HCP pipeline postsubmits must stay serialized, so they are restored to max_concurrency: 1. Since concurrency=1 already provides the serialization their Boskos leases did, both leases and their now-unused quota slices are dropped from the boskos generator:

• global-pipeline-postsubmit: max_concurrency 4 → 1; remove lease + aro-hcp-dev-global-pipeline-quota-slice.
• cspr-pipeline-postsubmit: max_concurrency 4 → 1; remove lease + aro-hcp-dev-cspr-pipeline-quota-slice. At 4, fast-landing commits on ARO-HCP main triggered concurrent CSPR pipeline executions.

Dependencies

Depends on Azure/ARO-HCP#6148 (introduces topology-dev-ci.yaml and the two entrypoints). Holding until it lands.

/hold
/cc @geoberle

Summary by CodeRabbit

• Adds a dev-ci-pipeline-postsubmit Prow job (with max_concurrency: 1) that deploys the ARO-HCP Microsoft.Azure.ARO.HCP.DevCI.Unprivileged entrypoint via a new aro-hcp/provision/dev-ci registry step, and runs only when dev-ci configuration/topology/infrastructure/tooling inputs change; the owner-only Microsoft.Azure.ARO.HCP.DevCI.Privileged entrypoint is explicitly excluded. Held pending feat: bump CIHealth and split dev-ci e2e RBAC pipeline by privilege Azure/ARO-HCP#6148.
• Serializes existing postsubmit jobs by setting max_concurrency: 1 for both global-pipeline-postsubmit and cspr-pipeline-postsubmit, removing their Boskos lease usage.
• Updates Boskos quota slices for ARO-HCP dev by removing the global and CSPR pipeline quota slices and adding an aro-hcp-dev-hypershift-westus3-slot quota slice.
• Introduces the new step-registry implementation for DevCI provisioning, including ownership metadata, the aro-hcp-provision-dev-ci reference YAML, and aro-hcp-provision-dev-ci-commands.sh (Azure credentialed az login + dev-ci-local-run execution).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.