Skip to content

chore(ci): migrate fic-using workflows to ADO pipelines - #2536

Merged
Alex Castilio (alexcastilio) merged 11 commits into
mainfrom
ci/migrate-fic
Jul 16, 2026
Merged

chore(ci): migrate fic-using workflows to ADO pipelines#2536
Alex Castilio (alexcastilio) merged 11 commits into
mainfrom
ci/migrate-fic

Conversation

@alexcastilio

Copy link
Copy Markdown
Contributor

Description

This PR migrates all GH workflows using FIC to Azure Pipelines due to internal policy changes regarding FIC in GH.
It adds 3 pipelines under .azure-pipelines that mirrors today's GH workflows:

ADO Pipeline Mirrors
ci.yml images.yaml (build + manifests + e2e + perf-basics + perf-advanced on merge-queue)
scale.yml daily-scale-test.yaml + scale-test.yaml (daily cron + manual)
perf.yml perf-schedule.yaml + perf-manual.yaml + perf-template.yaml (twice-daily cron + manual)

Bash bodies inside each AzureCLI@2 task are copied verbatim from the existing GH workflows. Only the outer scaffolding is rewritten to ADO YAML schema (different from GH Actions YAML)

Also added:

  • setup-go.yml: parses the Go version from go.mod (ADO's GoTool@0 doesn't natively support from go.mod like GH's setup-go@v6).
  • free-disk-space.yml: reclaims ~30GB of pre-installed toolchains on agents (needed to avoid lack of space for large image builds).

Nothing in workflows is deleted or disabled by this PR. Both pipelines are available to run in parallel at this stage. A new PR will be raised to remove old workflows.

What this PR does NOT do (deferred to a follow-up PR)

  • Branch protection changes. Required checks still point at the GH workflows. Once ADO pipelines are validated, a subsequent admin action + PR will:
    • Add workflows to required status checks.
    • Remove the corresponding GH Actions checks.
    • Delete the FIC-using workflow files: images.yaml, e2e.yaml, perf-template.yaml, scale-test.yaml, daily-scale-test.yaml, perf-manual.yaml, perf-schedule.yaml.
  • Secret / var cleanup.

Behavior parity with today's GH workflows

The ADO pipelines match today's semantics exactly on all these axes:

  • Triggers. ci.yml fires on merge-queue pushes (gh-readonly-queue/main/*) and PRs targeting main. Scale/perf fire on their existing crons.
  • Merge-queue gating. ACR push, APP_INSIGHTS_ID embedding, and the manifests/e2e/perf jobs only run when isMergeGroup=True (mirrors GH's if: ${{ github.event_name == 'merge_group' }} gate). PR builds and manual queues run as dry-run.
  • Fork PRs. The three pipelines are configured with "Disable building pull requests from forked GitHub repositories" — same fork-safety posture as today's FIC-gated jobs (which never ran on fork PRs).
  • Image tag. Both build and consumer use $(make version). The migration proactively fixed a latent tag-mismatch bug (see below).

Required side-effect changes to retina source

Because ADO doesn't have ARM64 agents, arm64 image builds have to cross-compile from amd64 hosts. Retina's Dockerfiles have latent bugs / assumptions that only surface on cross-compile (native ARM64 CI on GH has always masked them since PR #2059 introduced systemcrypto/CGO). This PR includes minimal fixes:

Change Reason
operator/Dockerfile, cli/Dockerfile: dropped --platform=$BUILDPLATFORM from final image stages Otherwise arm64-built binaries got copied into amd64 base images (wrong arch → exec format error).
operator/Dockerfile, cli/Dockerfile, controller/Dockerfile: added ENV GOEXPERIMENT=ms_nocgo_opensslcrypto in builder stages Microsoft Go's default systemcrypto requires CGO_ENABLED=1, but Go auto-disables CGO when cross-compiling. This experiment (documented in Microsoft Go MigrationGuide) enables the cgo-less OpenSSL backend so systemcrypto works with CGO_ENABLED=0. Auto-selected in Go 1.27+; env var can be removed then.
operator/Dockerfile: RUN make manifests wrapped in env -u GOOS -u GOARCH make manifests shells out to controller-gen — a build-time host tool, not the target binary. Without unset, it cross-compiled to arm64 and failed to exec on the amd64 builder.
controller/Dockerfile: split intermediate into two stages — new bpf-gen at $TARGETPLATFORM, existing intermediate stays at $BUILDPLATFORM Retina's //go:generate bpf2go -target ${GOARCH} directives require target-arch execution (bpf2go invokes clang; can't cleanly cross-compile bpf2go itself). Split isolates that one requirement to a small emulated stage while keeping the Go build fast at $BUILDPLATFORM. Restores fast arm64 build without breaking bpf .o file generation.

Impact on amd64 builds: zero — all these changes are inert when host == target (CGO stays enabled, no cross-compile, bpf-gen runs natively).

Tag consistency fix

The migration surfaced a pre-existing latent bug: make retina-image / make manifest didn't have TAG set by CI, so the Makefile fell back to git describe --tags --always (produces v1.2.2-76-geade8685), while all consumers (e2e, perf-*, scale) used make version (produces the 7-char SHA). Images were pushed under one tag, consumers looked for another. GH masked it because actions/checkout defaults to shallow (no tags, so git describe also falls back to short SHA); ADO defaults to full-clone.

ADO pipelines now pass TAG=$(make version) explicitly to every make invocation, ensuring builder and consumer speak the same tag string on both platforms.

Temporary items in this PR (will be reverted after cutover)

  • forceMergeGroup pipeline parameter in ci.yml. Boolean checkbox in the "Run pipeline" UI that treats a manual queue as if it were a merge-queue event (push + telemetry + full test suite). Default false, safe. Used during dark-launch to validate the full flow without needing an actual merge-queue event. Revert once we've moved past validation.

Verification

  • manual run of pipelines completed successfully

Checklist

  • I have read the contributing documentation.
  • I signed and signed-off the commits (git commit -S -s ...). See this documentation on signing commits.
  • I have correctly attributed the author(s) of the code.
  • I have tested the changes locally.
  • I have followed the project's style guidelines.
  • I have updated the documentation, if necessary.
  • I have added tests, if applicable.

Screenshots (if applicable) or Testing Completed

Please add any relevant screenshots or GIFs to showcase the changes made.

Additional Notes

Add any additional notes or context about the pull request here.


Please refer to the CONTRIBUTING.md file for more information on how to contribute to this project.

…C migration

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…er cutover)

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…mpile

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
@alexcastilio Alex Castilio (alexcastilio) changed the title Ci/migrate fic ci: migrate fic-using workflows to ADO pipelines Jul 14, 2026
@alexcastilio Alex Castilio (alexcastilio) changed the title ci: migrate fic-using workflows to ADO pipelines chore(ci): migrate fic-using workflows to ADO pipelines Jul 14, 2026
@github-actions

Copy link
Copy Markdown

Retina Code Coverage Report

Total coverage no change

Increased diff

Impacted Files Coverage
pkg/controllers/operator/retinaendpoint/retinaendpoint_controller.go 82.25% ... 83.28% (1.03%) ⬆️

@ritwikranjan

Copy link
Copy Markdown
Contributor

over all lgtm, potentially a safer migration in my opinion would be using templates which can be reused and is independent of platform. So for all these pipelines you can have one single template referenced for both github actions and ado

@alexcastilio
Alex Castilio (alexcastilio) added this pull request to the merge queue Jul 16, 2026
Merged via the queue into main with commit 5811a0f Jul 16, 2026
42 of 44 checks passed
@alexcastilio
Alex Castilio (alexcastilio) deleted the ci/migrate-fic branch July 16, 2026 11:01
Laksh (lakshk98) pushed a commit to lakshk98/retina that referenced this pull request Aug 4, 2026
# Description

This PR migrates all GH workflows using FIC to Azure Pipelines due to
internal policy changes regarding FIC in GH.
It adds 3 pipelines under `.azure-pipelines` that mirrors today's GH
workflows:

| ADO Pipeline | Mirrors |
| :--- | :--- |
| `ci.yml` | `images.yaml` (build + manifests + e2e + perf-basics +
perf-advanced on merge-queue) |
| `scale.yml` | `daily-scale-test.yaml` + `scale-test.yaml` (daily cron
+ manual) |
| `perf.yml` | `perf-schedule.yaml` + `perf-manual.yaml` +
`perf-template.yaml` (twice-daily cron + manual) |

Bash bodies inside each `AzureCLI@2` task are copied verbatim from the
existing GH workflows. Only the outer scaffolding is rewritten to ADO
YAML schema (different from GH Actions YAML)

Also added:
- `setup-go.yml`: parses the Go version from `go.mod` (ADO's `GoTool@0`
doesn't natively support `from go.mod` like GH's `setup-go@v6`).
- `free-disk-space.yml`: reclaims ~30GB of pre-installed toolchains on
agents (needed to avoid lack of space for large image builds).

> Nothing in `workflows` is deleted or disabled by this PR. Both
pipelines are available to run in parallel at this stage. A new PR will
be raised to remove old workflows.

### What this PR does NOT do (deferred to a follow-up PR)
- Branch protection changes. Required checks still point at the GH
workflows. Once ADO pipelines are validated, a subsequent admin action +
PR will:
  - Add workflows to required status checks.
  - Remove the corresponding GH Actions checks.
- Delete the FIC-using workflow files: `images.yaml`, `e2e.yaml`,
`perf-template.yaml`, `scale-test.yaml`, `daily-scale-test.yaml`,
`perf-manual.yaml`, `perf-schedule.yaml`.
- Secret / var cleanup.

### Behavior parity with today's GH workflows

The ADO pipelines match today's semantics exactly on all these axes:

- Triggers. `ci.yml` fires on merge-queue pushes
(`gh-readonly-queue/main/*`) and PRs targeting main. Scale/perf fire on
their existing crons.
- Merge-queue gating. ACR push, APP_INSIGHTS_ID embedding, and the
manifests/e2e/perf jobs only run when isMergeGroup=True (mirrors GH's
if: ${{ github.event_name == 'merge_group' }} gate). PR builds and
manual queues run as dry-run.
- Fork PRs. The three pipelines are configured with "Disable building
pull requests from forked GitHub repositories" — same fork-safety
posture as today's FIC-gated jobs (which never ran on fork PRs).
- Image tag. Both build and consumer use $(make version). The migration
proactively fixed a latent tag-mismatch bug (see below).

### Required side-effect changes to retina source

Because ADO doesn't have ARM64 agents, arm64 image builds have to
cross-compile from amd64 hosts. Retina's Dockerfiles have latent bugs /
assumptions that only surface on cross-compile (native ARM64 CI on GH
has always masked them since PR microsoft#2059 introduced systemcrypto/CGO). This
PR includes minimal fixes:

| Change | Reason |
| :--- | :--- | 
| `operator/Dockerfile`, `cli/Dockerfile`: dropped
`--platform=$BUILDPLATFORM` from final image stages | Otherwise
arm64-built binaries got copied into amd64 base images (wrong arch →
exec format error). |
| `operator/Dockerfile`, `cli/Dockerfile`, `controller/Dockerfile`:
added `ENV GOEXPERIMENT=ms_nocgo_opensslcrypto` in builder stages |
Microsoft Go's default `systemcrypto` requires `CGO_ENABLED=1`, but Go
auto-disables CGO when cross-compiling. This experiment (documented in
Microsoft Go MigrationGuide) enables the cgo-less OpenSSL backend so
systemcrypto works with `CGO_ENABLED=0`. Auto-selected in Go 1.27+; env
var can be removed then. |
| `operator/Dockerfile`: `RUN make manifests` wrapped in `env -u GOOS -u
GOARCH` | `make manifests` shells out to `controller-gen` — a build-time
host tool, not the target binary. Without unset, it cross-compiled to
arm64 and failed to `exec` on the amd64 builder. |
| `controller/Dockerfile`: split intermediate into two stages — new
`bpf-gen` at `$TARGETPLATFORM`, existing intermediate stays at
`$BUILDPLATFORM` | Retina's `//go:generate bpf2go -target ${GOARCH}`
directives require target-arch execution (bpf2go invokes clang; can't
cleanly cross-compile bpf2go itself). Split isolates that one
requirement to a small emulated stage while keeping the Go build fast at
$BUILDPLATFORM. Restores fast arm64 build without breaking bpf .o file
generation. |

> Impact on amd64 builds: zero — all these changes are inert when host
== target (CGO stays enabled, no cross-compile, bpf-gen runs natively).

### Tag consistency fix
The migration surfaced a pre-existing latent bug: `make retina-image` /
`make manifest` didn't have `TAG` set by CI, so the Makefile fell back
to `git describe --tags --always` (produces `v1.2.2-76-geade8685`),
while all consumers (`e2e`, `perf-*`, `scale`) used make version
(produces the 7-char SHA). Images were pushed under one tag, consumers
looked for another. GH masked it because actions/checkout defaults to
shallow (no tags, so git describe also falls back to short SHA); ADO
defaults to full-clone.

ADO pipelines now pass `TAG=$(make version)` explicitly to every `make`
invocation, ensuring builder and consumer speak the same tag string on
both platforms.

### Temporary items in this PR (will be reverted after cutover)

- `forceMergeGroup` pipeline parameter in `ci.yml`. Boolean checkbox in
the "Run pipeline" UI that treats a manual queue as if it were a
merge-queue event (push + telemetry + full test suite). Default `false`,
safe. Used during dark-launch to validate the full flow without needing
an actual merge-queue event. Revert once we've moved past validation.

### Verification
- manual run of pipelines completed successfully

## Checklist

- [ ] I have read the [contributing
documentation](https://retina.sh/docs/Contributing/overview).
- [ ] I signed and signed-off the commits (`git commit -S -s ...`). See
[this
documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification)
on signing commits.
- [ ] I have correctly attributed the author(s) of the code.
- [ ] I have tested the changes locally.
- [ ] I have followed the project's style guidelines.
- [ ] I have updated the documentation, if necessary.
- [ ] I have added tests, if applicable.

## Screenshots (if applicable) or Testing Completed

Please add any relevant screenshots or GIFs to showcase the changes
made.

## Additional Notes

Add any additional notes or context about the pull request here.

---

Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more
information on how to contribute to this project.

---------

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Laksh (lakshk98) pushed a commit to lakshk98/retina that referenced this pull request Aug 4, 2026
# Description

This PR migrates all GH workflows using FIC to Azure Pipelines due to
internal policy changes regarding FIC in GH.
It adds 3 pipelines under `.azure-pipelines` that mirrors today's GH
workflows:

| ADO Pipeline | Mirrors |
| :--- | :--- |
| `ci.yml` | `images.yaml` (build + manifests + e2e + perf-basics +
perf-advanced on merge-queue) |
| `scale.yml` | `daily-scale-test.yaml` + `scale-test.yaml` (daily cron
+ manual) |
| `perf.yml` | `perf-schedule.yaml` + `perf-manual.yaml` +
`perf-template.yaml` (twice-daily cron + manual) |

Bash bodies inside each `AzureCLI@2` task are copied verbatim from the
existing GH workflows. Only the outer scaffolding is rewritten to ADO
YAML schema (different from GH Actions YAML)

Also added:
- `setup-go.yml`: parses the Go version from `go.mod` (ADO's `GoTool@0`
doesn't natively support `from go.mod` like GH's `setup-go@v6`).
- `free-disk-space.yml`: reclaims ~30GB of pre-installed toolchains on
agents (needed to avoid lack of space for large image builds).

> Nothing in `workflows` is deleted or disabled by this PR. Both
pipelines are available to run in parallel at this stage. A new PR will
be raised to remove old workflows.

### What this PR does NOT do (deferred to a follow-up PR)
- Branch protection changes. Required checks still point at the GH
workflows. Once ADO pipelines are validated, a subsequent admin action +
PR will:
  - Add workflows to required status checks.
  - Remove the corresponding GH Actions checks.
- Delete the FIC-using workflow files: `images.yaml`, `e2e.yaml`,
`perf-template.yaml`, `scale-test.yaml`, `daily-scale-test.yaml`,
`perf-manual.yaml`, `perf-schedule.yaml`.
- Secret / var cleanup.

### Behavior parity with today's GH workflows

The ADO pipelines match today's semantics exactly on all these axes:

- Triggers. `ci.yml` fires on merge-queue pushes
(`gh-readonly-queue/main/*`) and PRs targeting main. Scale/perf fire on
their existing crons.
- Merge-queue gating. ACR push, APP_INSIGHTS_ID embedding, and the
manifests/e2e/perf jobs only run when isMergeGroup=True (mirrors GH's
if: ${{ github.event_name == 'merge_group' }} gate). PR builds and
manual queues run as dry-run.
- Fork PRs. The three pipelines are configured with "Disable building
pull requests from forked GitHub repositories" — same fork-safety
posture as today's FIC-gated jobs (which never ran on fork PRs).
- Image tag. Both build and consumer use $(make version). The migration
proactively fixed a latent tag-mismatch bug (see below).

### Required side-effect changes to retina source

Because ADO doesn't have ARM64 agents, arm64 image builds have to
cross-compile from amd64 hosts. Retina's Dockerfiles have latent bugs /
assumptions that only surface on cross-compile (native ARM64 CI on GH
has always masked them since PR microsoft#2059 introduced systemcrypto/CGO). This
PR includes minimal fixes:

| Change | Reason |
| :--- | :--- | 
| `operator/Dockerfile`, `cli/Dockerfile`: dropped
`--platform=$BUILDPLATFORM` from final image stages | Otherwise
arm64-built binaries got copied into amd64 base images (wrong arch →
exec format error). |
| `operator/Dockerfile`, `cli/Dockerfile`, `controller/Dockerfile`:
added `ENV GOEXPERIMENT=ms_nocgo_opensslcrypto` in builder stages |
Microsoft Go's default `systemcrypto` requires `CGO_ENABLED=1`, but Go
auto-disables CGO when cross-compiling. This experiment (documented in
Microsoft Go MigrationGuide) enables the cgo-less OpenSSL backend so
systemcrypto works with `CGO_ENABLED=0`. Auto-selected in Go 1.27+; env
var can be removed then. |
| `operator/Dockerfile`: `RUN make manifests` wrapped in `env -u GOOS -u
GOARCH` | `make manifests` shells out to `controller-gen` — a build-time
host tool, not the target binary. Without unset, it cross-compiled to
arm64 and failed to `exec` on the amd64 builder. |
| `controller/Dockerfile`: split intermediate into two stages — new
`bpf-gen` at `$TARGETPLATFORM`, existing intermediate stays at
`$BUILDPLATFORM` | Retina's `//go:generate bpf2go -target ${GOARCH}`
directives require target-arch execution (bpf2go invokes clang; can't
cleanly cross-compile bpf2go itself). Split isolates that one
requirement to a small emulated stage while keeping the Go build fast at
$BUILDPLATFORM. Restores fast arm64 build without breaking bpf .o file
generation. |

> Impact on amd64 builds: zero — all these changes are inert when host
== target (CGO stays enabled, no cross-compile, bpf-gen runs natively).

### Tag consistency fix
The migration surfaced a pre-existing latent bug: `make retina-image` /
`make manifest` didn't have `TAG` set by CI, so the Makefile fell back
to `git describe --tags --always` (produces `v1.2.2-76-geade8685`),
while all consumers (`e2e`, `perf-*`, `scale`) used make version
(produces the 7-char SHA). Images were pushed under one tag, consumers
looked for another. GH masked it because actions/checkout defaults to
shallow (no tags, so git describe also falls back to short SHA); ADO
defaults to full-clone.

ADO pipelines now pass `TAG=$(make version)` explicitly to every `make`
invocation, ensuring builder and consumer speak the same tag string on
both platforms.

### Temporary items in this PR (will be reverted after cutover)

- `forceMergeGroup` pipeline parameter in `ci.yml`. Boolean checkbox in
the "Run pipeline" UI that treats a manual queue as if it were a
merge-queue event (push + telemetry + full test suite). Default `false`,
safe. Used during dark-launch to validate the full flow without needing
an actual merge-queue event. Revert once we've moved past validation.

### Verification
- manual run of pipelines completed successfully

## Checklist

- [ ] I have read the [contributing
documentation](https://retina.sh/docs/Contributing/overview).
- [ ] I signed and signed-off the commits (`git commit -S -s ...`). See
[this
documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification)
on signing commits.
- [ ] I have correctly attributed the author(s) of the code.
- [ ] I have tested the changes locally.
- [ ] I have followed the project's style guidelines.
- [ ] I have updated the documentation, if necessary.
- [ ] I have added tests, if applicable.

## Screenshots (if applicable) or Testing Completed

Please add any relevant screenshots or GIFs to showcase the changes
made.

## Additional Notes

Add any additional notes or context about the pull request here.

---

Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more
information on how to contribute to this project.

---------

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants