deploy-firebase.yml: CI apphosting rollout collides with FAH native auto-rollout → red jobs on successful deploys

[MyAlterLego]

Summary

For every consumer repo whose App Hosting backend has FAH native GitHub auto-rollout enabled (confirmed: tmi), every dev/main push produces a red deploy-* job even though the deploy succeeds. The CI apphosting:rollouts:create step races FAH's native auto-rollout, both build the same commit, and the CI one fails with a build-ID collision 409. The retry logic only catches "Failed to authenticate", so the 409 fails the job hard.

This has masked all TMI deploys as "failed" since ~2026-06-25 14:00 (last green run #406), and earlier sent us chasing the auth-flake red herring (#42/#44) when auth was actually succeeding.

Root cause

deploy-firebase.yml (both the main branch and branch preview steps) runs:

firebase apphosting:rollouts:create "$BACKEND" --git-branch "$DEPLOY_BRANCH" --non-interactive

rollouts:create --git-branch asks FAH to create a new build from the branch HEAD and roll it out, and FAH auto-names that build build-YYYY-MM-DD-NNN.

But the backend also has a codebase.repository GitHub connection with native auto-rollout on push. So a single push to dev/main triggers two builds for the same commit:

1. FAH native integration → creates build-YYYY-MM-DD-NNN, rolls out, succeeds (this is what actually serves).
2. CI rollouts:create → tries to create another build for the same commit → 409:
   • Resource '...builds/build-2026-06-26-006' already exists, or
   • 409, unable to queue the operation (native rollout still in-flight).

The retry block only continues on grep -qi "Failed to authenticate"; every other error (incl. these 409s) hits exit "$fb_status" → red job. Reruns just collide again (the counter re-picks an already-used ID), so "rerun until it catches a window" never converges here.

The existing comment at deploy-firebase.yml:41 ("some backends … static/FAH-native, e.g. www … never deploy through it") shows the FAH-native-vs-CI-deploy split is already known — but there's no input to express it, so gated backends end up running both paths.

Evidence (TMI, 2026-06-26)

PR #416 → dev, PR #417 (dev→main) → prod. Both CI runs show deploy-*/Firebase Deploy = failure, yet both are live:

Backend	Live build (FAH traffic API)	Commit	State	Traffic
tmi-dev	build-2026-06-26-005	#416 merge (v5.2.1)	READY	100%
tmi (prod)	build-2026-06-26-001	#417 merge (v5.2.1)	READY	100%

Corroborating: the App Hosting - Rollout (triarch-dev-tmi/.../tmi-dev) commit-status check PASSES (FAH native), while ci-cd / deploy-dev / Firebase Deploy FAILS (CI) on the same run. Auth log shows Successfully authenticated immediately before the 409 — not the #42/#44 auth flake.

Verify any time:

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  https://firebaseapphosting.googleapis.com/v1beta/projects/triarch-dev-tmi/locations/us-central1/backends/tmi/traffic

Why it matters beyond red runs

It also breaks the CL-4 gating contract: FAH native auto-rollout deploys main on push independently of CI, so gate-prod-version / quality-gate never actually gate the prod rollout — the native integration ships whatever lands on main. The double-deploy isn't just noisy; it means the gate is advisory on these backends.

Recommended fix (decision needed)

Pick one deploy authority per backend:

• Option A — CI is the sole deployer (preserves CL-4 gating). Recommended. Disable FAH native auto-rollout on gated backends (tmi, tmi-dev, and any other repo running the CI apphosting deploy). Backend → Settings → rollout/automatic-deployment off, or detach the branch trigger on the connection. No workflow change; the CI rollouts:create becomes the only build → no collision, and the gate is real again.
• Option B — FAH native is the deployer. Remove the CI apphosting:rollouts:create for these backends and have CI only verify the native rollout (via the traffic API / commit-status). Loses CL-4 gating unless the gate moves to a pre-merge required check.

Code-level stopgap (so red runs stop before the config decision)

Add an opt-in input app_hosting_native_rollout (default false, behavior unchanged). When true, the apphosting deploy steps treat the collision 409s as "native integration already rolled out this commit": re-query the backend traffic/latest build for the deploy-branch HEAD, and exit 0 iff a READY build for that commit is serving, else fail. Gating it behind an explicit input avoids masking genuine failures on repos where native rollout is not enabled. Happy to open the PR (@release/v9 + forward-port to main) once we agree A vs B.

cc: relates to the auth-flake thread (#42, #44) — this is the non-auth failure those reruns were actually hitting.

[MyAlterLego]

Resolved for TMI via Option A (CI = sole deployer)

Confirmed the root cause from the traffic resource — both TMI backends had active native auto-rollout:

• tmi-dev traffic.rolloutPolicy = {"codebaseBranch": "dev"}
• tmi traffic.rolloutPolicy = {"codebaseBranch": "main"}

(No disabled flag = enabled. This is why a build appeared ~seconds after each push — before the CI deploy step ran — and the CI rollouts:create then 409'd on the auto-named build ID.)

Applied Option A — disabled native auto-rollout on both backends (reversible), keeping the branch association so CI rollouts:create --git-branch remains the single, gated deploy path:

PATCH .../backends/{tmi-dev,tmi}/traffic?updateMask=rolloutPolicy.disabled
{"rolloutPolicy":{"disabled":true}}

Now:

• tmi-dev → {"codebaseBranch": "dev", "disabled": true}
• tmi → {"codebaseBranch": "main", "disabled": true}

The next dev/main push should produce a green deploy-* job (CI is the only build creator → no collision). Reverting is disabled:false.

Still open for the other 6 repos

The fix was applied only to TMI's backends. The other consumer repos (security-admin, security-portal, platform, dev-portal, truthtreason, darksouls) likely have the same native-auto-rollout + CI-deploy double path and would benefit from the same one-line traffic PATCH. Leaving this issue open to track a portfolio sweep + a note in the deploy-firebase.yml docs that gated backends must have native auto-rollout disabled. To find them: GET .../backends/<b>/traffic → rolloutPolicy missing disabled:true.