Harden prod backend Cloud Run deploy workflow (#9164) by Git-on-my-level · Pull Request #9169 · BasedHardware/omi

Git-on-my-level · 2026-07-06T21:47:05Z

Summary

Hardens gcp_backend.yml against wedged Cloud Run traffic (spec.traffic on failed revisions) with preflight secret/traffic checks, revision readiness gating, repair-traffic-only mode, and cloud-run-only deploy target.
Removes stale prod secret bindings (SERVICE_ACCOUNT_JSON, POSTHOG_PROJECT_API_KEY) that caused SecretsAccessCheckFailed on revision create.
Adds operator tooling: pre-deploy-check.sh (hermetic + optional live), repair_cloud_run_traffic.py, preflight-cloud-run-deploy.py, offline --check-rendered-cloud-run validation, and prod hotfix runbook.

Pre-deploy validation (no deploy required)

cd backend
CLOUD_RUN_VPC_NETWORK=offline-check-network \
CLOUD_RUN_VPC_SUBNET=offline-check-subnet \
  ./scripts/pre-deploy-check.sh

With gcloud auth for live read-only checks:

./scripts/pre-deploy-check.sh --live prod based-hardware

CI runs the hermetic path via lint.yml when deploy-safety files change.

Test plan

backend/scripts/pre-deploy-check.sh (hermetic)
python3 backend/scripts/check_workflow_contracts.py
25 unit tests: test_backend_runtime_env_validator, test_repair_cloud_run_traffic, test_preflight_cloud_run_deploy, test_deploy_status_report
validate-backend-runtime-env.py --env prod --check-workflows --check-rendered-cloud-run
Manual: dispatch gcp_backend.yml with mode=repair-traffic-only against dev (optional)
Manual: dispatch gcp_backend.yml with deploy_targets=cloud-run-only against dev after merge (optional)

Closes #9164

Made with Cursor

Add preflight/repair scripts, offline rendered-env validation, pre-deploy-check.sh CI gate, and workflow modes for traffic repair and Cloud Run-only deploys so failed revisions cannot wedge spec.traffic. Verification: backend/scripts/pre-deploy-check.sh; python3 backend/scripts/check_workflow_contracts.py; 25 unit tests in deploy safety suite. Co-authored-by: Cursor <cursoragent@cursor.com>

mintlify · 2026-07-06T21:48:16Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
omi	🟢 Ready	View Preview	Jul 6, 2026, 9:49 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c386fe0f42

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-06T21:52:42Z

            --expect-cloud-run-traffic backend-sync=${{ steps.capture-backend-sync-revision.outputs.revision }} \
-            --expect-cloud-run-traffic backend-integration=${{ steps.capture-backend-integration-revision.outputs.revision }}
+            --expect-cloud-run-traffic backend-integration=${{ steps.capture-backend-integration-revision.outputs.revision }} \
+            | tee /tmp/cloud-run-deploy-status.txt || true


Do not mask failed prod traffic verification

In the prod deploy path I inspected, this command is the only post-shift readback that verifies status.traffic is 100% on the captured revisions; backend/scripts/deploy_status_report.py returns non-zero on describe errors, spec/status mismatches, or stale traffic. Because its output is piped to tee and followed by || true (and the workflow does not enable pipefail), any of those failures is swallowed after the update-traffic commands, so the workflow can report a successful prod deploy even when Cloud Run is still serving the old revision. Keep the summary generation but preserve the Python exit status on the success path.

Useful? React with 👍 / 👎.

cubic-dev-ai

10 issues found across 19 files

Confidence score: 2/5

In .github/workflows/gcp_backend.yml and .github/workflows/gcp_backend_auto_dev.yml, deploy_status_report.py is force-succeeded with || true, so traffic/revision mismatches can still produce a green deploy result and let a bad rollout pass unnoticed—remove || true (or only tolerate failures in explicitly non-blocking paths) before merging.
backend/scripts/preflight-cloud-run-deploy.py currently accepts blank SERVICE/REVISION and throws a traceback for bad --wait-revision-ready input, which can turn fast validation errors into long noisy waits and harder CI diagnosis—fail fast on empty values and route parse failures through parser.error(...).
Offline/state-driven repair flow is inconsistent across backend/scripts/deploy_status_report.py and backend/scripts/repair_cloud_run_traffic.py: saved project/region are dropped, so generated repair commands can target the wrong Cloud Run service—propagate state-derived project/region through report and repair command generation before merging.
Guardrails are a bit late or brittle in .github/workflows/gcp_backend.yml and backend/scripts/pre-deploy-check.sh (environment validation timing, Python runtime drift), and backend/tests/unit/test_preflight_cloud_run_deploy.py leaks module state; these can cause avoidable CI surprises—add early dispatch/runtime validation and clean sys.modules via the existing snapshot fixture to de-risk follow-up runs.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/gcp_backend.yml">

<violation number="1" location=".github/workflows/gcp_backend.yml:46">
P2: The environment name is only checked inside the job, so an invalid dispatch value can still reach the job-level `environment:` binding before the validation step runs. Moving the validation into a workflow-level dispatch filter or encoding the allowed values in the input itself would avoid binding an invalid environment.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai · 2026-07-06T22:02:17Z

 jobs:
+  repair-traffic:
+    if: ${{ github.event.inputs.mode == 'repair-traffic-only' }}
+    environment: ${{ github.event.inputs.environment }}


P2: The environment name is only checked inside the job, so an invalid dispatch value can still reach the job-level environment: binding before the validation step runs. Moving the validation into a workflow-level dispatch filter or encoding the allowed values in the input itself would avoid binding an invalid environment.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/gcp_backend.yml, line 46: <comment>The environment name is only checked inside the job, so an invalid dispatch value can still reach the job-level `environment:` binding before the validation step runs. Moving the validation into a workflow-level dispatch filter or encoding the allowed values in the input itself would avoid binding an invalid environment.</comment> <file context> @@ -19,13 +19,78 @@ on: jobs: + repair-traffic: + if: ${{ github.event.inputs.mode == 'repair-traffic-only' }} + environment: ${{ github.event.inputs.environment }} + permissions: + contents: 'read' </file context>

…-9164

Remove || true from post-deploy traffic verification, harden preflight/repair scripts, propagate Cloud Run project/region through offline state, rename render_backend_runtime_env for import consistency, and fix desktop import guard paths after Onboarding/ move. Merged main for backend unit test harness updates. Co-authored-by: Cursor <cursoragent@cursor.com>

GitHub Actions lint runners use system python3 (3.12.x) while .python-version pins 3.11.15; keep the exact-version guard for local dev only. Co-authored-by: Cursor <cursoragent@cursor.com>

cubic-dev-ai

1 issue found across 17 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/gcp_backend.yml">

<violation number="1" location=".github/workflows/gcp_backend.yml:46">
P2: The environment name is only checked inside the job, so an invalid dispatch value can still reach the job-level `environment:` binding before the validation step runs. Moving the validation into a workflow-level dispatch filter or encoding the allowed values in the input itself would avoid binding an invalid environment.</violation>
</file>

<file name="backend/tests/unit/test_preflight_cloud_run_deploy.py">

<violation number="1" location="backend/tests/unit/test_preflight_cloud_run_deploy.py:9">
P2: The new test code imports from `tests.unit.memory_import_isolation`, which is a deprecated module the project is actively migrating away from. The project's test isolation guidelines (AGENTS.md, test_isolation.md) explicitly state not to extend this module and instead use `backend/testing/import_isolation.py` (`stub_modules` context manager) or `monkeypatch.setattr` on lazy-held singletons. Since this is adding *new* usage of the deprecated module, consider using `backend/testing/import_isolation.stub_modules` with `load_module_fresh` instead — or, since the production scripts already have clean top-level imports (stdlib only), the snapshot/restore may not be needed at all.</violation>
</file>

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

Git-on-my-level · 2026-07-07T00:59:03Z

@cursor address CI failure and review comment

cursor · 2026-07-07T00:59:09Z

Addressed both the CI failure and the review comment.

CI failure

The Backend unit suite failed on the fast-unit duration guard, not on a test assertion:

0.19s > 0.12s  tests/unit/test_backend_runtime_env_validator.py::test_repo_rendered_cloud_run_matches_manifest

That test does heavier work (check_rendered_cloud_run=True for dev + prod), so it exceeded the 100ms CPU target (+ 2ms grace) on CI. I added it to backend/tests/fast_unit_duration_allowlist.txt alongside the other runtime-env validator tests.

Review comment

Removed the deprecated tests.unit.memory_import_isolation import from test_preflight_cloud_run_deploy.py. The preflight script only uses stdlib imports, so snapshot/restore isn’t needed — the test now matches the simpler importlib pattern used in test_backend_runtime_env_validator.py.

Verification

test_preflight_cloud_run_deploy.py (4 tests) — passed
test_repo_rendered_cloud_run_matches_manifest with fast-unit guard enabled — passed

Pushed as 2e233f79d on harden/backend-deploy-9164. CI should pass on the next run.

- Allowlist test_repo_rendered_cloud_run_matches_manifest (0.19s CPU on CI). - Simplify preflight deploy tests to match validator importlib pattern without memory_import_isolation snapshot/restore. Co-authored-by: David Zhang <Git-on-my-level@users.noreply.github.com>

mintlify Bot deployed to staging - docs July 6, 2026 21:49 View deployment

chatgpt-codex-connector Bot reviewed Jul 6, 2026

View reviewed changes

cubic-dev-ai Bot reviewed Jul 6, 2026

View reviewed changes

Git-on-my-level and others added 2 commits July 6, 2026 19:48

Merge remote-tracking branch 'origin/main' into harden/backend-deploy…

d411740

…-9164

mintlify Bot deployed to staging - docs July 7, 2026 00:23 View deployment

Skip Python version pin in CI pre-deploy checks.

62469d1

GitHub Actions lint runners use system python3 (3.12.x) while .python-version pins 3.11.15; keep the exact-version guard for local dev only. Co-authored-by: Cursor <cursoragent@cursor.com>

cubic-dev-ai Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread backend/tests/unit/test_preflight_cloud_run_deploy.py Outdated

Git-on-my-level merged commit f5be39d into main Jul 7, 2026
7 checks passed

Git-on-my-level deleted the harden/backend-deploy-9164 branch July 7, 2026 01:27

Git-on-my-level mentioned this pull request Jul 7, 2026

Prod GKE backend ExternalSecret still references missing SERVICE_ACCOUNT_JSON #9187

Open

github-actions Bot mentioned this pull request Jul 7, 2026

Upstream sync 51db88333 — 378 behind cybervoid/omi#6

Closed

Uh oh!

Conversation

Git-on-my-level commented Jul 6, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Pre-deploy validation (no deploy required)

Test plan

Uh oh!

mintlify Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Git-on-my-level commented Jul 7, 2026

Uh oh!

cursor Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI failure

Review comment

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Git-on-my-level commented Jul 6, 2026 •

edited by cubic-dev-ai Bot

Loading

mintlify Bot commented Jul 6, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cursor Bot commented Jul 7, 2026 •

edited

Loading