Stop idle App Engine versions after deploy

[anth-volk]

What & why

The v1 API runs on App Engine Flexible, which keeps at least one always-on VM for every SERVING version regardless of traffic (current app.yaml: 4 vCPU / 24 GB ≈ ~$212/mo per version). The deploy pipeline promotes traffic to the new version but never deactivates the ones it supersedes, so both staging and old prod versions accumulate as idle-but-billing VMs. At the time of writing the default service had grown to 97 versions (16 SERVING / 81 STOPPED) with only one serving traffic — the idle SERVING ones are the bulk of a ~$6,050/mo App Engine bill.

This adds post-deploy cleanup so idle versions stop costing money, while preserving rollback.

Changes

• stop-staging-app-engine-version (after integration-tests-staging): stops the staging version once its tests pass. Staging holds 0 VMs between deploys; each deploy makes a fresh one.
• cleanup-prod-app-engine-versions (after promote-production): keeps the live version plus the newest 2 prod versions warm for instant rollback, and stops every other SERVING version (older prod + leftover staging). It then prunes stopped versions to the newest 10 prod and newest 3 staging, deleting everything else stopped (older prod, older staging, and legacy timestamp-named versions) to stay under App Engine's 210-versions-per-service limit. Never touches the version receiving traffic.
• New scripts stop_app_engine_version.sh and cleanup_app_engine_versions.sh — idempotent, DRY_RUN=1 supported, written for bash 3.2 so they dry-run locally too.
• A fail-safe guard: the cleanup aborts before stopping anything if versions are SERVING but none report traffic (so an empty/failed traffic query can't deactivate production), plus unit tests in tests/unit/test_app_engine_cleanup_scripts.py.

Why stop, not delete (for the warm/rollback set)

Stopping deactivates the VM (→ ~$0) but keeps the version deployed and restartable, so rollback still works: gcloud app versions start <v> then gcloud app services set-traffic default --splits=<v>=1. Keeping the 2 newest prod versions warm gives instant rollback, and the newest 10 stopped prod versions remain as cold rollback targets — the original reason old versions were kept SERVING (a new deploy's simulation-API dependency could fail). Only prod versions are meaningful rollback targets, so staging retention is kept small.

Verification

• gcloud app versions stop/start confirmed working live on this project's automatic-scaling Flex versions (stopped prod-2393 → STOPPED → started → SERVING); it's the same Versions.UpdateVersion call already used to trim the fleet on 2026-07-02. No app.yaml scaling-mode change needed.
• Against the current live state (97 versions on default: 16 SERVING / 81 STOPPED), the first real run keeps the live version + 2 warm prod, stops the other ~14 idle SERVING versions, then prunes the stopped set (now ~95) to the newest 10 prod + 3 staging — deleting ~82 and leaving the service at ~15 versions.
• Note on DRY_RUN: the preview reports fewer deletions (currently 68) than the real run (~82) because it doesn't perform the stops first, so the STOPPED set it prunes is smaller. The larger real-run count is expected and stays hard-bounded by the retention policy (newest 10 prod + 3 staging kept; everything else stopped is deleted — it can't exceed that).
• Live version is protected: the fail-safe guard aborts (non-zero) before stopping anything if versions are SERVING but none report traffic; and delete only ever targets STOPPED, so the live version can never be deleted.
• New unit tests (tests/unit/test_app_engine_cleanup_scripts.py, 10 cases with a stubbed gcloud) cover live/warm retention, protection of an older traffic-serving version, delete-only-stopped, the guard, missing-staging robustness, and the stop script — all passing, ruff clean. push.yml validated as YAML; changelog fragment added.

Scope

Deploy-pipeline / cost only — no API contract or route changes; everything stays on the App Engine v1 path. This is a bridge until v1 finishes migrating to Cloud Run (min=0 = true scale-to-zero, at which point this machinery retires — see #3737 for the Cloud Run revision-cleanup follow-up).

Expected impact

Steady-state running VMs drop from ~14–16 to 2 (live + previous prod); staging ~$0 except during a deploy's test window → roughly $6,050/mo → ~$450–500/mo.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.82%. Comparing base (74c146f) to head (4407075).
⚠️ Report is 17 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3733   +/-   ##
=======================================
  Coverage   79.82%   79.82%           
=======================================
  Files          70       70           
  Lines        4336     4336           
  Branches      808      808           
=======================================
  Hits         3461     3461           
  Misses        655      655           
  Partials      220      220           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.