Fix HOSTED_PARAKEET_API_URL: https → http for Cloud Run services

[beastoin]

Issue

PR #9437 (commit c70d19dc26, "Harden macOS and backend against recurring bug classes") added HOSTED_PARAKEET_API_URL entries to backend/deploy/runtime_env.yaml for Cloud Run services with https://parakeet.omi.me.

The parakeet ILB is HTTP-only (port 80, targetHttpProxy, no TLS termination). HTTPS requests to it timeout.

Incident Impact

• Downtime: ~14 hours (Jul 12 01:13 UTC – ongoing, not yet resolved)
• Affected feature: Voice message transcription via Parakeet (/v2/transcribe pre-recorded STT) — 100% failure rate on all Cloud Run services (backend, backend-sync, backend-integration)
• User impact: All voice messages sent by users fail to transcribe. Backend returns 200 (graceful fallback) but transcription content is empty — users see no text for their voice messages.
• Scope: Every user who sent a voice message during the outage window is affected. Parakeet request rate dropped to 0 rps at 01:37 UTC and has remained at 0 since (verified via Grafana parakeet-asr-monitoring dashboard).
• Not affected: Real-time streaming STT via WebSocket (/v3/stream) from GKE backend-listen — this uses ClusterIP (bypasses ILB), so live conversation transcription continued working.

Root Cause

PR #9437 added HOSTED_PARAKEET_API_URL to runtime_env.yaml with https:// scheme for Cloud Run services. The gcp_backend.yml workflow renders this file via render_backend_runtime_env.py and sets the env var on deploy. The main branch deploy at 01:11 UTC overwrote the correct http:// value with https://.

Timeline:

• Jul 12 01:11 UTC — gcp_backend.yml main branch workflow starts
• Jul 12 01:13:55 UTC — revision backend-f755063-1 goes live with https://parakeet.omi.me
• Jul 12 01:19:14 UTC — last successful parakeet transcription (old instance draining)
• Jul 12 01:19:35 UTC — first timeout, 100% failure from this point
• Jul 12 01:37 UTC — parakeet request rate drops to 0 rps (confirmed via Prometheus parakeet_requests_total)
• Ongoing — 0 rps sustained for 14 hours, all voice message transcription failing

Evidence

Previous revision (backend-f92baff, deployed 00:06 UTC):

HOSTED_PARAKEET_API_URL=http://parakeet.omi.me   ← correct

Current revision (backend-f755063, deployed 01:13 UTC):

HOSTED_PARAKEET_API_URL=https://parakeet.omi.me  ← wrong

ILB config:

• Forwarding rule: port 80 only
• Target proxy: targetHttpProxy (not HTTPS)
• No TLS certificate configured

Fix Required

In backend/deploy/runtime_env.yaml, change all Cloud Run service entries from:

HOSTED_PARAKEET_API_URL:
  value: https://parakeet.omi.me

to:

HOSTED_PARAKEET_API_URL:
  value: http://parakeet.omi.me

Audit: nllb.omi.me already uses http://. Other ILB services (diarizer, vad, pusher) use ClusterIP in runtime_env — no other instances of this bug.

@Git-on-my-level — this came from your PR #9437.

[beastoin]

Resolved — PR #9549 merged and deployed via gcp_backend.yml run 29186591303.

• New revision backend-sync-52d74be live with HOSTED_PARAKEET_API_URL=http://parakeet.omi.me (correct)
• Parakeet request rate recovering: 0.13 rps at 09:06 UTC (was 0 rps since 01:37 UTC)
• Total outage window: ~8 hours (Jul 12 01:13 – 09:06 UTC)