Skip to content

fix(desktop-backend): fix prod crash and add missing env vars#6042

Merged
kodjima33 merged 2 commits into
mainfrom
worktree-fix-desktop-backend-prod
Mar 25, 2026
Merged

fix(desktop-backend): fix prod crash and add missing env vars#6042
kodjima33 merged 2 commits into
mainfrom
worktree-fix-desktop-backend-prod

Conversation

@kodjima33
Copy link
Copy Markdown
Collaborator

Summary

  • Removed unused RESEND_API_KEY from Helm values.yaml — it was never used in the Rust backend code but referenced a nonexistent K8s secret key, causing CreateContainerConfigError on pod restart
  • Added prod_values.yaml with all required env vars (GEMINI_API_KEY, OPENAI_API_KEY, DEEPGRAM_API_KEY, ENCRYPTION_SECRET, REDIS, PINECONE) — previously only dev had a proper values file

Context

desktop-api.omi.me went down today after a GKE node pool upgrade by thainguyensunya@gmail.com evicted the desktop-backend pod. The replacement pod couldn't start due to the missing RESEND_API_KEY in the secret. This was a latent bug since the initial Helm chart import (Feb 12).

Also updated the expired Gemini API key in GCP Secret Manager and restarted affected deployments (desktop-backend + backend-listen).

Already deployed to prod — this PR syncs the repo with what's live.

Test plan

  • desktop-api.omi.me/health returns healthy
  • Gemini proxy returns 401 (auth required), not 502
  • Verified new Gemini key works via direct API call
  • Verified backend-listen pods picked up new key

🤖 Generated with Claude Code

kodjima33 and others added 2 commits March 25, 2026 17:29
RESEND_API_KEY is not used anywhere in the Rust backend code but was
referenced in the default values.yaml pointing to prod-omi-backend-secrets.
This key never existed in the K8s secret, causing CreateContainerConfigError
whenever the pod restarts (triggered today by GKE node pool upgrade).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The default values.yaml was missing critical env vars (GEMINI_API_KEY,
OPENAI_API_KEY, DEEPGRAM_API_KEY, ENCRYPTION_SECRET, REDIS, PINECONE)
that the dev_values.yaml had. This adds a proper prod values file
matching the dev setup but pointing to prod-omi-backend-secrets.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@kodjima33 kodjima33 merged commit 711b121 into main Mar 25, 2026
3 checks passed
@kodjima33 kodjima33 deleted the worktree-fix-desktop-backend-prod branch March 25, 2026 21:47
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Mar 25, 2026

Greptile Summary

This PR fixes a production outage on desktop-api.omi.me caused by a CreateContainerConfigError: the default values.yaml referenced RESEND_API_KEY from a Kubernetes secret, but that key never existed (and the Rust backend never used it). After a GKE node pool upgrade evicted the pod, it could not restart. The fix removes the bad reference from values.yaml and adds a proper prod_values.yaml with all env vars the backend actually needs.

Key changes:

  • values.yaml: Removes the stale RESEND_API_KEY secret reference that caused CreateContainerConfigError.
  • prod_values.yaml (new): Provides a complete production Helm override with GEMINI_API_KEY, OPENAI_API_KEY, DEEPGRAM_API_KEY, ENCRYPTION_SECRET, REDIS_DB_HOST, REDIS_DB_PASSWORD, and PINECONE_API_KEY wired to prod-omi-backend-secrets, plus production-appropriate node affinity.
  • PINECONE_HOST is present in dev_values.yaml but absent from the new prod_values.yaml — since pinecone_host defaults to None with no startup warning, Pinecone-backed features will silently not work in prod.
  • The identical RESEND_API_KEY reference still exists in dev_values.yaml and will cause the same crash in dev if that secret key is absent.

Confidence Score: 4/5

  • Safe to merge — the production crash is fixed and already deployed; minor follow-up items remain.
  • The core fix (removing the nonexistent RESEND_API_KEY reference) is correct and proven in production. The new prod_values.yaml covers all keys the Rust backend actively validates. The two remaining issues — missing PINECONE_HOST in prod and the stale RESEND_API_KEY in dev_values.yaml — are non-blocking P2/P1 follow-ups that don't affect pod startup, but should be addressed in a follow-up to avoid silent feature failures and a potential dev environment crash.
  • dev_values.yaml still contains the RESEND_API_KEY reference that was the root cause of this incident; prod_values.yaml is missing PINECONE_HOST.

Important Files Changed

Filename Overview
desktop/Backend-Rust/charts/desktop-backend/values.yaml Removes the unused RESEND_API_KEY secret reference that was causing CreateContainerConfigError on pod restart. The key was never referenced in the Rust backend code.
desktop/Backend-Rust/charts/desktop-backend/prod_values.yaml New production Helm values file with all required secret env vars. Missing PINECONE_HOST (present in dev) which will silently disable Pinecone-backed features; also includes OPENAI_API_KEY which is not consumed by the Rust backend.

Sequence Diagram

sequenceDiagram
    participant GKE as GKE Node Pool Upgrade
    participant Pod as desktop-backend Pod
    participant K8s as Kubernetes Secret<br/>(prod-omi-backend-secrets)
    participant App as Rust App (desktop-backend)

    GKE->>Pod: Evict pod (node pool upgrade)
    Pod->>K8s: Fetch env vars from secret
    Note over K8s: Before fix: RESEND_API_KEY key<br/>did not exist in secret
    K8s-->>Pod: CreateContainerConfigError ❌

    Note over Pod,K8s: After this PR

    Pod->>K8s: Fetch env vars (no RESEND_API_KEY)
    K8s-->>Pod: GEMINI_API_KEY, OPENAI_API_KEY,<br/>DEEPGRAM_API_KEY, ENCRYPTION_SECRET,<br/>REDIS_DB_HOST, REDIS_DB_PASSWORD,<br/>PINECONE_API_KEY ✅
    Pod->>App: Start with all required env vars
    App-->>Pod: Healthy ✅
Loading

Comments Outside Diff (1)

  1. desktop/Backend-Rust/charts/desktop-backend/dev_values.yaml, line 35-39 (link)

    P1 Same latent RESEND_API_KEY bug remains in dev

    This PR correctly removes RESEND_API_KEY from values.yaml because the key does not exist in the K8s secret, causing CreateContainerConfigError. The same reference still exists here in dev_values.yaml pointing to dev-omi-backend-secrets. If RESEND_API_KEY is equally absent from the dev secret, the dev desktop-backend pod will fail to start for exactly the same reason after the next node eviction. Since RESEND_API_KEY is never read in config.rs or anywhere in the Rust backend, this entry can safely be removed.

Reviews (1): Last reviewed commit: "feat(desktop-backend): add prod_values.y..." | Re-trigger Greptile

@@ -0,0 +1,106 @@
# Production environment values for desktop-backend
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 OPENAI_API_KEY injected but not consumed by the Rust backend

OPENAI_API_KEY is included in the secret env vars (line 22–26) but has no corresponding field in config.rs — the Rust binary never reads it. The proxy routes only cover Gemini and Deepgram; no OpenAI client is wired up. This is harmless today but could create confusion (e.g., someone assumes OpenAI calls are forwarded when they are not). If it's intentionally reserved for a future feature, a comment to that effect would help. Otherwise it can be removed to keep secrets minimal.

Comment on lines +52 to +56
- name: PINECONE_API_KEY
valueFrom:
secretKeyRef:
name: prod-omi-backend-secrets
key: PINECONE_API_KEY
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 PINECONE_HOST missing alongside PINECONE_API_KEY

dev_values.yaml sets both PINECONE_API_KEY and PINECONE_HOST as secrets. prod_values.yaml includes the API key but omits the host. In config.rs, pinecone_host is loaded from PINECONE_HOST; without it the field stays None and any Pinecone-backed feature (knowledge graph / vector search) will silently fail in production without any error at startup (the config validation does not warn on a missing Pinecone host).

Consider adding a PINECONE_HOST entry pointing to prod-omi-backend-secrets to match the dev configuration.

Glucksberg pushed a commit to Glucksberg/omi-local that referenced this pull request Apr 28, 2026
…ardware#6042)

## Summary
- Removed unused `RESEND_API_KEY` from Helm `values.yaml` — it was never
used in the Rust backend code but referenced a nonexistent K8s secret
key, causing `CreateContainerConfigError` on pod restart
- Added `prod_values.yaml` with all required env vars (GEMINI_API_KEY,
OPENAI_API_KEY, DEEPGRAM_API_KEY, ENCRYPTION_SECRET, REDIS, PINECONE) —
previously only dev had a proper values file

## Context
`desktop-api.omi.me` went down today after a GKE node pool upgrade by
thainguyensunya@gmail.com evicted the desktop-backend pod. The
replacement pod couldn't start due to the missing `RESEND_API_KEY` in
the secret. This was a latent bug since the initial Helm chart import
(Feb 12).

Also updated the expired Gemini API key in GCP Secret Manager and
restarted affected deployments (desktop-backend + backend-listen).

**Already deployed to prod** — this PR syncs the repo with what's live.

## Test plan
- [x] `desktop-api.omi.me/health` returns healthy
- [x] Gemini proxy returns 401 (auth required), not 502
- [x] Verified new Gemini key works via direct API call
- [x] Verified backend-listen pods picked up new key

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant