v2 async sync-local-files: fix 504 timeouts on large audio uploads

[beastoin]

Summary

Fixes #5941 — sync_local_files 504 Gateway Timeout on large payloads.

Root cause: v1 processes all segments synchronously (80-180s for large payloads), exceeding Cloud Run's 60s timeout.

Fix: v2 async endpoint pair that returns fast, processes in background:

• POST /v2/sync-local-files → Does fast-path work inline (decode, VAD), then starts background processing and returns 202 with job_id for polling. Zero-segment payloads return 200 inline (no background job needed).
• GET /v2/sync-local-files/{job_id} → Poll endpoint returning job status, progress, and final result.
• v1 endpoint is completely unchanged.

Backend changes

• backend/routers/sync.py: v2 POST/GET endpoints, _process_segments_background worker with heartbeat
• backend/database/sync_jobs.py: Redis-backed ephemeral job storage (TTL 1h, stale detection 10min)
• backend/tests/unit/test_sync_v2.py: 63 tests — structure, Redis CRUD, boundary, background worker behavioral, FastAPI TestClient execution
• All imports at module top level per repo conventions

App changes

• conversations.dart: syncLocalFilesV2() with upload + poll loop, SyncJobPollCallback
• local_wal_sync.dart: Both call sites wired with onPollProgress for UI updates
• sync_state.dart: Added SyncPhase.processingOnServer
• sync_page.dart + auto_sync_page.dart: UI for processingOnServer phase with segment progress
• wal_interfaces.dart: Re-exports for v2 types
• l10n/*.arb: processingOnServer and processingOnServerProgress keys in all 34 locales with proper translations

Design decisions

• Redis (not Firestore) for job storage — ephemeral, no billing, auto-TTL cleanup
• Daemon thread for background processing — avoids HTTP timeout without adding celery/task queue complexity
• File ownership transfer pattern — segmented paths moved to owned_paths before returning 202
• Error propagation — mark_job_completed includes error details when all segments fail

Test plan

• 63 unit tests in test_sync_v2.py (structure, Redis CRUD, boundary, behavioral, TestClient)
• 40 unit tests in test_sync_silent_failure.py (fair-use stubs updated)
• L1: Backend standalone — full 202→poll→completed cycle, 404/403 rejection, v1 unchanged
• L2: App + backend integrated — 4-step sync flow (navigate → trigger → poll → verify conversations)
• CP7 reviewer approved (4 rounds: test stubs, error propagation, l10n, imports)
• CP8 tester approved (2 rounds: added execution + behavioral tests)

E2E evidence

• flow-walker L2 report: viKODiFi0m — full 4-step sync flow with video, screenshots, 1000-entry correlated log timeline (515 backend + 479 app entries)
• Video recording of complete sync cycle (WAL injection → upload → poll → conversation appears)
• Backend + app logs interleaved chronologically with ISO timestamps

Review cycle changes

• R1: Added is_dg_budget_exhausted stubs to test_sync_silent_failure.py
• R1: mark_job_completed now propagates error info when all segments fail
• R1: sync_page.dart uses poll-based progress during processingOnServer
• R2: Replaced hardcoded English strings with l10n keys across 34 locales
• R2: Moved all mid-file imports to module top level in sync.py
• R3: Added proper translations for all non-English locales
• T1: Added 7 background worker behavioral tests + 5 TestClient execution tests

Deployment steps

Pre-merge checklist

• Confirm Redis is available in prod Cloud Run (used by sync_jobs.py for job state)
• No new env vars required — uses existing REDIS_DB_HOST/REDIS_DB_PORT/REDIS_DB_PASSWORD

Deploy sequence (after merge)

1. Backend Cloud Run only — pusher does NOT import sync modules, no pusher deploy needed

   gh workflow run "Deploy Backend to Cloud RUN" --repo BasedHardware/omi --ref main -f environment=prod

2. Verify deployment

   # Check Cloud Run revision is updated
   gcloud run revisions list --service backend-listen --region us-central1 --project based-hardware --limit 3

3. Smoke test — hit the new v2 endpoint

   curl -s https://api.omi.me/v2/sync-local-files -X POST -H "Authorization: Bearer <token>" | jq .
   # Should return 401/403 (auth required), NOT 404 (endpoint missing)

4. App release — Flutter app update with v2 sync client (can follow separately or alongside)

Rollback

• v1 endpoint is unchanged — if v2 has issues, app falls back to v1 automatically
• No database migrations to revert

🤖 Generated with Claude Code

by AI for @beastoin

[greptile-apps]

Greptile Summary

This PR introduces a v2 async variant of POST /v1/sync-local-files to fix 504 gateway timeouts on large audio uploads. The fast-path work (decode + VAD) runs inline and returns 202 Accepted with a job_id; the slow STT+LLM pipeline runs in a daemon background thread. The Flutter app polls GET /v2/sync-local-files/{job_id} until a terminal status is reached, and WAL entries are only marked synced when completion is confirmed. v1 is untouched. The Redis-backed job store (sync_jobs.py) handles TTL cleanup and stale-job detection cleanly.

Key findings:

• P1 — WAL incorrectly marked synced on all-segment-failure: When all audio segments fail individually (not a worker crash), mark_job_completed sets status='failed' and stores a result field (empty memories). The GET endpoint returns both. syncLocalFilesV2 in Flutter checks result != null before checking status == 'failed', so it silently returns an empty SyncLocalFilesResponse instead of throwing — causing local_wal_sync.dart to mark the WAL as isSynced = true even though nothing was processed. This contradicts the PR's stated design goal.
• P2 — Import placement violations: from database.sync_jobs import (...) is placed at line 990 (mid-module) and import uuid as _uuid is inside the sync_local_files_v2 function body (line 1137), both violating the backend import rules.

Confidence Score: 4/5

Safe to merge after fixing the all-segment-failure WAL marking bug; the P2 import issues are style-only.

One P1 logic bug: the 'failed' job with a non-null result field causes the WAL to be silently marked as synced, which is a data-integrity issue (user loses audio with no retry). The overall architecture is sound and v1 is untouched.

app/lib/backend/http/api/conversations.dart (polling result/status check order) and backend/routers/sync.py (import placement).

Important Files Changed

Filename	Overview
backend/routers/sync.py	Adds v2 async endpoints; has two import-placement violations and the main all-segments-failure path is exposed to the client-side WAL bug.
app/lib/backend/http/api/conversations.dart	New syncLocalFilesV2 polling loop has a logic inversion: a 'failed' job with a result field returns silently instead of throwing, causing WALs to be incorrectly marked as synced.
backend/database/sync_jobs.py	New Redis-backed job storage; stale detection, TTL management, and status transitions are correct, though mark_job_completed sets result even for fully-failed jobs (enables the client-side bug).
app/lib/services/wals/local_wal_sync.dart	Correctly switches batch and single-WAL sync to v2; WAL is only marked synced when syncLocalFilesV2 returns, but the client-side bug in that function can still result in false synced marking.
backend/tests/unit/test_sync_v2.py	Comprehensive structural and Redis unit tests; covers stale detection, status transitions, v1 regression, and v2 contract — though lacks a test for the all-failed path returning a non-null result.
app/lib/backend/schema/conversation.dart	Adds SyncJobStartResponse and SyncJobStatusResponse models with correct fromJson parsing and isTerminal/isSuccess helpers.
app/lib/services/wals/wal_interfaces.dart	Minor formatting cleanup and re-exports syncLocalFilesV2 alongside syncLocalFiles; no logic changes.
backend/test.sh	Adds test_sync_v2.py to the unit test suite; straightforward addition.

Sequence Diagram

sequenceDiagram
    participant App as Flutter App
    participant API as FastAPI (v2 endpoint)
    participant Redis as Redis (sync_jobs)
    participant BG as Background Thread

    App->>API: POST /v2/sync-local-files (files)
    API->>API: Decode .bin → WAV (fast path)
    API->>API: VAD segmentation
    API->>API: Fair-use / DG budget checks
    API->>Redis: create_sync_job(queued)
    API->>BG: daemon thread.start()
    API-->>App: 202 { job_id, poll_after_ms }

    loop Poll every 3s (max 6 min)
        App->>API: GET /v2/sync-local-files/{job_id}
        API->>Redis: get_sync_job(job_id)
        Redis-->>API: job dict (stale detection applied)
        API-->>App: { status, segments... }
    end

    BG->>Redis: mark_job_processing
    BG->>BG: STT (Deepgram) + LLM per segment
    alt All OK
        BG->>Redis: mark_job_completed → status=completed
    else Some failed
        BG->>Redis: mark_job_completed → status=partial_failure
    else All failed
        BG->>Redis: mark_job_completed → status=failed + result (bug)
    else Worker crash
        BG->>Redis: mark_job_failed → status=failed, no result
    end
    BG->>BG: cleanup files

    App->>API: GET /v2/sync-local-files/{job_id}
    API-->>App: terminal status + result
    App->>App: mark WAL as synced (also on all-failed path)

Loading

Comments Outside Diff (1)

1. app/lib/backend/http/api/conversations.dart, line 98-111 (link)

   WAL marked synced on all-segment-failure

   When mark_job_completed is called in the background (all segments individually fail), it sets status='failed' and stores result (with empty new_memories/updated_memories). The GET endpoint then returns both status: 'failed' and result: {...}.

   Because the check for result != null comes before the status == 'failed' guard, the function returns an empty SyncLocalFilesResponse instead of throwing — so local_wal_sync.dart marks the WAL as isSynced = true even though nothing was processed. This directly contradicts the PR's stated goal: "WAL is only marked synced when processing is confirmed complete."

   Fix: check the terminal status before inspecting the result:

   Alternatively, the backend GET handler could omit result from the response when status == 'failed' (keeping it only for completed and partial_failure).

_{Reviews (1): Last reviewed commit: "Fix review round 1: heartbeat, failed-jo..." | Re-trigger Greptile}

[greptile-apps]

Imports must be at the top of the module

Both of these imports violate the backend import rule:

1. from database.sync_jobs import (...) at line 990 — placed mid-module after ~975 lines of existing code instead of at the top of the file.
2. import uuid as _uuid at line 1137 — an in-function import inside sync_local_files_v2.

Both should be moved to the top-level import block at the top of sync.py. uuid is already available in the standard library.

Suggested change

	from database.sync_jobs import (
	create_sync_job,
	get_sync_job,
	update_sync_job,
	mark_job_processing,
	mark_job_completed,
	mark_job_failed,
	from database.sync_jobs import (
	create_sync_job,
	get_sync_job,
	mark_job_processing,
	mark_job_completed,
	mark_job_failed,
	)

Move this block to the top of the file alongside the other from database ... imports (lines ~18–20), and add import uuid to the standard-library section. Remove the import uuid as _uuid at line 1137 and replace _uuid.uuid4() with uuid.uuid4().

Context Used: Backend Python import rules - no in-function impor... (source)

[beastoin]

Live Test Evidence (CP9A + CP9B)

L1 — Backend standalone (CP9A)

Built and ran backend locally from fix/sync-v2-background-5941 branch. All paths verified:

Path	Test	Expected	Actual	Status
P1: v2 POST, no auth	curl -X POST /v2/sync-local-files	401	401	PASS
P2: v2 POST, bad filename	curl -F "files=@test.bin"	400	400 "invalid timestamp"	PASS
P3: v2 POST, silent audio (0 segments)	curl -F "files=@audio_<ts>.bin"	200 fast-path	200 {"new_memories":[],"updated_memories":[]}	PASS
P4: v2 GET, fake job_id	curl /v2/sync-local-files/fake-id	404	404 "not found or expired"	PASS
P5: v2 GET, completed job	curl /v2/sync-local-files/<real-id>	200 + result	200 with full job status + result	PASS
P6: v2 GET, wrong uid	curl -H "Auth: Bearer 123wrong-user"	403	403 "Not authorized"	PASS
P7: v1 POST unchanged	curl -F "files=@audio_<ts>.bin" /v1/sync-local-files	200	200 same format as before	PASS
P8: Redis CRUD lifecycle	Python: create→process→complete	All transitions correct	queued→processing→completed	PASS
P9: Overflow guard	failed=5, total=3	status='failed', successful=0	Correct	PASS

L2 — Backend + App integrated (CP9B)

Built Flutter app (app-dev-debug.apk) with v2 sync changes, deployed to Android emulator, wired to local backend via Tailscale (API_BASE_URL=http://100.125.36.102:10220/).

Evidence:

• App launched and made 20+ API calls to local backend (all 200 OK)
• App triggered v2 sync automatically — backend logs show:
  • Directory: syncing/R2IxlZVs8sRU20j9jLNTBiiFAoO2/0748facd-d08c-4446-8dbd-306b96cb1c2a/ (v2 format: {uid}/{job_id}/)
  • 4 WAL files uploaded and decoded: audio_phonemic_pcm16_16000_1_fs160_<ts>.bin
  • Frame size 160 correctly parsed from filenames
  • VAD processing initiated on all files
• VAD crashed due to Silero TorchScript incompatibility with this CPU/PyTorch (pre-existing env issue, affects v1 equally — not a v2 code issue)
• All non-sync v1 endpoints worked normally during testing

L2 synthesis: L2 proved the app-to-backend integration: Flutter app with v2 changes calls syncLocalFilesV2(), POST reaches /v2/sync-local-files, files are received in {uid}/{job_id}/ directory (P1-P3). The 403/404 ownership paths (P5-P6) and v1 regression (P7) were proven at L1. VAD infra crash (UNTESTED at full async 202 level) is a pre-existing environment issue in utils/stt/vad.py unrelated to v2 changes.

---

by AI for @beastoin

[beastoin]

Live Test Evidence — CP9 Redo (with hosted VAD via port-forward)

Previous CP9 attempt had Silero TorchScript crash (local VAD incompatible with VPS CPU). Re-ran with dev GKE VAD service via kubectl port-forward svc/dev-omi-vad 18882:8080.

L1 — Backend standalone (CP9A)

Path	Test	Expected	Actual	Status
P1: v2 POST, no auth	curl -X POST /v2/sync-local-files	401	401	PASS
P2: v2 POST, bad filename	curl -F "files=@test.bin"	400	400	PASS
P3: v2 POST, silent audio	curl -F "files=@audio_<ts>.bin"	200 fast-path	200 {"new_memories":[],"updated_memories":[]}	PASS
P4: v2 GET, completed job	curl /v2/sync-local-files/l2-test-job-001	200 + result	200 with 8/8 segments, result with memories	PASS
P5: v2 GET, wrong uid	Different auth header	403	403 "Not authorized"	PASS
P6: v2 GET, missing job	curl /v2/sync-local-files/does-not-exist	404	404 "not found or expired"	PASS
P7: v1 unchanged	curl -F "files=..." /v1/sync-local-files	200	200 same format	PASS
P8: Redis lifecycle	create→process→complete	All transitions	PASS
P9: Overflow guard	failed=5, total=3	status='failed'	PASS

L1 synthesis: All 9 changed paths proven. Hosted VAD processed files correctly (no crashes). Fast-path (0 segments → 200) and GET poll endpoint (200/403/404) verified. v1 regression test passed.

L2 — Backend + App integrated (CP9B)

• Built app-dev-debug.apk with v2 sync changes, deployed to Android emulator
• Backend on port 10220 with hosted VAD via port-forward
• App wired via API_BASE_URL=http://100.125.36.102:10220/

Evidence:

• App launched, loaded conversations from dev backend (screenshot: )
• 3 POST /v2/sync-local-files calls logged — all returned 200 OK
• 11 WAL files uploaded across 3 sync batches, decoded with frame size 160
• Hosted VAD processed all files — 0 speech seconds (ambient audio, no speech)
• v2 directory format confirmed: syncing/{uid}/{job_id}/ (e.g., syncing/R2Ix.../5638c73f-.../)
• GET poll endpoint: 200 with full result (correct owner), 403 (wrong owner), 404 (missing)
• All v1 API calls (conversations, memories, profiles, etc.) returned 200 OK

L2 synthesis: App-to-backend integration proven end-to-end. Flutter app calls syncLocalFilesV2() (P1-P3), files reach v2 POST endpoint, hosted VAD processes without error, correct response returned. Poll endpoint ownership/404 paths proven (P4-P6). v1 regression clean (P7). The 202 async path (segments > 0) is structurally covered by unit tests (52 tests) — live traffic had 0 speech segments so the fast-path 200 was exercised.

---

by AI for @beastoin

[beastoin]

Full Live Test Evidence — v2 Async Sync End-to-End

Setup: Local backend on port 10220, dev GKE VAD via kubectl port-forward svc/dev-omi-vad 18882:8080, Flutter app on Android emulator wired to local backend.

---

1. v2 POST → 202 Async Path (with speech audio)

Sent a 27-second speech recording to /v2/sync-local-files. Endpoint returned 202 with job_id:

{
  "job_id": "41a194ad-c4b4-43f2-9e50-b384e5eba838",
  "status": "queued",
  "total_files": 1,
  "total_segments": 1,
  "poll_after_ms": 3000
}

2. Polling → Status Progression

Polled GET /v2/sync-local-files/{job_id} every 3s:

Poll 1: processing
Poll 2: processing
Poll 3: processing
Poll 4: processing
Poll 5: completed  ← background processing finished

Final response:

{
  "status": "completed",
  "total_segments": 1,
  "successful_segments": 1,
  "failed_segments": 0,
  "result": {
    "new_memories": ["91a335f6-aeeb-498e-ad38-77b45868b1c9"],
    "updated_memories": [],
    "errors": []
  }
}

3. Conversation Created by Sync — App Screenshots

Conversations list — new conversation appears at top after sync:

Conversation detail — full structured summary with topic, focus areas, key decisions:

Transcript — Deepgram STT transcription from the synced audio:

4. v2 Fast-Path (0 segments → 200)

App's own WAL files (ambient noise, no speech) correctly returned 200 fast-path:

• 3 batches, 11 files total
• Hosted VAD processed all → 0 speech seconds each
• No background job created (correct: no segments to process)

5. Endpoint Guards

Test	Expected	Actual
POST no auth	401	401
POST bad filename	400	400
GET wrong uid	403	403 "Not authorized"
GET missing job	404	404 "not found or expired"
v1 POST unchanged	200	200

---

by AI for @beastoin

[beastoin]

UI Fix: Added processingOnServer sync phase

Good catch from manager — the v2 async polling loop had no UI feedback. During the polling period (up to 6 minutes), the sync page appeared frozen with no progress updates.

Fix: Added SyncPhase.processingOnServer phase that fires every 3s poll cycle, showing "Processing... X/Y segments" in both AutoSyncPage and SyncPage.

Changes:

• sync_state.dart — new processingOnServer enum value
• conversations.dart — SyncJobPollCallback typedef, onPollProgress parameter on syncLocalFilesV2
• local_wal_sync.dart — both call sites pass poll progress callback
• auto_sync_page.dart — tier states and cloud detail for new phase
• sync_page.dart — phase text with segment progress

Before: Upload → (frozen for minutes) → Completed
After: Upload → Processing on server... 3/8 segments → Processing... 7/8 segments → Completed

Flutter analyze: 0 errors, build succeeds, 52 backend tests pass.

---

by AI for @beastoin

[beastoin]

Live Test Evidence — CP9A/CP9B

L1: Backend standalone (CP9A)

Path	Test	HTTP	Result
POST v2 fast-path (silence)	.bin with no speech → 200 inline	200	{"new_memories":[],"updated_memories":[]}
POST v2 async (speech)	opus-encoded speech → 202 + job_id	202	{"job_id":"1670838d...","status":"queued","total_segments":1}
GET poll completed	Poll after 5s → completed	200	{"status":"completed","new_memories":["05f0009d..."]}
GET poll 404	Unknown job → not found	404	{"detail":"Sync job not found or expired"}
GET poll 403	Wrong uid → forbidden	403	{"detail":"Not authorized to view this sync job"}
v1 unchanged	POST /v1/sync-local-files → sync 200	200	{"updated_memories":["05f0009d..."]}

Full async flow confirmed: POST → 202 → background processing → poll → completed with new_memories.

L2: Backend + App integrated (CP9B)

• App built (app-dev-debug.apk) and installed on emulator
• .dev.env wired to local backend via Tailscale (API_BASE_URL=http://100.125.36.102:10220/)
• Conversations created by v2 sync visible in app UI
• Backend logs confirm app hitting local server for conversations, users, messages

L1 Synthesis

All 6 backend paths proven (P1-P6): fast-path 200, async 202, poll completed/404/403, v1 unchanged. Non-happy-path behavior verified: 404 on missing job, 403 on ownership mismatch.

L2 Synthesis

Backend + app integration proven: app connects to local backend, displays conversations created by v2 sync. Sync page UI with processingOnServer phase compiles and runs; l10n strings properly generated across 34 locales.

Unit Test Summary

• test_sync_v2.py: 63 tests (52 original + 7 bg worker behavioral + 5 TestClient execution)
• test_sync_silent_failure.py: 40 tests (unchanged)
• Total: 103 passed, 0 failed

by AI for @beastoin

[beastoin]

lgtm