fix: replace the backend health poller with a connectivity state machine (no more false Server Unreachable during recordings)

[Valtora]

Pull Request

Description

Replaces the frontend backend-health poller with a connectivity state machine, eliminating the false "Server Unreachable" alerts that fire during meeting recordings.

The bug

"Server Unreachable" was raised while the backend was fully healthy (verified from container logs: nojoin-api had zero restarts and served /api/v1/system/health 200 throughout the incident). The old poller (frontend/src/lib/serviceStatusStore.ts) committed a category error: it equated "a synthetic health probe failed" with "the backend is down". During a recording the Nojoin tab is backgrounded, the browser throttles its timers (the 60s-cadence backup poll in the logs is the tell-tale), and on the hidden to visible transition the wake-up probes time out client-side within the 5s window before ever reaching the server. Three aborts in ~7s tripped backendFailCount > 2 and raised the alert. #113 only guarded the visibilityState === "hidden" case, not the wake-up burst.

The architecture

Reachability is now derived from observed request outcomes, not from a parallel synthetic probe channel:

1. Passive health. Every real request already flows through the single axios instance (frontend/src/lib/api/client.ts). Its response interceptor now classifies each outcome by failure mode and feeds the monitor: any HTTP response (even 4xx/5xx) proves the server answered; only a missing response or a 502/503/504 gateway error indicates unreachability. During a recording, segment uploads (which route through the same client) continuously prove reachability, so the false alarm is now structurally impossible.
2. A pure state machine (connectivity/reducer.ts) with explicit states online | checking | offline | unreachable. Status is a derived function of (browserOnline, freshness of last success, probe-failure streak). It is a pure (state, event) -> state reducer, exhaustively unit-tested without timers or network.
3. Browser-native connectivity. online/offline events give client-offline its own distinct state and message ("You're offline") instead of blaming the backend.
4. Active probing demoted to an idle fallback (connectivity/monitor.ts). A dedicated liveness probe runs only when real traffic has not proven reachability recently; escalation to unreachable requires the browser to be online AND no success for a staleness window AND repeated dedicated-probe failures. A real request failure forces a prompt verification probe so genuine mid-session outages are still caught quickly.

serviceStatusStore is slimmed to health content only (db, worker, version, deployment warnings); its failure to load can no longer assert "unreachable". MeetingControls reads reachability from the connectivity store.

Backend cleanup (included)

Exempts /health and /api/health from EnforceCanonicalHttpsMiddleware (backend/main.py). Liveness endpoints are called by container/orchestrator health checks and internal uptime monitors over plain HTTP without forwarded-proto headers; they were being 307-redirected to the canonical HTTPS origin (log noise, and a broken plain-HTTP monitor). They now answer a clean 200. The three middleware tests that used /api/health as their probe were repointed at a guarded path so they still exercise the redirect/reject/forwarded-proto logic.

New dependencies: none.

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that changes existing behaviour)
• Documentation update

Checks run

• Backend tests: pytest backend/tests 865 passed locally.
• Python quality: ruff check and ruff format --check on the changed backend files, both clean.
• Frontend lint: cd frontend && npm run lint clean.
• Frontend unit tests: cd frontend && npm run test 199 passed (39 files), including 21 new connectivity reducer/driver tests.
• Frontend build: cd frontend && npm run build succeeded.
• Docs validation: not applicable (no docs change).
• Alembic validation: not applicable (no migration).

Migration impact

• No database migration in this PR.

Documentation impact

• No documentation change required (internal reliability fix; no setup/operations/user-workflow change).

Security impact

• Touches auth/exposure-adjacent code. EnforceCanonicalHttpsMiddleware now exempts only the two liveness endpoints (which already returned a static {"status": "ok"} and carry no sensitive data). HTTPS enforcement and TrustedHostMiddleware (which runs first) are unchanged for every other path; the redirect, reject, and forwarded-proto tests were repointed at a guarded endpoint and still pass. docs/SECURITY.md boundaries preserved.

Manual verification

Reachability is now driven by real traffic, so the key manual check is a recording: start a meeting capture, background the tab (switch to the meeting app) for a few minutes, and confirm no false "Server Unreachable" toast appears while segment uploads continue. Additional checks: toggle the OS network off to confirm the distinct "You're offline" message; stop the backend container mid-session to confirm a genuine outage still surfaces "Server Unreachable" within ~15-30s and clears on recovery; over plain HTTP, curl http://<host>/api/health now returns 200 rather than a redirect.