You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The MonitoringService fleet-health loop is started unconditionally in the FastAPI lifespan of every uvicorn worker, with no cross-worker leader election. The prod compose runs the backend with --workers 2, so two independent monitoring loops run concurrently: every health check is performed and stored twice per cycle, and — because check_network_health() feeds probe failures into the per-agent circuit breaker (#631) — every probe failure is counted twice, effectively halving the breaker's failure threshold and probe budget. This directly amplifies #1463 (breaker opening for busy-but-healthy agents).
Component
Backend / Monitoring Service (startup wiring in main.py)
Priority
P2
Error
Observed on a production instance — every monitoring pass is duplicated, a few ms apart:
And agent_health_checks stores two identical rows per layer per cycle (same second, e.g. two docker + two network + two business + two aggregate rows at 08:09:47, 08:10:50, 08:11:52, ...).
Location
File: src/backend/main.py (~lines 532–545) — _start_monitoring_delayed() / asyncio.create_task(...) in the lifespan, runs in every worker process
File: src/backend/services/monitoring_service.py — start_monitoring_service() / MonitoringService._run_loop() have no distributed lock
Root Cause
Lifespan code executes per worker process. The scheduler was long ago split into its own single-process service (with Redis locking) precisely to avoid multi-worker duplication — the monitoring loop (re-armed at boot by #1121) has no equivalent guard, so with N workers the fleet gets probed N times per interval.
Consequences:
Duplicate rows in agent_health_checks (storage bloat, misleading dashboards/rollups)
Doubled probe load against every agent-server, worst exactly when an agent is already busy
Reproduction Steps
Run the backend with --workers 2 (prod compose default) and monitoring enabled
grep "fleet_health_check pass complete" <backend logs> — two lines per cycle, milliseconds apart
Query agent_health_checks — two rows per check layer per agent per cycle
Suggested Fix
Give the loop a cross-process guard, mirroring the scheduler's approach:
# start_monitoring_service(): acquire a Redis lock before starting the loop;# non-holders skip. Holder refreshes TTL each cycle so a dead worker's lock expires.lock=redis.lock("monitoring:leader", timeout=interval*3)
ifnotlock.acquire(blocking=False):
logger.info("Monitoring loop not started in this worker (another worker holds the lease)")
return
Alternatively: move the loop into the scheduler service (already single-process and lock-aware), or gate on an env flag set for exactly one worker.
Summary
The MonitoringService fleet-health loop is started unconditionally in the FastAPI lifespan of every uvicorn worker, with no cross-worker leader election. The prod compose runs the backend with
--workers 2, so two independent monitoring loops run concurrently: every health check is performed and stored twice per cycle, and — becausecheck_network_health()feeds probe failures into the per-agent circuit breaker (#631) — every probe failure is counted twice, effectively halving the breaker's failure threshold and probe budget. This directly amplifies #1463 (breaker opening for busy-but-healthy agents).Component
Backend / Monitoring Service (startup wiring in main.py)
Priority
P2
Error
Observed on a production instance — every monitoring pass is duplicated, a few ms apart:
And
agent_health_checksstores two identical rows per layer per cycle (same second, e.g. twodocker+ twonetwork+ twobusiness+ twoaggregaterows at 08:09:47, 08:10:50, 08:11:52, ...).Location
src/backend/main.py(~lines 532–545) —_start_monitoring_delayed()/asyncio.create_task(...)in the lifespan, runs in every worker processsrc/backend/services/monitoring_service.py—start_monitoring_service()/MonitoringService._run_loop()have no distributed lockRoot Cause
Lifespan code executes per worker process. The scheduler was long ago split into its own single-process service (with Redis locking) precisely to avoid multi-worker duplication — the monitoring loop (re-armed at boot by #1121) has no equivalent guard, so with N workers the fleet gets probed N times per interval.
Consequences:
agent_health_checks(storage bloat, misleading dashboards/rollups)record_failure()feed into the circuit breaker: the 3-failure open threshold is reached in ~1.5 real probe rounds and the 10-probe dormant budget in ~5 — see bug: circuit breaker opens for busy-but-healthy agents — /health probe timeouts during long CPU-bound executions count as liveness failures #1463Reproduction Steps
--workers 2(prod compose default) and monitoring enabledgrep "fleet_health_check pass complete" <backend logs>— two lines per cycle, milliseconds apartagent_health_checks— two rows per check layer per agent per cycleSuggested Fix
Give the loop a cross-process guard, mirroring the scheduler's approach:
Alternatively: move the loop into the scheduler service (already single-process and lock-aware), or gate on an env flag set for exactly one worker.
Environment
85a7a85a(docker-compose.prod.yml,--workers 2, PostgreSQL backend)Related