feat: Active watchdog — remediate stuck executions (#129)

[AndriiPasternak31]

Summary

• Cleanup service now actively reconciles DB execution state against agent process registries every 5 minutes
• Orphaned executions (DB says running, agent has no record) are recovered with descriptive error messages
• Executions exceeding their schedule's timeout_seconds are auto-terminated on the agent
• Capacity slots and execution queue state properly released on recovery
• WebSocket events broadcast to notify frontend of watchdog actions

Changes

• src/backend/db/schedules.py — Added get_running_executions_with_agent_info() (SQL JOIN with schedule + agent_ownership for timeout resolution) and mark_execution_failed_by_watchdog() (conditional UPDATE with race guard)
• src/backend/database.py — Exposed new DB methods via facade
• src/backend/services/cleanup_service.py — Watchdog reconciliation as first cleanup operation, shared _recover_execution() helper, parallel agent HTTP via asyncio.gather, per-execution error isolation, systemic failure detection, WebSocket broadcasting
• src/backend/main.py — Wired WebSocket manager to cleanup service
• docs/memory/changelog.md — Added changelog entry

Key Design Decisions

1. Watchdog runs BEFORE stale cleanup — releases resources before passive DB-only cleanup
2. Batch HTTP — one GET /api/executions/running per agent, parallel via asyncio.gather
3. Race-condition guard — WHERE status='running' prevents overwriting normal completions
4. Queue safety — only force_release if recovered execution holds the queue slot
5. Timeout resolution — COALESCE(schedule.timeout, agent.timeout, 900) respects per-agent config

Test Plan

• New integration tests pass: pytest tests/test_watchdog.py -v
• New unit tests pass: pytest tests/test_watchdog_unit.py -v
• Existing cleanup tests unaffected: pytest tests/test_cleanup_service.py -v
• Manual: Create schedule with short timeout, stop agent, verify watchdog recovers execution

Closes #129

Generated with Claude Code

[vybe]

This PR requires the following changes before merge:

• Update docs/memory/architecture.md lines 184 and 444 to mention active watchdog reconciliation (per Trinity methodology: feature change = changelog + architecture)
• Remove duplicate test files — keep either tests/ or src/backend/tests/, not both
• Fix default test password in conftest.py ("trinity" should be "password" to match dev environment)

Code and feature design look solid — the watchdog reconciliation logic, race-condition guards, Lua-script atomic release, and decision matrix are well-engineered. Just need the doc/test cleanup items above.

[vybe]

This PR still requires the same three changes from the previous review (March 25) before merge:

• Update docs/memory/architecture.md to mention active watchdog reconciliation (lines ~184 and ~444 — cleanup service description and monitoring API table)
• Remove duplicate test files — keep either tests/ or src/backend/tests/, not both
• Fix default test password in conftest.py ("trinity" → "password" to match dev environment)

Core feature implementation remains solid — watchdog reconciliation logic, race-condition guards, Lua-script atomic release, and decision matrix are all well-engineered. Just need the doc/test cleanup items above.

[vybe]

Inline Notes

src/backend/tests/conftest.py — Line 450: Default TRINITY_TEST_PASSWORD is "trinity" but the dev environment uses "password". Tests will fail auth out of the box.

tests/ vs src/backend/tests/ — Both locations contain identical test_watchdog.py and test_watchdog_unit.py. Pick one location and remove the other. The new src/backend/tests/ infrastructure (conftest.py, utils/ — ~800 lines) is also substantial and may conflict with existing test conventions.

docs/memory/architecture.md — Not in the diff at all. Needs watchdog mentions added.