Repository navigation
RivetMSP 26.10.20 - the integration sync watcher stops crying wolf
Fixes the false-positive and repeating "sync stopped / sync failing" emails the Wave 1 recovery watcher produced on a live install (about twelve a day): UniFi sync has stopped: no run for 94 days (expected every 17 min) (UniFi now syncs from an external daily script that does not write unifi_sync_log; the old in-app scheduled runs were still used to infer the interval) and Sophos Firewalls: the last 3 runs all ended in an error (an external per-minute script looped over every enabled integration, including the built-in rivetit_agent type, and wrote that type's refusal, "Endpoint agent devices are managed from the device page, not through an RMM client.", against the previous integration's row). No RivetCore change: the pin stays at 1.0.0-rc.16.
Deploy notes: back up the database, pull this tag, php scripts/update_cli.php --update_db (2.6.91 to 2.6.92: ONE new table, recovery_watch_overrides, CREATE TABLE IF NOT EXISTS, nothing existing altered; the code works before it runs, only saving a mute or an expected interval needs the table). No new cron line, no nginx change, no new composer package.
- Dormant integrations. An integration whose newest run is older than Dormant after (recovery setting
sync_dormant_days, default 14 days, 0 = never) is dormant: no email, shown as "dormant since " with Mute and Forget history actions. At most ONE "no longer running" notice per 30 days, and only when its last run was good and it has just gone quiet (within a week of crossing the line): an integration that went quiet long ago, such as the 94-day UniFi case, produces nothing. A never-run integration older than the setting is dormant too. - Per-integration overrides (table
recovery_watch_overrides: source, integration id, muted, expected interval in seconds, note, updated by/at). Admin > Backup > Integration sync watcher: Mute / Unmute, Edit (expected to run every N hours, a note, muted), Forget history. An external daily script can be declared "expected every 24 h"; a declared interval is judged at 3x and opts out of dormancy (a declared script that stops is stale). Every change is CSRF-protected and written to the audit log with who, which integration and the new values; Forget history needs its confirmation box and removes only that integration's sync-log rows and alert state. - Repeat cadence. The same problem is mailed at most once per 24 hours (new setting
alert_repeat_hours, default 24, was a fixed 6; applies to every recovery alert, the e-mail footer shows the live value), the first occurrence is immediate, and a recovery ("sync is working again" / "running again") is announced once, only when the problem had actually been mailed. Stale and failing no longer both keep state for one integration (so no double recovery notice). - Failure rule. "The last N runs failed" counts only that integration's runs with a real error: a failed status with a message that is not the "Endpoint agent devices are managed from the device page" sentinel. Sentinel rows are not the integration's runs at all (they neither fail it nor keep it looking fresh). Integration types without a vendor API (
rivetit_agent,SyncWatch::UNWATCHED_RMM_TYPES) are not watched. - Panel. A per-integration table: state (healthy / failing / stale / dormant / muted), last run (UTC), expected interval (and whether declared), the reason, actions. Names and notes are escaped; no inline handlers.
- Docs.
docs/RECOVERY_RUNBOOK.mdsection 2a: the watcher, and the rule for external sync scripts (writermm_sync_log/unifi_sync_log, or declare the interval or mute; any script that copies the "sync all enabled integrations" loop must skip typerivetit_agent). - Other.
SyncWatch::snapshot()(pure read, used by the panel) andSyncWatch::evaluate()(pure verdict);RecoveryAlerts::recovered();RecoveryAlerts::raise()takes an optional repeat window (the dormant notice uses 30 days). New:src/Recovery/WatchOverrides.php,tests/recovery_watch.php,tests/browser/recovery_watch_smoke.mjs.
DB and tests: DB 2.6.92; db.sql gains recovery_watch_overrides (a fresh import and the 2.6.91-to-2.6.92 update produce the same schema, no uca1400; the step is idempotent). Tests: new tests/recovery_watch.php (80 checks: the exact live cases, UniFi with 94-day-old runs, Sophos with sentinel rows mixed in, a genuinely failing integration mailed once then not for 24 h, the 23 h / 24 h / 48 h boundaries, recovered notice once, a declared 24 h interval, mute, the 30-day dormant notice, forget history, the panel's rendering and escaping, the handlers with their audit lines and clamps, the session-time-zone guard) and a browser smoke tests/browser/recovery_watch_smoke.mjs (6 checks: real login, every state rendered, Edit / Mute / Unmute / Forget history through the real forms, phone width). tests/recovery_drill.php 125 (the 6 h window became 24 h, failed runs carry a message), tests/wave1_integration.php 87 and tests/endpoint_agent_module.php 87 (the 2.6.92 step and its idempotence), tests/rmm_ui.php 1755, tests/endpoint_agent_smoke.php 47, tests/rmm_installer_ui.php 141, golden replay 10, RivetCore PHPUnit 229 tests, and the security / crypto / platform / compliance / mail-intake suites all pass. Known failures, identical on 26.10.19 (verified on a clean checkout): sla_pause 2, recovery_integration 3, the strict golden replay.