Repository navigation
RivetIT 26.10.43: quieter integration sync alerts
Fixes false-positive and repeated integration-sync alerts from the recovery watcher (src/Recovery/SyncWatch.php, run from cron/cron.php). DB 2.6.165: one idempotent step gated on 2.6.164 that creates one new table, recovery_watch_overrides (CREATE TABLE IF NOT EXISTS, nothing existing altered; db.sql carries the same table). Nothing is switched on or off by the update; until it runs everything works and only the new Mute / Expected interval / Forget history buttons are disabled. Two behaviour changes apply at once, with no setup: a sync whose newest run is over 14 days old is dormant and no longer mailed, and the same problem is mailed at most once per 24 hours.
What went wrong on a live install (about 12 emails a day): (1) "UniFi sync has stopped: no run for 94 days (expected every 17 min)": the watcher inferred the interval from old scheduled runs of the in-app UniFi sync, which now runs from an external script that does not write unifi_sync_log. (2) "Sophos Firewalls: the last 3 runs all ended in an error": an external script copied the old "sync every enabled integration" loop, hit the rivetit_agent integration (no vendor API) and logged its "Endpoint agent devices are managed from the device page" refusal as a failure.
- Dormant rule. If an integration's newest run is older than Sync dormant after (days) (new recovery setting, default 14, 0 = off; never sooner than twice its stale limit, so a monthly job is allowed to be quiet) it is dormant: no stale or failing mail, shown as "dormant since ". An integration that never ran in that long is dormant too. One "no longer running" notice goes out only if it was healthy before and has just become dormant (within 30 days of turning dormant), at most every 30 days; one that never ran, whose last run failed, or that has been dormant for longer is silent.
- Per-integration controls (table
recovery_watch_overrides: source, integration id, muted, expected interval in seconds, forget-before timestamp, note, who and when). On Admin > Backup > Integration sync watch: Mute / Unmute (no mail at all; the panel still shows what the state would be), Expected interval (1 minute to 90 days, minutes / hours / days, with a note; wins over the per-source setting and over an inferred interval, so an external daily script is declared as every 24 h and goes stale after three missed days), Forget history (every run so far is ignored; nothing is deleted; the integration shows as dormant until it runs again). Admin only, CSRF-checked, the integration must be one the watcher watches (neverrivetit_agent), every change is written to the log (Settings / Edit) and the audit trail (recovery.sync_watch_changed, with the acting user). - Repeat cadence. The same problem is mailed at most once per 24 h (new setting Repeat a problem alert after (hours), 1 to 720; it was a fixed 6 h). The first occurrence is still immediate. A recovered notice ("sync has recovered") goes out once when an integration that had alerted becomes healthy again, then the state is cleared so the next failure is immediate again. The window applies to every recovery alert (backup failed / stale, restore drill stopped, audit trail chain) as they share the same alert table.
- Failure rule. "The last N runs failed" counts only runs of that integration whose status is failed AND whose error text is real: blank text,
[],nulland therivetit_agentsentinel ("... agent devices are managed from the device page") are not errors, and a sentinel or empty-text run in the last N breaks the chain. A run stuck "running" counts only if it carries error text (it had none before, so this rule is now effectively off). Therivetit_agenttype is excluded everywhere (watcher, panel, handler). - State per integration on the panel: healthy, failing, stale, dormant or muted, with the last run (and its status), the expected interval and where it came from (set by an admin, the setting, inferred), why it alerted (the detail text and when the last alert went out) and the note. The inferred interval now uses scheduled runs only (
triggered_by = 0) once there are three. - Docs.
docs/RECOVERY_RUNBOOK.md: new "Integration sync watch and external sync scripts" section: an external sync script should write tormm_sync_log/unifi_sync_log(own integration id, afailedrow only with real error text) or the admin should set its expected interval; any script that copies the old "sync all enabled integrations" loop must skip typerivetit_agent. The Sync alerts row and the 24 h repeat are updated. - Tests: new
tests/recovery_syncwatch.php(100 checks): the exact live cases (UniFi, 94-day-old runs with a 17 min inferred interval, is dormant with no email; Sophos with mixed success / failed / sentinel rows is not failing, even six sentinel failures in a row; a genuinely failing integration with three real errors sends one email and none for 24 h, then one more; the expected-interval override (20 h healthy, 80 h stale); mute; the recovered notice once and a new failure immediate afterwards), the dormant boundaries and notice rules, forget history, the repeat setting, only-own-runs and the endpoint-agent exclusion, panel rendering (all five states, escaping, CSRF, no inline handlers), the handler (mute, interval, forget, refusals, audit and log lines) and a database without the new table.tests/recovery_drill.phpupdated for the 24 h window and the real-error rule;tests/endpoint_agent_migration.php+8 (2.6.165: gated,db.sqlequals the migrated schema, replays keep data, a dropped table is recreated). - Not verified: the mail was exercised through the queue stub and the in-app notification stub, not a real SMTP server. Nothing was deployed.