v0.3.2 - Monitoring Loop and Probe Fixes

v0.3.2 - Monitoring Loop and Probe Fixes

A patch release fixing two ways the monitoring engine could quietly drift: duplicate background loops, and remote probes reporting stale results after falling behind.

Fix: probes skip stale checks instead of replaying a backlog (#59)

A probe only skipped its backlog when it started. A probe that fell behind while running kept working through old checks and reported their results against rounds dispatched long ago. One production probe was nearly 17 days behind, so its votes in multi-probe decisions were for long-finished rounds.

• Probes now skip checks older than MAX_CHECK_AGE (default 2m) and catch up immediately
• Age is measured against Redis's clock, so clock skew between a probe and the server cannot make it skip every check

Fix: exactly one copy of each monitoring loop (#60)

The check dispatcher and result processor are self-requeuing jobs, and every Horizon restart or retried job used to leave one more copy running forever. Only one copy of each now survives, and leftover copies stop on their next run after upgrading.

• Starting Horizon always starts the loops fresh, so restarting Horizon (php artisan horizon:terminate) is a clean reset. The monitoring:dispatcher:initialized cache flag is no longer used
• If the loop state is ever lost, monitoring recovers on its own instead of stopping

Upgrading

git pull
./setup.sh --upgrade

Update remote probes to probe-node v0.3.2 to get stale-check skipping, e.g. with the probe repo's upgrade.sh. Older probes keep working. See UPGRADING.md.