Custom checks: user-defined scheduled checks with notifications (Monitor)

[MaEscalanteHe]

Hi! Now that 1.2.7 is out I'd like to pick up a Monitor feature and check it fits before writing code.

What. "Custom checks": checks the user defines from Settings, run on a schedule, that notify through the existing channels when they start failing, remind about once a day while they keep failing, and notify when they recover.

• Types: HTTP(S) (2xx or an expected status, optional "body contains", optional "accept invalid certificates"), TCP port, systemd service on the host, and an advanced "command" type (bash, exit 0 = OK).
• Schedule: simple picker (every N minutes ≥ 5 / every N hours / daily at HH:MM / weekly) or a raw 5-field cron, evaluated with the existing lxc_apps.cron_matches.
• Notifications: two new event types, custom_check_failed and custom_check_recovered (group health).

How. It follows the LXC scheduled-update precedent rather than health_monitor.py: a dedicated scheduler thread like _scheduler_loop, each run in its own worker with a timeout (max 4 concurrent), and events emitted with notification_manager.emit_event. The reason: checks are user-scheduled and can take seconds, and get_detailed_status() also runs synchronously from /api/health/* requests. Config lives in /etc/proxmenux/custom_checks.json (0600, atomic writes like custom links); only the last result is kept, in user_settings. Not shown in the Health panel.

Security. Commands run as root, so every route (GET included) uses @require_admin_scope, the same guard as the terminal ticket: read-only API tokens, and the read-only profiles planned for later, can't see or run checks. The form also shows a "runs as root" warning.

PR. One PR to develop, unittest coverage, runtime templates in the 8 languages, UI strings in English and a docs section under Notifications. I'll test it on PVE 9.2.

Does this overlap with the planned Addons work, or would you prefer anything done differently?

[MacRimi]

Hi @MaEscalanteHe, thank you for writing this up before starting. It's clear and well thought out, and I like the idea a lot.

It doesn't overlap with the Addons idea. Addons would be third-party applications that plug into ProxMenux; custom checks are a feature of the Monitor itself. If Addons ever happen, a check type could be one of the things an addon brings, but nothing in your design would get in the way.

The approach looks right to me: following the scheduled-update precedent instead of health_monitor.py, a timeout per run, the config in /etc/proxmenux with atomic writes, and @require_admin_scope on every route, GET included. A few thoughts:

The command type is the one I'd be most careful with. Since it runs as root, I'd keep the warning very visible and make the command something only an admin can create or edit, exactly as you describe.
For the reminders, it would be good if the new events followed the same rules as the other health notifications (Quiet Hours, the daily digest and dismissals), so a user handles them the same way.
It's fine as one PR to develop.
1.2.8 is going to be a short corrective release, so this would fit in the beta that comes after it. Thanks again for offering to take it on!

[murtazox04]

I read the notification path on develop (a6ada40) with MacRimi's note in mind ("follow the same rules as the other health notifications"). A few things in notification_manager.py decide whether that happens on its own or needs help from the scheduler:

1. Quiet Hours will buffer every run, not every failure. In _dispatch_to_channels, a sub-CRITICAL event inside the window goes to _buffer_quiet_event and the loop continues, so delivered stays False. _finish_delivery_claim only stamps notification_last_sent when something was delivered, so the cooldown never starts. A check that fails every 5 minutes through a 22:00–06:00 window inserts ~96 rows into quiet_pending (that insert has no dedup, unlike _buffer_digest_event), and the morning summary reads "Health: 96" with 8 lines and "+88 more". I'd keep the transition logic in the scheduler itself: per check, store last_state and last_notified_at next to the last result, emit custom_check_failed only on OK→FAIL or when 24 h have passed since the last emit, and emit custom_check_recovered only on FAIL→OK. Then Quiet Hours and the digest see one event per real change and work as they do for everything else.

2. The 24 h default cooldown hides flaps. Neither new type is in the overrides in _cooldown_seconds, so both fall to DEFAULT_COOLDOWNS (86400). Fingerprint is hostname:entity:entity_id:event_type, so failed and recovered have separate cooldowns: fail 10:00, recover 10:10, fail again 11:00 → the second failure is dropped by _claim_delivery, and the check is down with the last message saying "recovered". If the scheduler owns the cadence (point 1), give both types a short in-code cooldown like _STATE_EVENTS (5 s), only to absorb duplicates. Use the check id as entity_id so two checks never share a fingerprint.

3. Recoveries should skip aggregation and the digest. The burst aggregator groups any type not in _AGGREGATION_EXEMPT_EVENTS by event_type:hostname, and the comment there explains why error_resolved is exempt. custom_check_recovered belongs in the same set. If it's INFO, _should_buffer_for_digest also holds it until digest_time on digest channels: a failure at 14:00 arrives live and the recovery comes the next morning. Add it to _DIGEST_EXEMPT_EVENTS (or send it at the failure's severity) so the pair stays together.

4. The health rate limit drops events, it doesn't defer them. GROUP_RATE_LIMITS['health'] is 3/min, 20/h, shared with the existing health monitor, and when allow() says no, _dispatch_event just returns. A network blip that fails or recovers 5 HTTP checks on the same tick sends 3 and silently loses 2. With recoveries exempt from aggregation (point 3), they hit this limit directly. One event per tick that lists every check that changed state avoids it, the same way app updates are batched by their producer (AppImage/scripts/tests/test_lxc_app_notification_batch.py).

5. Severity decides Quiet Hours. Only CRITICAL gets through the window. A per-check severity (default WARNING) lets someone mark "router reachable" as CRITICAL without paging overnight for a homelab dashboard check.

6. Dismissals. There's no Health panel entry to dismiss from, but the store already exists: health_persistence.acknowledge_error(key, suppression_hours=...) and is_error_acknowledged(key), and Settings → Active Suppressions lists them (get_dismissed_errors). Checking is_error_acknowledged(f"custom_check_{id}") before emitting would give the same "silence for N hours / permanently" semantics. Two things to confirm first: _acknowledge_error_impl derives the category from a prefix table when no errors row exists, and custom_check_ isn't in it; and writing a row into errors may surface it in the Health panel you wanted to stay out of. A plain muted_until per check in custom_checks.json is the simpler alternative if the shared panel isn't worth it.

Disclosure: I maintain Cue (https://github.com/murtazox04/Cue), an open-source notification policy library (quiet hours, caps, digests). It isn't a fit here: ProxMenux already has the whole pipeline, and the fixes above are a few lines inside it.

[MacRimi]

Thank you so much @murtazox04, this is a wonderful review. Going through the real delivery path like this saves a lot of work before a single line is written, and I'm very grateful for it.

@MaEscalanteHe, I'd take this into account when you start. For me the key is having the scheduler notify only when a check changes state, plus the daily reminder while it keeps failing, because the rest becomes much simpler that way. A severity per check also sounds like a good idea.

On dismissals, I think I'd lean towards the simpler option: a mute stored with the check itself would keep custom checks out of the Health panel, as you planned. It's your call when you get to the design, though.

Thank you both, it's a pleasure to see this taking shape.

[murtazox04]

Thanks, glad it helps. One small thing for the mute stored with the check, since it interacts with the "notify on state change" rule: keep evaluating and recording the check's state while it is muted, and only skip the send. Otherwise the stored "last state" goes stale, and when the mute ends the first run looks like a fresh transition (a "failing" alert for something that has been failing all week, or a "recovered" for a check that recovered days ago). With the state kept current, unmuting can simply send one current-status notice if it is still failing and restart the daily reminder clock from there. A muted_until timestamp rather than a boolean also lets "mute for 24 h" expire on its own without anyone remembering to undo it.