monitor:aggregate-check-metrics runs out of memory (exit 137) in production — OOM takes down MySQL and hangs the server

[Satendra-SR]

Summary

The scheduled command monitor:aggregate-check-metrics is exhausting memory on the production host (monitor.coloredcow.com). It gets OOM-killed (exit code 137), and because the box is memory-constrained, the OOM-killer then takes down MySQL, which cascades into failed uptime checks and a fully unresponsive server. This recurred repeatedly on 2026-07-03 (site down, SSH/HTTP timing out).

Impact

• Production monitoring app (monitor.coloredcow.com) goes fully down and needs a manual reboot / stop-start to recover.
• When it's down, we lose uptime monitoring for the whole fleet — the tool that's supposed to catch outages is itself offline.
• Recurs on every scheduler cycle until the process is stopped.

Evidence (production laravel.log, 2026-07-03)

[08:18:19] production.ERROR: Scheduled command [... monitor:aggregate-check-metrics] failed with exit code [137].
[10:55:03] production.ERROR: SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, Host: 127.0.0.1 ...)
[10:55:03] production.ERROR: Scheduled command [... monitor:check-uptime] failed with exit code [1].

Exit code 137 = 128 + 9 (SIGKILL) — the kernel OOM-killer terminated the process for using too much memory. The Connection refused immediately after is MySQL going down under the same memory pressure.

Root cause

monitor:aggregate-check-metrics appears to load a large result set into memory (aggregating check/metric history in PHP) rather than aggregating in the database or streaming. On a small host running MySQL + PHP-FPM alongside it, that pushes total memory usage past the limit → OOM.

Likely introduced/exposed by #82 (organization-dashboard) — the metrics aggregation added for the new dashboard.

Proposed fix (app side)

1. Aggregate in SQL, not PHP — replace any ->get()/->all()/large-collection aggregation with selectRaw(...) + GROUP BY, or process in batches via chunkById() / a lazy cursor so the whole dataset is never held in memory at once.
2. Bound memory per run — cap the time window / rows processed per invocation (e.g. aggregate incrementally rather than reprocessing all history each run).
3. Lower concurrent_checks in config/uptime-monitor.php from 10 (reduces peak memory during monitor:check-uptime too).
4. Make the scheduler resilient — ensure withoutOverlapping() on these commands so runs can't stack, and that one command failing doesn't abort the rest of the schedule.
5. (Optional) Move heavy work off the inline sync queue (QUEUE_CONNECTION=sync) if a worker can be run within the host's memory budget.

Acceptance criteria

• monitor:aggregate-check-metrics completes within a bounded memory ceiling (e.g. < 128 MB) regardless of history size.
• No exit code [137] / OOM entries in laravel.log after deploy.
• Peak memory during a full scheduler cycle (check-uptime + aggregate) verified to stay within the host's available RAM.

Environment

• Production monitor host: small single instance (~1 GB RAM), MySQL + PHP-FPM 8.3 co-located.
• Package: spatie/laravel-uptime-monitor, 25 active monitors.
• Infra mitigation already applied (2026-07-03): root volume grown and a 2 GB swap file added, so the box now degrades under memory pressure instead of hard-hanging. This makes the command survivable but not efficient — the app-side fix above is still needed so aggregation isn't chronically swapping.

[andrianvaleanu]

Production crashes during monitoring are really annoying. Most teams fix this by using chunking or a tool like https://pulsetic.com/ too offload the heavy work.