Runner Box

🌍 View in other languages

Self-Hosted Runner Box Operations (.113 pool)

The self-hosted pool (self-hosted, omni-release labels) runs on the 16 GB box at 192.168.0.113. Two failure modes recurred on release days and were, until v3.8.49, manual discipline; the janitor script codifies them (WS3.3 of the quality plan):

1. Orphaned temp/work dirs filling the disk → disk-full SQLite errors mid-job.
2. >4 concurrent runners → OOM-killed jobs (8-wide killed jobs twice on the v3.8.47 release day; 4-wide is the proven ceiling).

Install the janitor (one-time, on the box)

sudo mkdir -p /opt/omniroute-ops
sudo cp scripts/ops/runner-janitor.sh /opt/omniroute-ops/
sudo chmod +x /opt/omniroute-ops/runner-janitor.sh
( sudo crontab -l 2>/dev/null; echo '*/30 * * * * /opt/omniroute-ops/runner-janitor.sh >> /var/log/runner-janitor.log 2>&1' ) | sudo crontab -

What it does every 30min: sweeps runner temp leftovers older than 24h, alerts at ≥85% root-disk usage, and alerts when more than the runner ceiling (default 4, tunable via the script's own environment) of Runner.Listener processes are up. Alerts land in /var/log/runner-janitor.log with a non-zero exit (grep for ⚠).

Operating rules

• Ceiling: 4 runners on the 16 GB box. Runners 5–8 stay STOPPED except for explicit off-peak experiments — never during a release window.
• Stopping a runner mid-job cancels the job (observed live): systemctl stop only when its runner is idle (Runner.Listener without a Runner.Worker child).
• The .15 VPS is homologation-only — never runs CI runners.