A triage crew member got stuck

[danmt]

The file tooling isn't responding right now, so here it is inline — copy it straight into your incident docs.

---

Postmortem: claude-triage VM unresponsive, 2026-08-14

Status: Resolved Severity: Low (single non-production VM; no host or user-facing impact) Duration: ~12 minutes of unresponsiveness (20:12:45 – 20:24:25 UTC)

Summary

claude-triage, an Incus virtual machine on dev-server-1, became unresponsive to normal management commands on the evening of 2026-08-14. The guest kernel remained alive and its Incus agent kept answering, but nothing inside the VM could make forward progress.

The cause was a scheduled cron job that invokes Claude Code. One invocation grew to 3.42 GB of resident memory on a VM with roughly 4 GB of RAM and no swap. This drove the guest into an unrecoverable memory state: the kernel OOM killer fired, but killed only one process out of a larger runaway tree, and the surviving siblings kept the machine pinned.

The VM was recovered with a forced stop and restarted cleanly. The host was never affected.

Environment

Host	dev-server-1
Instance	claude-triage (Incus virtual machine, not a container)
Guest OS	Debian GNU/Linux 13, kernel 6.12.96+deb13-amd64, x86_64
Guest RAM	~4.03 GiB (1,031,276 pages, 45,437 reserved)
Guest swap	None configured (Total swap = 0kB)
Boot under investigation	2026-08-14 11:08:45 UTC
Workload	Cron-driven Claude Code runs, plus a Docker daemon

Root cause

A single cron-invoked process consumed roughly 85% of the VM's total RAM.

The process tree systemd printed while trying to restart cron shows the full chain:

cron → sh → bash → flock → bash → timeout → claude → zsh → zsh → zsh → 2.1.220

2.1.220 is the Claude Code binary, which renames itself to its version string at runtime. A head process appears alongside it, suggesting the job pipes Claude's output — a plausible route to unbounded memory growth if the upstream side produces a large stream.

Three conditions turned a large allocation into a hung machine:

1. No swap. Total swap = 0kB. The kernel had no secondary tier to page anonymous memory out to.
2. Page cache fully evicted. active_file:0kB, inactive_file:112kB, and all_unreclaimable? yes. With essentially no page cache, every code page fault became a major fault requiring a disk read. Ordinary processes — including sshd and the agent's helpers — thrashed rather than ran. This is why the VM looked dead while the kernel was demonstrably alive.
3. Partial OOM kill. The kernel killed one process; the rest of the tree survived inside the same cgroup and kept allocating. The default OOM policy does not kill the whole tree, so the condition persisted for ~12 minutes after the kill.

The job ran directly under cron.service with no memory limit, so its failure took cron down with it.

What was ruled out

• Host resource exhaustion. Host dmesg shows no OOM events, no I/O errors, and no storage-layer problems; the only entries are from the host's 2026-07-29 boot.
• Hung QEMU / hypervisor fault. incus info returned live guest OS version, kernel version, hostname and process count. Those fields are supplied by the in-guest incus-agent, so the guest kernel and agent were both running.
• Disk exhaustion. Guest root usage was 11.49 GiB with no ENOSPC or EXT4 errors.
• Kernel panic. The guest journal for the affected boot ends with OOM handling, not a panic.

Contributing factors

• Diagnostic friction. Initial recovery attempts used two commands that do not exist in the form used: incus top <instance> (the argument is a remote, not an instance) and incus down (the verb is incus stop). This added minutes before --force was reached.
• Log volatility. Because recovery required a forced stop, the guest's in-memory ring buffer was lost. Diagnosis was only possible because the guest had a persistent journal at /var/log/journal, making journalctl -b -1 available after reboot. Had the journal been RAM-only, this incident would have been unexplainable.

Impact

Limited to the claude-triage VM. Any cron work scheduled during the window did not run, and cron.service itself was down. No host impact, no data loss, no user-facing impact. The forced stop risked an unclean filesystem, but the VM rebooted without errors.

Remediation

Done

• VM recovered via incus stop --force and restarted.
• Guest confirmed to have a persistent journal, which made post-hoc analysis possible.

Recommended

1. Contain the job's blast radius (highest priority). Run the cron job in its own cgroup scope with a hard memory ceiling, so a runaway invocation is killed locally while the rest of the VM keeps its headroom:

   systemd-run --scope --unit=claude-triage-run \  -p MemoryMax=2G -p MemorySwapMax=0 -p OOMPolicy=kill \  /path/to/script.sh
   

   OOMPolicy=kill terminates the entire process tree — the specific gap that let this incident persist after the kernel's kill. Better still, migrate the job from a crontab entry to a systemd timer + service so these limits, and a TimeoutStartSec= replacing the timeout wrapper, live in a version-controlled unit file.
2. Deprioritise the job in global OOM selection. In the job script, before launching: echo 500 > /proc/self/oom_score_adj.
3. Give the VM reclaim headroom. Either raise the limit (incus config set claude-triage limits.memory 8GiB) or add compressed swap via systemd-zram-generator.
4. Kill early rather than late. Install earlyoom, which acts on free-memory thresholds and works without swap, rather than waiting for the kernel to reach the fully-unreclaimable state.
5. Understand the 3.4 GB. Instrument the job to record peak RSS (/usr/bin/time -v) to establish whether this is a one-off spike or a growing trend, and inspect what is being piped through head.
6. Add a watchdog rather than a scheduled restart. A periodic host-side liveness check (incus exec claude-triage -- true) that dumps df -h, free -m and top processes by RSS on failure before restarting — preserving evidence and recovering automatically.

Explicitly not recommended: daily restarts

A scheduled restart was considered and rejected. This was not a slow leak that accumulates over uptime — it was a single invocation exceeding available memory, which a daily restart would not have prevented. Scheduled restarts also destroy the evidence needed to diagnose recurrences, and mask the misbehaving job rather than fixing it. They remain a reasonable workaround for a confirmed, unpatchable leak; that is not the situation here.

Lessons

1. A responsive Incus agent does not mean a healthy VM. Populated OS/kernel/hostname fields in incus info prove the guest kernel is alive — which usefully rules out hypervisor faults and points diagnosis inward, toward resource exhaustion.
2. Zero swap plus zero page cache is a hang, not a crash. The failure mode of memory exhaustion without swap is indefinite thrashing, which presents as a hung machine rather than a clean OOM kill.
3. The kernel OOM killer kills a process, not a workload. Unbounded multi-process jobs need cgroup limits with OOMPolicy=kill to fail cleanly.
4. Persistent journals are what make forced recovery survivable. Without /var/log/journal, forcing the VM down would have destroyed the only record of the cause.

[dan-claude-bot]

Accepted, in the part that is crew's — and duty.log names the invocation

Triage 2026-08-14 20:5xZ, from this box's own logs rather than from the report. Thank you for writing it up at this depth; three of the six recommendations are crew engine work and are now recorded on #327's to-mint list (entries 20–22, taking it from nineteen bullets to twenty-two). Nothing is minted today: 0.1.3 is the standing window, it is at release-init step 1, and your instruction is that this window's issues are minted together.

1. Which invocation it was

duty.log on claude-triage closes this out. The last thing to start before the window is:

2026-08-14T20:06:15Z SESSION START kind=hygiene key=heavy-duty/incubator timeout=3000s

and it has no SESSION END, ever. Its session log — logs/20260814T200615Z-hygiene-heavy-duty_incubator.log — is 0 bytes: it was killed before one line of output flushed. The two tick boundaries inside your window, 20:15 and 20:20, wrote nothing at all, and the next line in the file is 20:25:01Z duty run start, the post-reboot tick. So the runaway was the hygiene sweep on heavy-duty/incubator, ~6½ minutes in, and your 20:12:45–20:24:25 window is independently confirmed by the log's silence.

Worth saying plainly: tick.sh's evidence contract — "silence at a boundary means exactly one thing: cron itself is dead" — was exactly right, and it is why the outage window is readable at all after a forced stop.

2. One correction: there is no pipe

A head process appears alongside it, suggesting the job pipes Claude's output — a plausible route to unbounded memory growth if the upstream side produces a large stream.

run_session redirects the CLI to a file, not a pipe — common.sh:229:

( cd "$dir" && env -u DUTY_LOCKED -u NOTIFY_LOCKED -u DUTY_SNAPSHOT \
    timeout -k 60 "$tmo" "${BOT_CLI_CMD[@]}" "$prompt" ) </dev/null >"$slog" 2>&1 || rc=$?

Nothing in the chain buffers the CLI's output, and a pipe buffer is 64 KiB regardless of what either end does. The engine's only head uses are two head -1s (:487, :649) — and a session runs head freely as an ordinary tool call, which is the likelier sibling you saw. The 3.42 GB was the CLI's own RSS. That does not weaken recommendation 5; it relocates it — the useful half is the peak-RSS instrument, and "inspect what is piped through head" has no target.

3. Where each recommendation lands

#	Recommendation	Owner
1	cgroup ceiling + OOMPolicy=kill	crew for the ceiling; the crontab→systemd-unit half is your decision, below
2	oom_score_adj before launch	crew — one line in run_session, needs no decision
3	more guest RAM / zram	host + heavy-duty/box
4	earlyoom	host + heavy-duty/box
5	record peak RSS	crew — SESSION END gains the figure; the head half is void (§2)
6	host-side liveness watchdog	host + heavy-duty/box

crew provisions no boxes and has no surface for 3, 4 or 6 — they are real and they are not filed here.

4. The one question I am not answering for you

Recommendation 1's "better still" — a systemd timer + service replacing the crontab entry, with TimeoutStartSec= replacing timeout — is not a fix, it is a change to the install channel. shared/crontab.example is deliberately one line per box because "five hand-edited variants" drifted, and the engine's dependency floor is stated as bash+gh+jq+git+flock+timeout (common.sh:85) — systemd-run adds systemd to it. That is yours, and it is written into entry 20 as an open question rather than picked by me. oom_score_adj needs no ruling either way and rides regardless.

Placement is also yours: entries 21 (instrument) and 22 (evidence) are 0.1.3's charter in as many words; entry 20 is hardening and may belong to 0.1.9. The roadmap (#338) is where it moves if you want it moved. No needs-ruling is set: nothing on the board is stopped by either question, and release-init step 4 already puts the wave in front of you.

5. What the postmortem could not see, and it is crew's

A session killed with the box leaves a SESSION START with no end — a fourth shape the evidence contract does not name, because tick.sh writes its FAILED line from an rc that never existed. The consequence is not cosmetic: _session_terminal_record runs at common.sh:240, after the SESSION END line, so the #388 session breaker records neither a failure nor a reset for it. A prompt that OOMs the guest deterministically would repeat every tick forever and never trip the one mechanism built to stop that. Entry 22.

6. Not the day's only incident on this box

A ~10-hour credential-exhaustion window preceded it — 147 failed sessions, 03:04:25Z–13:03:44Z, all on the weekly limit — recorded on #207. Unrelated in cause, and worth holding beside this one: the board was clean and the tick cadence unbroken through both, so neither is visible from anything but the log. That is the same argument entries 21 and 22 make.

Leaving this thread open for §4. If you would rather the two questions were on the board as a ruling ask instead of here, say so and I will put them there in the canonical form.

[danmt]

I'd add a fix to this in 0.1.3 --- my take is simple, we have to prevent ways in which the crew stops working and breaks our invariants and constraints. this is a perfect case of something that can happen at any time, at the very least we should have a way to get back up: a force stop in the crew floor, a mechanism to start a box that had crashed, a restart option for boxes in the crew floor. also a way to monitor this or similar situations. having a way to get back on our feet @dan-claude-bot --- if this involves new issues, they should be added as comment in thorough detail in 0.1.3 release epic issue

[dan-claude-bot]

Recorded on #327 as asked — and two of the four already ship

Triage 2026-08-14 21:3xZ. Your ask is written out in full on the 0.1.3 epic: #327, the 21:3xZ comment — mechanism, decisions, the same-file cluster and must-fail cases per entry — and the three bullets are appended to that epic's to-mint list in the same tick, taking it from twenty-two to twenty-five. The comment is where the detail lives; the bullets are there because release-init step 1 counts bullets and does not read comments. Nothing is minted: the window is still at step 1, and this window's issues are minted together.

Your reply settles entry 20. "a fix to this in 0.1.3" — so containment (20), the peak-RSS instrument (21) and the orphaned-start repair (22) are all this window's, and the placement question I left open is closed. Correct me if that reads wrong.

Measured before writing: two of the four asks already exist

• Seeing it: shipped. The floor pings every box box exec -- true every 10s at a 5s timeout; three misses in a row publishes UNREACHABLE — N pings unanswered and overrides whatever the slower evidence poll last believed. Its own comment names your §1 lesson before you wrote it: "a listening port is answered by the guest kernel, so it stays green through a userspace that can no longer fork" — which is why the ping is an exec, and why this box's shape is exactly the shape it detects.
• Restart and start: shipped. restart, power-off, power-on per box; start-all, stop-all, wake-silent fleet-wide, each a confirmed button.

So the gap is not that the floor cannot act. It is that every stop the floor can fire is a graceful stop — box down is incus stop, no force, no timeout — and a guest that cannot schedule its own shutdown cannot answer one. Your recovery is the evidence: --force is what worked.

The three entries

1. The force stop. Its own per-box action, never a silent escalation from a graceful stop, plus force-then-start behind Restart when the box is unreachable.
2. Two levers that are unsound on exactly this state, and this one is a bug: restart fires box start whether or not the stop succeeded — so on a wedged box the button you would reach for spends 120s failing and then leaves the box untouched — and wake-silent sends its wake through box exec, the channel the wedge has taken out, so it can never wake the box it correctly marks offline. Neither is a regression; the detection got sharper and the actions did not follow it. The test suite cannot fail on either today, because the stub box models a down that always works — so the harness change leads the fix.
3. The alert. Detection is on the page and only on the page, and crew floor is a foreground server you start by hand — on the 14th duty.log wrote nothing at the 20:15 and 20:20 boundaries and the board stayed clean, so nothing told anyone. This one is the host's and not notify.sh's: that notifier is a singleton on the triage box, and a box has no path to a sibling, so it structurally cannot observe a wedged neighbour. It rides the floor process; what an always-on host process should be is 0.1.4's question, not this window's.

Two things you do not have to decide

The force path needs no ruling and no change in heavy-duty/box. box#11's own rule is "box owns a command when it must enforce an invariant incus cannot see … everything else is incus's job, and that door stays open." A force stop enforces nothing incus cannot see, so it is the escape hatch's — box incus <box> -- stop --force, which resolves the instance, warns only on isolation-touching verbs, and is non-interactive, so it is safe to fire from a server with no terminal. crew ships this alone.

The host-side half stays out of crew — more guest RAM or zram, earlyoom, and a watchdog that restarts unattended are still the host's and box's. Entry 3 is the alerting half only: it says the box is gone, it does not act.

Still open, still not blocking

The one question from my earlier §4 that this reply does not answer: whether the crontab entry becomes a systemd timer + service, with TimeoutStartSec= replacing timeout. That is an install-channel decision — systemd-run adds systemd to the engine's stated bash+gh+jq+git+flock+timeout floor — and entry 20 carries it as an open question rather than a picked option; oom_score_adj rides either way. No needs-ruling is set, because nothing on the board is stopped by it and release-init step 4 puts the whole wave in front of you regardless. Leaving this thread open for that.

[dan-claude-bot]

Closing as resolved — every ask on this thread is now a live issue on the 0.1.3 board

Triage 2026-08-25, board-sweep pass. This thread converged on 2026-08-14 and has been waiting on nothing since; release-init minted its asks on 2026-08-24. Verified against the board rather than against that comment:

your ask	issue	state today
containment — one session must not take the box down	#474	open, blocked (carries its own needs-ruling on the mechanism)
the peak-RSS instrument	#473	open, blocked
the orphaned SESSION START repair	#478	open, ready
reclaiming a box that accretes state	#457	open, ready
a force stop the floor can actually fire	#486	open, claimed — PR #525

Nothing is owed here and the record lives on the issues. Closing so the discussion board shows only threads that are actually waiting on somebody — reopen or open a fresh one if any of those specs reads wrong against the tree.