Session Safety Net: systemd reaper for orphaned Claude Code sessions

[ganexxa-quantum]

The Problem

PAI Cloud VMs running multiple PAI instances suffer from chronic memory exhaustion caused by Claude Code sessions accumulating without cleanup. On our 4GB RAM + 2GB swap VM with 3 PAI users, we hit 99.96% swap usage from 7+ orphaned sessions consuming 2.2GB.

How sessions become orphaned

1. User works in code-server terminal with claude
2. User closes browser tab instead of running /exit or closing the terminal tab (X)
3. code-server's ptyHost keeps the terminal alive (enablePersistentSessions: true)
4. Claude session persists in background at 140-250MB RSS — and grows due to memory leaks
5. Repeat across days/users → sessions accumulate → swap exhaustion → VM crawls

Why it's not the user's fault

Neither Claude Code nor PAI provides any built-in protection:

Feature	Exists?
CLI idle timeout	No
Max concurrent sessions	No
Auto-exit after inactivity	No
Memory/RAM limits	No
Process management (claude ps)	No (proposed in claude-code#11122, never built)

Claude Code also has confirmed memory leaks in idle sessions — a single idle session can grow to multi-GB over hours:

• #18859 — 4 idle sessions grew to ~15GB EACH over 18h
• #22365 — Session JSONL files grow unbounded (3.8GB)
• #16508 — VS Code: closed tabs don't release processes (~400MB/session)

PAI upstream has related issues too:

• [PAI-UI][Server] Session manager for tracking active sessions #510 / [PAI-UI][Server] Claude runner subprocess management #509 — SessionManager + ClaudeSession specs (closed with PAI-UI batch)
• SecurityValidator.hook.ts hangs when stdin doesn't close properly #452 — SecurityValidator hook hangs indefinitely, no process cleanup (open)
• fix: Prevent Inference.ts from creating noise session files #635 — Inference.ts creates ghost session files (open)

Our Solution: PAI Session Safety Net

A pure-bash systemd timer that runs every 30 minutes. No bun/node dependency — works even under severe memory pressure.

Architecture

systemd timer (every 30min)
  → pai-session-reaper.sh
    ├── Phase 1: reap_sessions(ttl)    — kill Claude sessions older than 8h
    ├── Phase 2: check_memory()        — swap watchdog, emergency reap at 95%
    └── Phase 3: report_stale_bun()    — log-only warning for old bun procs

Files

File	Purpose
/etc/pai-session-reaper.conf	Configuration (TTL, thresholds, Slack port)
/usr/local/bin/pai-session-reaper.sh	Main script
/etc/systemd/system/pai-session-reaper.service	Oneshot service
/etc/systemd/system/pai-session-reaper.timer	30-minute timer

Key Design Decisions

Process identification: ps -eo user,pid,etimes,rss,comm --no-headers | awk '$5 == "claude"' — matches the executable name cleanly, no false positives from bun/node/hooks.

Self-protection: When run manually from inside a Claude session, the script walks ancestor PIDs from $$ up to PID 1 and skips any Claude PID in its ancestry. When run from systemd timer (PPID=1), no Claude PID matches — all eligible sessions are reaped.

Graceful kill: SIGTERM first (30s grace period so PAI hooks can fire for session capture), then SIGKILL if still alive.

Memory watchdog:

• Swap ≥ 80%: Slack warning with top 3 memory consumers
• Swap ≥ 95%: Emergency reap with reduced TTL (1h instead of 8h)
• Reads from /proc/meminfo directly — no forks needed under memory pressure

Stale bun reporter: Finds bun processes older than 48h that aren't systemd services. Log-only, never auto-kills (too risky — could be Slack bots or schedulers).

Dry-run mode: DRY_RUN=true in config logs everything without killing. Great for testing.

Configuration

SESSION_TTL_SECONDS=28800         # 8 hours — normal reap threshold
EMERGENCY_TTL_SECONDS=3600        # 1 hour — used when swap > 95%
SIGTERM_WAIT_SECONDS=30           # Grace period before SIGKILL
SWAP_WARN_THRESHOLD=80            # Percent — triggers warning
SWAP_CRITICAL_THRESHOLD=95        # Percent — triggers emergency reap
DRY_RUN=false                     # true = log without killing

Results

First run freed ~470MB and dropped swap from 68% to 57%. The timer has been running reliably since deployment.

Gotcha: bash arithmetic under set -e

((var++)) returns exit code 1 when the pre-increment value is 0 (bash treats 0 as falsy). Under set -e, this kills the script. Use var=$((var + 1)) instead. Same issue with [[ condition ]] && action — the short-circuit returns exit code 1 when the condition is false. Use if/then blocks instead.

What's Not Built Yet

• Session count guard — prevent new claude sessions when a limit is reached (would need a shell wrapper or hook)
• Activity detection — check recent CPU time before killing, to avoid reaping active sessions that happen to be old

For Other PAI Users

If you're running PAI on a small VM with multiple users, you're probably hitting this too. The community workaround is ps | grep claude | kill, but a systemd timer makes it automatic and adds the memory watchdog layer.

Happy to share the full script if anyone wants to adapt it.

[danielmiessler]

Hey @ganexxa-quantum, thanks for raising this, and sorry it sat for a while.

We're changing how LifeOS ships. Instead of cloning a full ~/.claude directory and running it as a complete system, LifeOS is becoming a skill you install through an agentic installer. The installer hands integration to your own AI, which reads your actual machine (your OS, your paths, your harness) and wires the hooks and system prompt in where they belong.

That's aimed right at what you hit here. The old "one directory, one layout, hope it matches your setup" approach is exactly what broke for so many people, and the new model should handle it far better because your AI does the integration per machine instead of us guessing.

So we're closing this in prep for that release. If it still bites you once the skill-based version is out, reopen or file a fresh one and we'll jump on it. Appreciate you taking the time.