Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

exit-zero

Everything reported success. Nothing worked.

Field notes from running one always-on personal AI agent on a cheap cloud VM: 30 infrastructure failures that every layer reported as a success, and the fixes that made each one structurally impossible to repeat.

This is not about hallucination, context drift, or tool-call cascades. Those are well covered by people closer to the models than I am. This is about the plumbing around the model — cron jobs, locks, orphaned subprocesses, config drift, monitors watching the wrong thing. The failures that don't throw. The ones you find out about because a human noticed, four days late.


Why this exists

Search "AI agent failure modes" and you get a wall of writing about model behaviour, aimed at teams shipping multi-user products, usually attached to an observability vendor.

None of it described what actually kept breaking for me.

What broke was never the model. It was a drain that had been appended below an exit 0 and had therefore never run once. It was a monitor whose process pattern was loose enough to match the diagnostic command that was checking it. It was a security guard whose regex meant "line begins with a literal backslash" — so it inspected zero lines for six weeks while reporting that it had inspected everything.

Every one of those returned success. That is the whole problem, and it is why this repo is called what it is.

How to read this

Every entry has the same six sections, and one of them is ## Why it exited 0 — the specific reason nothing caught it. If you read only one section per file, read that one.

Entries are rated for generality: ★★★★★ means it will hit anyone running an autonomous system, ★★★ means you need a similar component to be exposed.

The failures

# Failure Detected by Days
01 Success-shaped failure — every layer exits 0 on emptiness ★★★★★ human 4
02 Timeout-kill recorded as success ★★★★★ audit 4
03 The filter that matched nothing ★★★★★ audit same day
04 Stale prompt drift — recurred three times ★★★★★ audit weeks
05 An LLM in a deterministic path ★★★★★ human 1
06 Reasoning-model leakage — output caps don't bind ★★★★★ human 8
07 The agent restarts its own runtime ★★★★★ human 1 hr down
08 Instrumentation that cannot fire ★★★★★ audit weeks
09 The untested guard — a security control that did nothing ★★★★ audit 6 weeks
10 Dead code after exit 0 ★★★★ audit 1
11 Two-writer clobbering ★★★★ human ongoing
12 Split-brain writes — the merge that un-merges itself ★★★★ monitor 1
13 Listed-but-dead dependency ★★★★ audit 12
14 Orphaned subprocesses fighting a lock ★★★★ human same day
15 A monitor that matches itself ★★★★ audit weeks
16 Append-only record rewritten ★★★★ audit 13
17 A queue that re-proposes what was already judged ★★★★ human weeks
18 Missed schedule invisible to its own monitor ★★★★ audit 10
19 Partial config re-pin ★★★★ monitor 2
20 Expiry-dated claims with no owner ★★★★ audit weeks
21 Alert false-positives cause the outage ★★★★ human weeks
22 An idempotency bug visible only on re-run ★★★★ self minutes
23 The "cache" you pruned was live state ★★★★ self minutes
24 Version tracking that was never deterministic ★★★★ human 5 releases
25 A capability that exists but never loads ★★★★ human weeks
26 Quota exhaustion misread as transient ★★★★ audit days
27 The double-applied fix ★★★★ audit hours
28 Sleep-skipping scheduler ★★★ audit 2
29 Transport corruption, silent no-op ★★★ self minutes
30 Traceback camouflaged by || true ★★★ audit weeks

Read down the Detected by column. That is the finding underneath all the other findings.

The four rules

Everything here collapses into four things. Each one recurs across at least five of the thirty.

1. Behavioural fixes recur. Chokepoint fixes don't. If a fix depends on anyone — a person or a model — remembering to do something, it is not a fix. It is a wish. Every fix that stuck moved the failure from discouraged to impossible: a hook that blocks the call, a filter at the send path, a flag the caller physically cannot pass. The test of a fix is not "will this work?" but "can this still happen?"

2. An exit code is not a result. In an agent pipeline, "nothing to do" and "nothing happened" return the same value. Verify the artifact, the log line, or a cross-layer invariant — did meetings occur but no notes exist? — never the exit status of the thing you asked to do the work.

3. Prove your monitor can go red before you trust it green. A guard with no test is indistinguishable from no guard, and it is worse than no guard, because it buys confidence. Every check here was validated by re-injecting the original bug, confirming detection, restoring, and confirming silence. Two of them were found to be structurally incapable of ever reporting a problem (#08, #09).

4. Take the LLM out of deterministic paths. Keep it for judgment; remove it from anything with a correct answer. A scheduled job wrapped around a deterministic operation gives a model latitude, and latitude is derail surface. The pattern that worked: script collects → model judges → human decides.

The setup

One always-on personal agent, on a free-tier ARM cloud VM, running roughly forty scheduled jobs that handle mail, calendar, meeting capture, messaging and research. Single operator. No team, no on-call, no staging environment. Built on the Hermes agent framework, though almost nothing here is specific to it — the failures are in cron, systemd, SQLite, OAuth, shell, and the seams between them.

Roughly five weeks of continuous operation and hardening produced these thirty. I expect there are more I still cannot see, which is rather the point.

What this is not

  • Not a product, a framework, or a tool. There is nothing to install.
  • Not about model behaviour. No prompt engineering, no evals, no RAG.
  • Not a claim that my architecture is good. Several entries exist because it wasn't.

Contributing

Failures you actually hit are very welcome — see CONTRIBUTING.md. The bar is simple: it reported success, and it was wrong. Anonymise people and companies; keep the mechanism and the numbers.

Licence

Prose under CC BY 4.0, code snippets under MIT.

About

Field notes from running an always-on personal AI agent: 30 infrastructure failures that all exited 0, and the deterministic fixes.

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages