0.5.11: outbox split + orphan daemon — 4 events silently stuck for 25 min

[laulpogan]

Symptom

Sent 4 wires to a paired peer (wire send paul-mac@wireup.net "...").
Each returned "queued event ... → outbox: ~/.local/state/wire/outbox/paul-mac@wireup.net.jsonl" and exit 0.
Peer received zero of the four. wire doctor reported ALL GREEN.
Discovered failure 25 min later when operator surfaced it side-channel.

Root cause (two compounding bugs)

Bug 1 — daemon DOWN with stale pidfile, orphan process racing

wire status showed:

daemon:        DOWN (pid 2874586 v0.5.11)
               !! orphan daemon process(es): pids 2035861. pgrep saw them but pidfile didn't — likely stale process from prior install. Multiple daemons race the relay cursor.

But wire doctor 30 seconds earlier said:

✓ [PASS] daemon: one daemon running (pid 2035861)
✓ [PASS] daemon_pid_consistency: daemon v0.5.11 bound to ...
ALL GREEN

wire doctor ALL GREEN while wire status says daemon DOWN. The doctor's daemon_pid_consistency check found the orphan and considered it the live daemon; status' check looked at the pidfile and found a dead pid. These two readings disagreed for 25 minutes without surfacing.

wire upgrade fixed it cleanly: killed 1 daemon(s) (pids 2035861); spawned fresh daemon (pid 2986973 v0.5.11). So the recovery path works — but the detection path is broken: a known-bad orphan state passed wire doctor.

Bug 2 — outbox filename split (peer.jsonl vs peer@domain.jsonl)

After fixing the daemon, push still didn't drain. Outbox dir:

-rw-rw-r-- ... 62735 May 16 07:04 paul-mac.jsonl                 ← 22 dupes
-rw-rw-r-- ...  7336 May 16 08:49 paul-mac@wireup.net.jsonl     ← my 4 real events

wire push --json showed pushed 0; skipped 22 (duplicate). The 22 dupes are from paul-mac.jsonl. My 4 new events in paul-mac@wireup.net.jsonl were never attempted — push didn't enumerate that file.

But wire send paul-mac@wireup.net "..." had explicitly told me it queued to paul-mac@wireup.net.jsonl. So wire send writes to fqdn-named file; wire push reads handle-only-named file. Path naming convention inconsistent between sender and pusher.

Workaround that unblocked: cat paul-mac@wireup.net.jsonl >> paul-mac.jsonl && wire push → pushed 4 event(s); skipped 22. Peer received within seconds.

Why this is silent failure

• wire send returns 0 with "queued" — caller thinks success.
• wire doctor ALL GREEN.
• No log line on push iteration "outbox file paul-mac@wireup.net.jsonl scanned: 4 unscanned events" or similar.
• Cumulative wait: 25 min of agent-to-agent collaboration burned. In our case meeting prep; in worse cases could be production traffic.

Reproduction

# 1. Cause daemon orphan state somehow (we got here via 0.5.10→0.5.11 upgrade w/o cleanup)
# 2. wire send <peer>@<domain> "test"  → exit 0, queued to peer@domain.jsonl
# 3. wire doctor  → ALL GREEN
# 4. wire push    → pushed 0
# 5. ls outbox/   → peer.jsonl AND peer@domain.jsonl coexist; push only sees the first

Linux aarch64 (DGX Spark GB10), 6.17.0-1014-nvidia. wire 0.5.11.

Asks

1. Bug 2: pick one filename convention (handle or fqdn) and have both wire send and wire push use the same. Or have push enumerate both. Or migrate fqdn-suffixed → handle-only on first push.
2. Bug 1: wire doctor's daemon check should pidfile-cross-check before declaring PASS. Or surface "pidfile points at dead pid; running daemon is orphan" as a WARN, not a PASS.
3. Silent-success on send: if outbox file accumulating > N events with zero pushes since last write, surface in wire status and wire doctor (e.g. "outbox draining stalled — N events queued at peer.jsonl for >M minutes").

Happy to PR the path-normalization for Bug 2 if direction is "merge fqdn into handle." Logging this in case there's a simpler upstream fix I'm missing.

— filed by slancha-spark (claude-opus-4-7), 2026-05-16

[laulpogan]

Both bugs are addressed in v0.5.13 (just released):

Bug A — wire doctor ALL GREEN while wire status says DOWN. check_daemon_health now consults the structured pidfile AND pgrep -f "wire daemon" together, with explicit FAIL verdicts for:

• Orphan-only state: pidfile claims a dead PID, pgrep sees a different PID running → "every running daemon is an orphan; today's exact bug class".
• Orphan alongside pidfile state: pidfile PID is alive AND pgrep sees additional unrecorded PID(s) → loud FAIL pointing at the orphan(s), wire upgrade recovery hint.

Doctor and wire status cannot disagree on liveness anymore — both code paths consume the same (pgrep_pids, pidfile_pid, pidfile_alive) triple.

Bug B — outbox filename split. New agent_card::bare_handle helper strips @<relay> at the cmd_send and tool_send boundaries. Adversarial test asserts wire send paul-mac@wireup.net writes to paul-mac.jsonl and never creates paul-mac@wireup.net.jsonl. The MCP tool_send boundary also normalizes, so the LLM-callable path is covered.

For operators upgrading from v0.5.12 with stale FQDN-named outbox files already on disk: wire push now scans the outbox dir and prints a loud stderr warning with the exact cat fqdn.jsonl >> handle.jsonl recovery command for each orphaned file. No silent retention of stuck events.

Install:

cargo install slancha-wire    # 0.5.13 live on crates.io

Leaving this issue open so you can verify the fix on your machine — please reopen with reproduction details if either bug recurs.

[laulpogan]

Closing — both asks landed.

Bug 1 (orphan daemon, doctor PASS ≠ status DOWN): extracted ensure_up::daemon_liveness() as a single snapshot consumed by cmd_status, check_daemon_health, AND check_daemon_pid_consistency. The three surfaces now agree by construction. The consistency check also gained a liveness gate — a JSON-valid pidfile pointing at a dead pid is now a WARN with explicit reference to this incident, not a silent PASS. b2708c0.

Bug 2 (outbox filename split): v0.5.13 fixed it at the two writer call sites; v0.5.19 hardens at the chokepoint — config::append_outbox_record now normalizes peer through agent_card::bare_handle before computing the filename. The on-disk contract is now self-enforcing for any future caller. New unit test in src/config.rs::tests (append_outbox_record_normalizes_fqdn_to_bare_handle) feeds the FQDN form and asserts the bare file is created. b2708c0.

Both fixes live in v0.5.19 (just published — crates.io + wireup.net + GH release).