[HIGH] WAL writer thread is unsupervised single point of failure

[austin-barrington]

Summary

The WAL group-commit writer runs on a single OS thread with no panic recovery, no supervision, and no respawn. Any panic in the writer loop permanently breaks all WAL operations — the only recovery is a full process restart.

Finding: #4

Root Cause

In adapters/wal/batching_wal.rs:

• Line 30: JoinHandle stored as _writer — never joined, never inspected.
• Lines 61-128: writer_loop runs a bare loop { ... } with no catch_unwind.
• Lines 49-52: Thread spawned once, never respawned.

let writer = thread::Builder::new()
    .name("hyperbytedb-wal-writer".into())
    .spawn(move || Self::writer_loop(sync_rx, wal, max_batch))
    .unwrap_or_else(|e| panic!("spawn hyperbytedb-wal-writer thread: {e}"));

Failure Chain

1. Writer thread panics (e.g., RocksDB error, bincode encoding issue).
2. Stack unwinds → rx (channel receiver) is dropped.
3. Channel disconnected → all subsequent try_send returns TrySendError::Disconnected.
4. Every enqueue call returns Err("WAL batcher channel closed").
5. Permanent, process-wide ingest outage. No health signal, no auto-recovery.

Impact

• Availability: Complete ingest failure until process restart.
• Silent: No health endpoint reflects writer death (the _writer handle is never checked).
• Unrecoverable: Cannot restart the writer without reconstructing the entire BatchingWal.

Recommended Fix

1. Wrap per-batch write in catch_unwind — fail only that batch's waiters, not the whole thread.
2. Supervise/respawn the writer thread (or use tokio::task::spawn_blocking with error handling).
3. Surface writer death via the /health endpoint.
4. Consider a watch channel or atomic flag for writer liveness.

Confidence

High on the failure mechanism. Trigger likelihood is low today (small panic surface on the bincode path), but the consequence is severe and unrecoverable.

[austin-barrington]

Fixed: supervisor task spawned alongside writer (batching_wal.rs:51-56), writer_alive AtomicBool health flag, pre-enqueue health gate in enqueue(). Logs error and blocks writes gracefully on failure.