docs(worker): the measured lock-wait split, and the aggregate that misled worker v6.2.1 made the split observable. Measured on the system pool: incomplete is 98.7% lock wait (376.6us) with 4.9us of actual work, while incremental and cold_rebuild show 1.3us of wait. So the mutex is not generally contended — only when the outcome is incomplete, because the drain holds it while applying the tip the build waits for. That cross-outcome contrast is the control. Corrects this page's own 49.1ms figure: it was a 2d19h aggregate and does not reproduce per-build (real value ~377us, 119x smaller). The aggregate was dominated by a startup burst with a larger WAL backlog than a normal roll faces. "Read the rate, not the total" applied to this page itself. Also records that the cold-start burst is now confirmed empirically — 4107 incompletes in 45 minutes on a fresh pod vs 2297 over 2d19h on the old one — and that a worker roll therefore costs a brief dispatch blip (79s, 36s) before settling to 10s, better than the 12-16s pre-roll baseline. Expect it; not a regression. Adds the formulas for deriving mean wait and mean work from the series. Refs noetl/ai-meta#369, noetl/worker#344
docs(worker): correct a stale-gauge claim and record why the user pool is zero Two corrections. 1. This page cited "index_events 156 resident" as a live measurement. It was not: index_events was a stale gauge, refreshed at one call site and not the other, so it kept its old value after terminal eviction. Prod read indexed_executions 0 / index_bytes 0 / index_events 1, which is what exposed it. Fixed in noetl/worker#345. index_bytes was correct all along — the opposite of the hypothesis in ai-meta#368. 2. Records why all-zero state-builder metrics on noetl-worker-rust are CORRECT and not a misconfiguration: it does not set NOETL_STATE_BUILDER, so builder_mode() is Off, and WalDrain::from_env returns Ok(None) for Off — the WAL drain never starts, so there is nothing to index. Off means the server-built state is used. Written down so the next person does not chase it, which is what happened here. Refs noetl/ai-meta#368
docs(worker): the `incomplete` total is a cold-start burst, not steady state Measured on live prod 2026-10-01: 18 executions drove 87 state builds with ZERO incomplete, zero retry waits and zero fallbacks. Counters were also frozen across an idle window, and incomplete took +0 across ~40 minutes of mixed activity while incremental took +16. So the 2297 incomplete builds and their 112.83s are historical — most likely a pod-start burst with a cold WAL index, where every drive finds the tip unindexed while the drain backfills and the drain holds the mutex the build is waiting on. That also explains the 49ms. Records the consequence: read the RATE, not the cumulative total. Reading incomplete's share of cumulative time as an ongoing inefficiency is what sent an optimisation effort at a problem that is not occurring. Also records that build_duration_seconds includes lock acquisition, that worker#344 splits it, and that its ~45ms-wait prediction is untested because the split is not deployed. Refs noetl/ai-meta#369, noetl/ai-meta#367
docs(worker): where the off-server drive-build cost actually is Measured on prod 2026-10-01 (live system pool, 2d19h). Records two things that overturn reasonable assumptions: - cold_rebuild is the CHEAPEST outcome (0.23ms, 11 hops), not the expensive one its name suggests; advance_to already short-circuits an unchanged chain via head_event_id, an O(1) detector that exists today. - 87% of state-builder time is the "incomplete" outcome — the WAL index lagging the server's expected_head, ~49ms per occurrence. A waiting problem, not a folding problem, so fold optimisations leave it alone. Also records that NOETL_STATE_BUILDER=offserver runs on the SYSTEM pool, not noetl-worker-rust (whose state builder has never run), and the noetl_worker_ metric prefix needed to find any of this. This is the evidence behind not building a worker-side chain-certificate consumer — see noetl/ai-meta#367. Refs noetl/ai-meta#366, noetl/ai-meta#367