fix(overseer): stop replaced tick workers - #4981
Merged
Merged
Conversation
Observed Problem #3: the acting-overseer tick loop was missing scheduled ticks. Cadence is every 15 min, but status showed the last tick at 2026-07-28T19:23:39Z marked (stale), with large gaps (17:57:55Z -> 19:23:39Z ~86 min). Root cause: the acting-overseer tick uses a bare AtomicBool overlap guard (overseer_tick_running) cleared by a spawned thread's ClearOnDrop. If a tick HANGS on a long gh/network call, ClearOnDrop never runs, so the guard stays true forever and every subsequent scheduled tick is dropped -> staleness and multi-tick gaps. There was no watchdog to reclaim the stuck guard. Fix (additive / non-breaking): - Add a pure, clock-injected TickWatchdog + guard_generation_matches helper next to OverseerCadence in src/overseer/wiring.rs. The watchdog bounds max in-flight time (three cadence intervals, floored at 10 min) and is fully unit-tested (budget floor, reclaim on overrun, monotonic-safe inflight, generation-guard race, end-to-end hung-tick model). - Wire it into the daemon: record arm time + generation when a tick arms; before the cadence check, if a tick has been in flight past the bound, reclaim the guard, bump the generation, and emit a tracing::warn! plus a daemon_log staleness/liveness signal so a catch-up tick can fire. - Generation-token guard on ClearOnDrop: a late-finishing hung tick only clears the guard if its captured generation still matches current, avoiding the stale-clear race against a catch-up tick. Structured tracing + OTel only (no print!/println!). Preserves PRD and OODA cadence semantics. Adds 6 unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use one generation value to stop later actions and block outdated memory, daemon-log, and activity writes after watchdog replacement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rysweet
changed the title
📊 Coverage Summary
Coverage data from CI run. Test files matching |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prevent one stuck Overseer worker from blocking every later scheduled run, and prevent that old worker from changing shared state after it has been replaced.
Limits
This is cooperative stopping. It cannot interrupt a system call or external command that is already running. Time limits for individual external calls are separate work.
Validation
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com