Skip to content

v0.2.3 - scoreboard consistency by construction

Choose a tag to compare

@swayyaam swayyaam released this 23 Aug 21:09
· 79 commits to main since this release

Closes #6: the served scoreboard could intermittently disagree with the solve log — a rare
rest=0 fromscratch=500 miss the soak reproduced at ~1 in 8 two-minute runs. It was a
durability gap, not a race: the board's consistency rested on a best-effort per-solve
recompute with no guaranteed repair (no outbox, no retry, no periodic recompute in prod), so
a missed/slow/preempted recompute left the board stale until an unrelated tick.

Fixed by read-repair: a served snapshot records the valid-solve count it was computed from,
and a read recomputes before returning if the log has moved past it. The served-equals-log
invariant is now structural, not timing-dependent. Proven by a negative control
(-break-readrepair): with read-repair removed the mismatch reappears at ~the pre-fix rate
(3/5 under -race); with it on the invariant cannot fail by construction (0/30, 0/5).

Two supporting fixes found while triaging #6:

  • scoreboard recompute no longer holds the mutex across its DB reads + Redis write (the
    third "lock held across I/O" latency bug; see AGENTS.md), guarded on a data-derived count.
  • the submit/admin recompute no longer runs on the request context, so a client disconnect
    can't abandon the board update for an already-committed solve.

Also in this release: CI Go module/build cache fixed (keyed on api/go.sum, not the missing
repo-root path) and all Actions moved off the deprecated Node 20 runtime.

Operator action on upgrade: none. One new read-only query; no OpenAPI or database-schema
change. New metrics: osctf_scoreboard_stale_reads_total (read-repairs) and
osctf_scoreboard_stale_served_total (bounded fallback — alert on it).