Skip to content

v3.13.0 — Observability

Latest

Choose a tag to compare

@dl4rce dl4rce released this 19 Aug 11:25
· 1 commit to main since this release

Closes the 2026-08-19 incident entirely. The last and largest gap: Flaiwheel could not tell whether its knowledge repository was still connected to its remote.

Everything shipped in 3.12.x reports on pushes that were attempted. The failure that hid 325 documents inside a Docker volume for 2.5 months attempted nothing at all — the clone had drifted from its remote, so there was never anything to commit, so no push could fail, so no metric could go red. "Nothing to push (already in sync)" and genuinely being in sync were rendered identically.

Added

  • Divergence detection — GitWatcher.check_divergence(). Compares HEAD against @{u} with git rev-list --left-right --count and classifies the result as synced, ahead, behind, diverged, no-upstream, or unknown. The states are deliberately distinct rather than a boolean: behind is the normal state between two pulls and must not raise an alarm, whereas ahead, diverged and no-upstream each mean knowledge is not being backed up — for different reasons, with different fixes.
  • Divergence is checked on the noop path. This is the entire point. A repository with nothing to commit is precisely where a disconnected clone hides, so that path — the one that used to return an unconditional "already in sync" — now establishes whether that claim is true. Also checked after every successful push (confirming the commit landed), after a rejected push (a rejection is the classic symptom of divergence — now named rather than left as a raw git error), and on every pull, including a failed --ff-only pull, which refuses in exactly the diverged case.
  • HealthTracker records divergence and escalates on it. New divergence_status, commits_ahead, commits_behind, last_divergence_at. is_healthy returns False for diverged, ahead, no-upstream. A repository that indexes perfectly and pushes nothing can no longer report itself healthy. unknown is never persisted — a transient fetch failure must not erase a real diverged verdict.
  • /health exposes the divergence fields on both the per-project and aggregate endpoints.
  • Agents are told directly. Every write_* result appends an explicit warning when the repo has diverged, is ahead, or tracks no upstream — including on noop. A warning that lives only in an endpoint nobody polls does not exist; the agent writing the document is the one that needs to know it did not leave the machine.

Notes

  • 19 new tests (335 total) against real temp repositories: a force-pushed rewritten upstream (the real-world trigger — a secret purge or a squash silently desynchronises every clone), a repo with no upstream, behind treated as healthy, unknown not clearing a prior verdict.

  • The incident is now fully remediated. Items A, B, C, D and E shipped across v3.12.0 → v3.13.0. Item F — "watcher path scoping" — was retracted as a misdiagnosis, verified against the running container: /docs is not a git repo, every project owns its .git, so _find_git_dir() cannot escape upward past its own project, and no commit in any repo contains another project's path.

    The evidence that produced item F was the log line knowledge: update flaiwheel/telemetry.json, read as one project's file appearing in all 11 repos. The real file is .flaiwheel/telemetry.jsonwith a leading dot. That missing dot is the exact signature of the porcelain off-by-one fixed in v3.12.2, which truncated the first character of the first worktree-modified filename. Every project owns an identically-named telemetry file rewritten on each sync tick, so all 11 watchers emitted the same mangled string simultaneously — uniformity caused by shared code, misread as shared state. A log line is evidence of a symptom, not of a cause.

  • Automatic divergence repair is deliberately not implemented. The June 2026 remediation is the argument: 6 of the differing files were security redactions, and a naive reset would have re-published the plaintext secrets the rewrite existed to remove. Repair requires a human decision about which side of each file is authoritative. Flaiwheel reports; it does not guess.