fix(boatstack): guard waits for hydration lock release, not helper presence - #189
Merged
Conversation
…esence Concurrent first use of an empty shared-runtime slot could falsely deny a tool call. One guard wins the mkdir lock and hydrates; the losers waited by watching for the helper file to appear. The installer copies the helper before the manifest, so a waiter could observe that gap, judge the runtime "unsafe or incomplete", and exit 2 — the exact flake seen on the public sync PR's ubuntu job (TestGuardAutoHydrationSerializesConcurrentFirstUse). The waiter now waits for the hydrating peer to release the clone-wide lock. The peer removes the lock only after its installer returns, so a released lock means the slot is complete; the authoritative checksum and safety gates then accept it or fail closed. Fixed in both the bash and PowerShell guards. Bounded conformance tests: strengthen the concurrency test to 8-way contention with a start barrier (exactly-one-hydration invariant), and add a deterministic waiter-vs-slow-non-atomic-peer test that reproduces the exact failure mode against the old logic and passes on the fix, on any timing. Disclosure-Reviewed: reviewed — public-safe only (public guard script + tests + release-note prose), no private facet in this commit
bigboateng
marked this pull request as ready for review
July 25, 2026 16:09
…rs wait The guard judged slot readiness by the helper alone. An installer copies the helper before the manifest, so a guard arriving in that window skipped the hydrate/wait block and denied at the manifest gate while a peer held the lock and was about to finish. This surfaced as a flaky Linux CI denial under contention. The entry test now mirrors the downstream gates (helper AND manifest present, non-symlink) in both the bash and PowerShell guards, so a guard in the gap joins the hydrate lock and waits instead of denying a half-written slot. Adds a bounded, deterministic regression test driving a slow non-atomic peer from a half-written slot. Disclosure-Reviewed: reviewed — public-safe only, private facet kept out of this commit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The public sync PR operatorstack/boatstack#109 failed on the ubuntu
testjob with:It passed on macOS and all 6 Windows shards — a timing-dependent race, not related to the Simplified Technical English work.
Root cause
On concurrent first use of an empty shared-runtime slot, one guard wins the
mkdirlock and hydrates; the others wait. The waiting guard watched for the helper file to appear ([[ -x $HELPER ]]). The installer copies the helper before the manifest, so a waiter could observe that gap — helper present, manifest not yet — judge the runtime "unsafe or incomplete", and exit 2.Fix
The waiter now waits for the hydrating peer to release the clone-wide lock. The peer removes the lock only after its installer returns, so a released lock means the slot is complete. The guard then runs the same checksum/safety gates, which still fail closed if hydration was disabled, timed out, or failed. Applied to both the bash and PowerShell guards ().
Bounded conformance tests (remove the failure mode)
TestGuardAutoHydrationSerializesConcurrentFirstUse: 8-way contention with a start barrier; asserts exactly one hydration and every guard proceeds.TestGuardAutoHydrationWaiterAwaitsPeerCompletion: a deterministic peer hydrates non-atomically (helper, pause, manifest, release). The waiter must not judge the slot until the lock is released. Verified: this test fails against the old logic with the exact CI error, and passes on the fix — on any timing.Verification (all green locally)
go test ./...(product-engineering-loop) → all packages ok, 0 failures; hydration tests pass-count=3.release_notes.py validate+ append-onlypreflight→ PASS.Ships to the public repo via the normal projection, which is what the failing sync PR exercises.