Serialize deploys with a staging lock#13
Merged
Conversation
Now that a deploy can be triggered from the command, the webhook, and soon a deploy API, more than one can run at once. r10k rewrites the staging directory in place, so two overlapping deploys corrupt each other's trees — and unlike a mid-deploy read on the serve side, there is no verify-by-reseal backstop for a tree r10k itself is halfway through writing. deploy.Run now takes an exclusive flock on <state>/deploy.lock across r10k, sealing, and the --wait, so every deploy path on a host serializes through one lock rather than only within a single daemon. The lock polls with LOCK_NB and a timeout instead of blocking in the syscall, so it gives up on a wedged deploy rather than hanging; flock is released on process exit, so a crash does not leave it stuck. The test runs two deploys against one staging directory concurrently, with a fake r10k that fails if it ever sees a second copy running — proving they serialize. This is the first half of the deploy-API split: the API adds a third deploy front door, and this makes concurrent front doors safe before that lands. Signed-off-by: Michael Harp <mike@mikeharp.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First half of the deploy-API split. Before adding a third deploy front door (the API), make concurrent deploys safe.
Why
A deploy can now come from the command or the webhook, and soon the API. r10k rewrites the staging directory in place, so two overlapping deploys corrupt each other's trees. Unlike a mid-deploy read on the serve side — where the agent's verify-by-reseal rejects a bad artifact — there is no backstop for a tree r10k is halfway through writing. It must be prevented.
What
deploy.Runtakes an exclusiveflockon<state>/deploy.lockacross r10k, sealing, and--wait. Every deploy path on a host — CLI, webhook, future API — serializes through one lock, not just within a single daemon's worker.LOCK_NB+ a timeout (DefaultLockTimeout10m, r10k is slow) rather than blocking in the syscall, so it gives up on a wedged deploy instead of hanging.flockis advisory and released on process exit, so a crashed deploy doesn't leave the lock stuck.Test
TestConcurrentDeploysSerializeruns two deploys against one staging directory concurrently, with a fake r10k that fails if it ever sees a second copy running. It passes only because they serialize (runtime ≈ 2× the fake's work, not 1×).Verified:
go test -race ./...,golangci-lint0 issues, gofmt/vet clean, markdownlint clean.Next PR: the deploy API (
POST /v1/deploys+ status), unifying the webhook and API into onedeploy-serverdaemon that shares this now-safe deploy path.🤖 Generated with Claude Code