Serve immutable snapshots and reseal on SIGHUP (PE shape)#10
Merged
Conversation
The publisher only sealed at startup, so an r10k deploy went unseen until a restart, and it streamed artifacts from the live staging directory — so a fetch overlapping an r10k deploy could serve a half-written tree whose bytes no longer matched the advertised code_id. The agent's verify-by-reseal caught it, but by rejection every poll rather than convergence. PE's Code Manager solves both with one idea: after r10k finishes it commits the staging tree into an immutable git snapshot and serves that, never the live working dir; the commit is an explicit post-deploy trigger, not a filesystem watch. This adopts the same shape, adapted to codavox not owning the deploy. Snapshot: Reseal now materializes each current version as a deterministic .tar.gz under <state>/artifacts, written to a temp file and renamed into place, and serving streams that file. Bytes and code_id can no longer drift, because the bytes are a snapshot from when the tree was quiescent rather than whatever staging holds at request time. Superseded artifacts are reaped on the next reseal; an in-flight download holds an open descriptor, so unlinking the file it streams is safe on Unix. Only the current version is kept on disk — the same 'compilers retain old versions themselves' policy as before. Trigger: SIGHUP reseals the running publisher, wired to r10k's postrun hook (systemctl reload). The signal fires only after r10k returns, so the tree is quiescent and no reseal observes a half-written deploy. A watch mode would have to reconstruct that 'deploy finished' signal postrun already gives us, and codavox does not own the deploy, so the explicit trigger is both simpler and the correct coupling. SIGTERM/interrupt now drain in-flight downloads via graceful shutdown (Server.Serve(ctx) replaces the blocking ListenAndServeTLS). NewStore takes an artifact directory; the publish command derives it from --state. Tests that run the real publisher pass --state a temp dir, since the publisher now needs a writable data directory. Tests: artifact is the seal-time snapshot even after staging is mutated without a reseal; superseded artifacts are reaped; SIGHUP makes a running publisher advertise new code and a compiler converge without a restart. Signed-off-by: Michael Harp <mike@mikeharp.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adopts Puppet Enterprise's Code Manager shape for the two coupled problems in the publisher: nothing resealed after startup, and artifacts were streamed from live staging, so a fetch overlapping an r10k deploy could serve a half-written tree whose bytes no longer matched the advertised
code_id.Reading PE's own implementation (the
.cljsource ships inpuppet-server-release.jar) showed both are one idea: after r10k finishes, Code Manager commits the staging tree into an immutable git snapshot and serves that, and the commit is an explicit post-deploy trigger, not a filesystem watch. This mirrors that, adapted to codavox observing rather than owning the deploy.Snapshot (closes the live-staging race)
Resealnow materializes each current version as a deterministic.tar.gzunder<state>/artifacts— temp file + rename, so it publishes atomically — and serving streams that file. Bytes andcode_idcan no longer drift: the bytes are a snapshot from when the tree was quiescent, not whatever staging holds at request time.Trigger (closes the "nothing reseals after startup" gap)
SIGHUPreseals the running publisher, wired to r10k'spostrun:The signal fires only after r10k returns, so the tree is quiescent and no reseal observes a half-written deploy. This is deliberately not a filesystem watch: a watch would have to reconstruct the "deploy finished" signal
postrunalready gives us, and codavox doesn't own the deploy, so the explicit trigger is both simpler and the correct coupling. It's PE'scommit-content!-after-r10k, across a process boundary.SIGTERM/interrupt now drain in-flight downloads via graceful shutdown (Server.Serve(ctx)replaces the blockingListenAndServeTLS).API / signature changes
NewStore(stagingDir, artifactDir)— the publisher now needs a writable data dir. Thepublishcommand derives it from--state; tests that run the real publisher pass--state <tempdir>.--statenow holds both the provenance log and materialized artifacts.Tests
TestArtifactIsSnapshotNotLiveStaging— mutate staging after sealing, without a reseal; the served artifact still extracts to the sealedcode_idand carries the snapshot's bytes, not the mid-deploy write. This is the whole point of the change.TestResealReapsSupersededArtifact— a newcode_idreaps the old artifact and materializes the new one.TestSIGHUPTriggersReseal— deploy into a running publisher,SIGHUP, and a compiler converges to the new version with no restart.Verified:
go test -race ./...,golangci-lint0 issues, gofmt/vet/markdownlint clean.Docs: publishing.md — artifact endpoint, Resealing (SIGHUP +
postrun), and the--statepurpose.🤖 Generated with Claude Code