Sync github_runner profile to modules/ (all environments now identical) - #293
Merged
Conversation
Completes the promotion path for the bounded boot security upgrade: it has been running in development and sandbox, and modules/ was the last copy still on the old unbounded exec. Copies two files from environments/development, after confirming the global copy had no divergence of its own to clobber: manifests/github_runner.pp (bounded exec) files/github_runner/gha-boot-security-upgrade.sh (new) github_runner is now byte-identical across modules/, environments/development and environments/sandbox. The old exec had timeout => 1200, so a single attempt could consume the entire ASG bootstrap lifecycle-hook budget; the replacement caps total wall clock at 540s and retries within it. A resource failure here ABANDONs the instance (ih-puppet exits 4/6 under --detailed-exitcodes, tripping ih-bootstrap's ERR trap), so bounding it matters more than per-attempt tuning. Validated on real gha_runner bootstraps in both environments before promoting: script deploys, exec succeeds, the /run marker stops it re-running on the second catalog pass, and the lifecycle hook completes CONTINUE. Not included: profile::apt_lock_timeout and the profile::repos change that declares it remain development+sandbox only. Production runners therefore get the bounded upgrade without the apt lock timeout -- still better than today, where a lock collision fails apt-get instantly, whereas the bounded script retries and may well succeed. Also unchanged: needrestart can restart cloud-final.service mid-bootstrap when the boot upgrade pulls a library, which kills the provisioning run. That is pre-existing -- the old exec ran the same unattended-upgrade -- and is tracked separately. Refs #289 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
infrahouse8
approved these changes
Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final step of the promotion path for the bounded boot security upgrade. It has been running in development and sandbox;
modules/was the last copy still on the old unbounded exec.Two files copied from
environments/development, after confirming the global copy had no divergence of its own to clobber:manifests/github_runner.pp— bounded execfiles/github_runner/gha-boot-security-upgrade.sh— newgithub_runneris now byte-identical acrossmodules/,environments/developmentandenvironments/sandbox.What changes for production runners
The old exec had
timeout => 1200, so one attempt could consume the entire ASG bootstrap lifecycle-hook budget, and addingtrieson top would have multiplied that with no cumulative cap. The replacement caps total wall clock at 540s and retries inside it.That bound matters because a resource failure here is not a slow run —
ih-puppetapplies with--detailed-exitcodesand exits 4/6 on failure, which tripsih-bootstrap.sh'strap _ih_signal_abandon ERRand ABANDONs the instance immediately.Validation before promoting
Real
gha_runnerbootstraps in both environments:Exec[gha-boot-security-upgrade]succeeds/run/gha-boot-upgrade.donemarker stops it re-running on the second catalog passCONTINUEBounding logic separately verified against stubbed
apt-get/unattended-upgrade, 10/10 checks — gives up at its budget with no marker written, retries through transient failure, no needless delay on the fast path, and a hung command cannot overshoot the deadline (exits at 25s against a 600s sleep).puppet-lint --fail-on-warningsclean on all four module dirs;shellcheckclean.Deliberately not included
profile::apt_lock_timeoutand theprofile::reposchange that declares it stay development+sandbox only. Production runners therefore get the bounded upgrade without the apt lock timeout. That is still better than today: a lock collision currently failsapt-getinstantly and ABANDONs, whereas the bounded script retries and may well succeed. It just is not the full fix yet.Known, unchanged by this PR
needrestart can restart
cloud-final.servicemid-bootstrap when the boot upgrade pulls a library — which kills the provisioning run, since that service is the bootstrap. Observed on a sandbox runner. This is pre-existing rather than introduced here, because the old exec ran the sameunattended-upgrade; promoting neither causes nor fixes it. Tracked separately.Refs #289
🤖 Generated with Claude Code