Skip to content

Sync github_runner profile to modules/ (all environments now identical) - #293

Merged
akuzminsky merged 1 commit into
mainfrom
sync-gha-runner-global
Aug 1, 2026
Merged

Sync github_runner profile to modules/ (all environments now identical)#293
akuzminsky merged 1 commit into
mainfrom
sync-gha-runner-global

Conversation

@akuzminsky

Copy link
Copy Markdown
Member

Final step of the promotion path for the bounded boot security upgrade. It has been running in development and sandbox; modules/ was the last copy still on the old unbounded exec.

Two files copied from environments/development, after confirming the global copy had no divergence of its own to clobber:

  • manifests/github_runner.pp — bounded exec
  • files/github_runner/gha-boot-security-upgrade.sh — new

github_runner is now byte-identical across modules/, environments/development and environments/sandbox.

What changes for production runners

The old exec had timeout => 1200, so one attempt could consume the entire ASG bootstrap lifecycle-hook budget, and adding tries on top would have multiplied that with no cumulative cap. The replacement caps total wall clock at 540s and retries inside it.

That bound matters because a resource failure here is not a slow run — ih-puppet applies with --detailed-exitcodes and exits 4/6 on failure, which trips ih-bootstrap.sh's trap _ih_signal_abandon ERR and ABANDONs the instance immediately.

Validation before promoting

Real gha_runner bootstraps in both environments:

  • script deploys, Exec[gha-boot-security-upgrade] succeeds
  • the /run/gha-boot-upgrade.done marker stops it re-running on the second catalog pass
  • bootstrap lifecycle hook completes CONTINUE

Bounding logic separately verified against stubbed apt-get/unattended-upgrade, 10/10 checks — gives up at its budget with no marker written, retries through transient failure, no needless delay on the fast path, and a hung command cannot overshoot the deadline (exits at 25s against a 600s sleep).

puppet-lint --fail-on-warnings clean on all four module dirs; shellcheck clean.

Deliberately not included

profile::apt_lock_timeout and the profile::repos change that declares it stay development+sandbox only. Production runners therefore get the bounded upgrade without the apt lock timeout. That is still better than today: a lock collision currently fails apt-get instantly and ABANDONs, whereas the bounded script retries and may well succeed. It just is not the full fix yet.

Known, unchanged by this PR

needrestart can restart cloud-final.service mid-bootstrap when the boot upgrade pulls a library — which kills the provisioning run, since that service is the bootstrap. Observed on a sandbox runner. This is pre-existing rather than introduced here, because the old exec ran the same unattended-upgrade; promoting neither causes nor fixes it. Tracked separately.

Refs #289

🤖 Generated with Claude Code

Completes the promotion path for the bounded boot security upgrade: it has been
running in development and sandbox, and modules/ was the last copy still on the
old unbounded exec.

Copies two files from environments/development, after confirming the global copy
had no divergence of its own to clobber:
  manifests/github_runner.pp                     (bounded exec)
  files/github_runner/gha-boot-security-upgrade.sh   (new)

github_runner is now byte-identical across modules/, environments/development
and environments/sandbox.

The old exec had timeout => 1200, so a single attempt could consume the entire
ASG bootstrap lifecycle-hook budget; the replacement caps total wall clock at
540s and retries within it. A resource failure here ABANDONs the instance
(ih-puppet exits 4/6 under --detailed-exitcodes, tripping ih-bootstrap's ERR
trap), so bounding it matters more than per-attempt tuning.

Validated on real gha_runner bootstraps in both environments before promoting:
script deploys, exec succeeds, the /run marker stops it re-running on the second
catalog pass, and the lifecycle hook completes CONTINUE.

Not included: profile::apt_lock_timeout and the profile::repos change that
declares it remain development+sandbox only. Production runners therefore get
the bounded upgrade without the apt lock timeout -- still better than today,
where a lock collision fails apt-get instantly, whereas the bounded script
retries and may well succeed.

Also unchanged: needrestart can restart cloud-final.service mid-bootstrap when
the boot upgrade pulls a library, which kills the provisioning run. That is
pre-existing -- the old exec ran the same unattended-upgrade -- and is tracked
separately.

Refs #289

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@akuzminsky
akuzminsky merged commit 482c721 into main Aug 1, 2026
2 checks passed
@akuzminsky
akuzminsky deleted the sync-gha-runner-global branch August 1, 2026 03:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants