Phase 4.b: drop runner to non-root user — second attempt with bootstrap debuggability

[kurok]

Part of plan #15. Follow-up to Phase 4 (#10) — originally intended to be part of that PR but reverted after the dogfood failure.

Context

Phase 4 (#10 → PR #18) tried to drop the runner from root to a dedicated runner user via sudo -u runner -H bash <<'RUNNER_BOOTSTRAP' wrapping the download + configure + run sequence. The terraform-provider-namecheap#182 dogfood failed at Start self-hosted EC2 runner with the 5-minute registration timeout — the EC2 instance came up but the runner never reached ./run.sh polling GitHub.

PR #19 (the fix-forward) reverted the non-root transition and kept everything else from Phase 4 (runner-version input, --ephemeral, --unattended, --disableupdate, SHA-256 checksum, set -euo pipefail). Dogfood with that pin now passes.

The missing piece

Instrumentation. We can't post-mortem the failed instance — it was ephemeral and got terminated by the stop-runner step before we could SSH in. Any second attempt at non-root needs to leave a breadcrumb trail we can pull up after the fact.

Suggested approach

1. Add an opt-in debug mode (via the Phase 7 debug input)

When debug: true:

• set -x on the bootstrap so every command is echoed.
• Stream stdout+stderr to /var/log/ec2-github-runner-bootstrap.log (world-readable).
• Upload the log to S3 (bucket passed via new debug-log-bucket input) on exit, pass or fail. Requires an additional s3:PutObject permission on the runner's IAM instance profile.
• Or simpler: write the log to the instance metadata service as a User-Data/Instance-Identity field — no extra IAM, but size-limited.

2. Reproduce outside the ephemeral flow

Launch one test instance with the same AMI + the failing user-data, and watch the EC2 console output via aws ec2 get-console-output. No runner registration needed — just look at where cloud-init actually stopped.

3. Binary-search the bootstrap

Rather than adding back all the non-root machinery at once, add it in the smallest possible steps:

• Step a: useradd runner only. Keep root execution. Verify dogfood passes.
• Step b: sudo -u runner true before anything else. Verify dogfood passes.
• Step c: Move only the tarball download + extraction to the runner user. Keep root for config.sh + run.sh.
• Step d: Move config.sh as well.
• Step e: Move run.sh.

Any step that fails dogfood is the breaker and can be investigated in isolation.

Likely suspects

• requiretty or Defaults env_reset in sudoers on the DEVOPS/hardened-amazon-linux2023 AMI. Cloud-init runs user-data in a non-interactive, tty-less context; requiretty would kill sudo -u runner.
• SELinux enforcing a context on /home/runner that prevents config.sh from writing its .runner and .credentials files. Visible in audit.log / journalctl.
• Heredoc quoting in my JS — unlikely, but the quoted delimiter <<'RUNNER_BOOTSTRAP' vs unquoted might have interacted weirdly with the token containing special characters.
• Runner user's PATH — config.sh and run.sh inside /home/runner/actions-runner/ execute via ./ which doesn't need . in PATH, but some internal runner code might.

Acceptance criteria

• Debug mode exists that makes bootstrap failures diagnosable from outside the instance.
• Non-root runner user applied in the smallest coherent chunk that passes dogfood.
• A decision recorded about whether the hardened-AL2023 AMI's sudoers / SELinux config needs changing, or whether the action works around them.
• The RUNNER_ALLOW_RUNASROOT=1 escape hatch is gone at the end of this work.

Severity

Medium — security hardening, not a blocker. Phase 4 shipped the safer pieces; non-root is the outstanding goal.

[kurok]

Root-cause hypothesis after two dogfood failures

Both Phase 4 attempts (#18 with non-root, #19 without non-root) failed dogfood with the identical 6m15s registration timeout. Since the fix-forward kept the non-root revert, the non-root transition was not the breaker.

My current hypothesis: the breaker is the set -euo pipefail I added at the top of the user-data, in combination with:

mount -o remount,size=1G /tmp

On the DEVOPS/hardened-amazon-linux2023 AMI, /tmp appears to not be a separate mount point. Pre-Phase-4 the failure from this command was silently ignored (no set -e). With set -e, it kills the bootstrap immediately — before yum install, before curl, before anything.

When Phase 4 work resumes, the minimal reproduction would be: add JUST set -euo pipefail to the pre-Phase-4 bootstrap and dogfood. If it fails with the same timeout, that's the smoking gun.

Mitigations to try, in order of simplicity:

1. mount -o remount,size=1G /tmp || true — explicit fallthrough on the line that's likely failing.
2. Delete the mount line entirely — probably a legacy carry-over; AL2023 has /tmp large enough by default.
3. Move set -euo pipefail to AFTER the mount + yum install steps (lets them fail silently like the pre-Phase-4 path while the runner setup itself is strict).

Phase 4.b (non-root + debug instrumentation) still needs to happen on top of whichever mitigation lands.

[kurok]

Root cause confirmed

Not mount -o remount. The breaker was set -euo pipefail + curl -fsSL <tarball>.sha256 — actions/runner doesn't publish per-tarball .sha256 sidecar files. Empirically verified:

curl -fsSLI https://github.com/actions/runner/releases/download/v2.333.1/actions-runner-linux-x64-2.333.1.tar.gz.sha256
→ HTTP/2 404

The 404 + -f (fail-on-http-error) killed curl, set -e killed the whole bootstrap immediately. EC2 instance stayed up but never reached ./run.sh. Both #18 and #19 had this same code path, so both failed identically.

Checksums ARE published, just not where I looked

Inside the release body as markdown with HTML-comment markers:

## SHA-256 Checksums
- actions-runner-linux-x64-2.333.1.tar.gz <!-- BEGIN SHA linux-x64 -->…<!-- END SHA linux-x64 -->

Recommendation when Phase 4.b starts

Ship a hardcoded RUNNER_VERSION → SHA-256 table in src/aws.js, updated at the same time as the runner-version default. verify-runner-url CI job extends to cross-check the stored hash against api.github.com's release body on every PR. No runtime API hits, no 404 surprises.

Also worth a GitHub feature-request upstream: https://github.com/actions/runner/issues — ask for per-tarball .sha256 sidecars. Would let every consumer use the simpler pattern.

Debuggability (the actual lesson)

The reason this took three dogfood attempts and a full revert to isolate is that the ephemeral instance is terminated by stop-runner before anyone can SSH in. Two AWS-side changes would make future bootstrap bugs trivially diagnosable:

1. Redirect cloud-init output to a log and upload on exit:

   exec > >(tee /var/log/user-data.log) 2>&1
   trap 'aws s3 cp /var/log/user-data.log s3://$DEBUG_BUCKET/$(hostname)-$(date +%s).log' EXIT

   Needs a debug-log-bucket input + s3:PutObject on the runner's instance profile.

2. SSM Session Manager on the runner role — aws ssm start-session into a failing instance before it's terminated. Requires pausing termination on registration-timeout.

Either would have turned this 5-hour investigation into 5 minutes of log-reading.

[kurok]

Root cause — empirically confirmed

Launched a probe EC2 in the CI region with the same AMI / subnet / SG that CI uses with the Phase 4 user-data + set -euxo pipefail and progress markers. Grabbed console output via aws ec2 get-console-output --latest.

Steps 1–6 all succeeded (visible in console):

• mount -o remount,size=1G /tmp ✓ (tmp IS tmpfs on this AMI)
• yum install -y libicu make ✓
• mkdir actions-runner && cd actions-runner ✓
• case "$(uname -m)" → RUNNER_ARCH=x64 ✓
• curl -fsSLo "$TARBALL" "$BASE/$TARBALL" ✓ (tarball downloaded, 700 ms)

Step 7 died exactly here:

[   16.830788] + curl -fsSLo actions-runner-linux-x64-2.333.1.tar.gz https://github.com/actions/runner/releases/download/v2.333.1/actions-runner-linux-x64-2.333.1.tar.gz
[   17.544561] + echo '--- step 6 OK ---'
[   17.547]   + echo '--- step 7: curl .sha256 sidecar (this is the suspect) ---'
[   17.554]   ++ curl -fsSL https://github.com/actions/runner/releases/download/v2.333.1/actions-runner-linux-x64-2.333.1.tar.gz.sha256
[   17.792348] curl: (22) The requested URL returned error: 404
[   17.795991] + expected=

12 seconds after expected= the empty string was assigned, cloud-init reports cc_scripts_user.py[WARNING]: Failed to run module scripts-user — that's set -eo pipefail firing on the curl … | awk … pipeline's non-zero exit.

Narrowed candidate list

The Phase 4 fix is now unambiguous:

• The .sha256 fetch must be replaced or removed.
• No need to touch mount, yum install, or the set -e machinery; they all work.
• The non-root transition was also innocent (didn't reach the sudo -u runner line).

Fix options (in order of simplicity)

1. Drop the checksum check entirely. curl -fsSL over HTTPS + the tarball URL on github.com gives TLS integrity. Real residual risk is a GitHub CDN compromise — negligible for our threat model, and if it happened every consumer of this action is affected anyway.

2. Parse the checksum out of the release body. actions/runner publishes hashes as HTML-comment-wrapped markdown:

   ## SHA-256 Checksums
   - actions-runner-linux-x64-2.333.1.tar.gz <!-- BEGIN SHA linux-x64 -->…<!-- END SHA linux-x64 -->
   

   Fetch-and-grep from api.github.com/repos/actions/runner/releases/tags/vX.Y.Z. Trades a 404-risk for a rate-limit-risk (60/hr unauth).

3. Hardcode a version → sha256 table in the action source. Deterministic, no network round-trip. Maintenance: bump the table whenever runner-version bumps. CI job can cross-check the stored hash against api.github.com on every PR.

Recommend option 3 when Phase 4.b starts — the table is <15 lines and the verify-runner-url CI job naturally extends to enforce consistency.

Probe instance cleanup

Probe instance terminated. Cost impact: t3.medium for ~3 min ≈ $0.002.

[kurok]

Landed via #26. Hardcoded checksum table cross-checked against upstream release body in CI. Dogfood rotation going up on terraform-provider-namecheap next.