EC2 GitHub Runner Hardening & Modernization Plan (tracker)

[kurok]

Tracker for a multi-phase effort to modernize and harden namecheap/ec2-github-runner for production-grade usage. Child issues listed below; this one stays open until every phase is landed.

Overview

This action is the critical supply-chain link between every self-hosted CI pipeline in the org and AWS. Current state has known issues:

• Legacy node12 runtime (fixed via feat: declare action runtime as node24 #4 but more dep modernization needed).
• aws-sdk v2 (in maintenance mode, emits DEP0169).
• Long-lived static AWS keys recommended as default.
• Classic GitHub PAT recommended as default.
• Runner binary version hardcoded; no configurable override.
• Runner runs as root (RUNNER_ALLOW_RUNASROOT=1).
• Best-effort cleanup (no retries, no bounded timeouts).
• Ad-hoc logging; no debug mode.
• Zero unit tests.

Goals

1. Eliminate long-lived credentials.
2. Enforce ephemeral runner model.
3. Apply least-privilege IAM.
4. Modernize runtime and dependencies.
5. Improve lifecycle reliability and cleanup.
6. Reduce operational toil.
7. Provide scalable and secure defaults.

Phases

• Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 — Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization).
• Phase 2: support GitHub OIDC for AWS credentials #8 — Phase 2: support GitHub OIDC for AWS credentials.
• Phase 3: recommend GitHub App token / fine-grained PAT over classic PAT #9 — Phase 3: recommend GitHub App token / fine-grained PAT over classic PAT.
• Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version #10 — Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version.
• Phase 5: lifecycle & cleanup reliability — retries, timeouts, always-cleanup #11 — Phase 5: lifecycle & cleanup reliability — retries, timeouts, always-cleanup.
• Phase 6: secure defaults — IMDSv2, encrypted EBS, subnet/SG guidance #12 — Phase 6: secure defaults — IMDSv2, encrypted EBS, subnet/SG guidance.
• Phase 7: structured logging + debug mode #13 — Phase 7: structured logging + debug mode.
• Phase 8: unit tests for config, AWS params, and GitHub flow #14 — Phase 8: unit tests for config, AWS params, and GitHub flow.

Compatibility with the primary consumer (terraform-provider-namecheap acceptance tests)

Phase	Breaks acctest?	Notes
1 (aws-sdk v3)	No — if input/output contract preserved	Regression verified via dogfood SHA-pin rotation
2 (OIDC)	No (opt-in)	Provider repo keeps working on static keys; cross-repo migration separate
3 (token types)	No (opt-in)	Classic PAT stays accepted; docs change only
4 (non-root + --ephemeral)	Medium risk	make testacc = plain go test, setup is workspace-local — no root needed. But deserves a dogfood push to confirm
5 (lifecycle)	No	Strict improvement
6 (secure defaults)	No	Provider's existing SG + EIP + AMI continue to work; IMDSv2 is transparent to aws-sdk / SSM
7 (logging)	No	Output-only change
8 (tests)	No	Build-side

Phase 4 is the only place where a dogfood SHA-pin rotation on a terraform-provider-namecheap throwaway branch is strongly recommended before the underlying PR lands on feat/al2023-support.

Already-landed groundwork

Not part of the plan formally but relevant context:

• feat: bump actions/runner to v2.333.1 for node24 support #3 — bump actions/runner to v2.333.1 (node24 externals) + add verify-dist + verify-runner-url CI guards.
• feat: declare action runtime as node24 #4 — action.yml using: node12 → using: node24.
• fix: write outputs to GITHUB_OUTPUT file instead of ::set-output #5 — outputs written to $GITHUB_OUTPUT, not ::set-output.
• fix: silence DEP0169 url.parse deprecation from bundled aws-sdk v2 #6 — filter DEP0169 (url.parse) deprecation from bundled aws-sdk v2.

Suggested ordering

1. Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 first — aws-sdk v3 + ncc bump is the prerequisite for any further dep modernization (including @actions/core ≥ 1.10 which the current ncc can't parse).
2. Phase 8: unit tests for config, AWS params, and GitHub flow #14 in parallel — unit tests give confidence for every subsequent refactor.
3. Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version #10 — bootstrap hardening touches source that Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 already refactored; landing Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 first keeps the diff smaller.
4. Phase 5: lifecycle & cleanup reliability — retries, timeouts, always-cleanup #11 — lifecycle retries build on Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version #10's ephemeral foundation.
5. Phase 7: structured logging + debug mode #13 — logging changes read easier once Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7's SDK rewrite is in place.
6. Phase 6: secure defaults — IMDSv2, encrypted EBS, subnet/SG guidance #12 — secure defaults (code side) and Phase 2: support GitHub OIDC for AWS credentials #8 (OIDC docs + optional role-to-assume input) can land any time; no dependency.
7. Phase 3: recommend GitHub App token / fine-grained PAT over classic PAT #9 — token-type docs any time; no dependency.

Total estimated effort: 5–7 working days per the plan author.

[kurok]

Status update after Phase 4 rollback

Phase	Status
1 — aws-sdk v3 + ncc 0.38 (#7)	✅ merged + provider dogfood green
8.a — tests for utils + config (#14 partial)	✅ merged
4 — non-root + --ephemeral + runner-version (#10)	❌ rolled back
4.b — non-root retry with debug instrumentation (#20)	📋 filed, scoped, not started

Why Phase 4 rolled back

Two dogfood attempts (#18 with non-root, #19 without non-root but with the other four Phase-4 changes) both failed with the identical 6m15s registration timeout on Start self-hosted EC2 runner. Can't post-mortem the ephemeral instance. Full revert landed via #21 → tip 249efbd.

Root-cause hypothesis documented on #20: set -euo pipefail I added at the top of the user-data likely promotes a silent failure from mount -o remount,size=1G /tmp (which doesn't apply on the hardened AL2023 AMI where /tmp isn't a separate mount point) into a hard bootstrap termination.

What's still planned

• Phase 5 — lifecycle retries + timeouts (Phase 5: lifecycle & cleanup reliability — retries, timeouts, always-cleanup #11): can proceed independently of the bootstrap. Touches src/gh.js + src/aws.js error paths.
• Phase 7 — structured logging + debug mode (Phase 7: structured logging + debug mode #13): gives Phase 4.b: drop runner to non-root user — second attempt with bootstrap debuggability #20 the instrumentation it needs. Should probably precede any retry of Phase 4.
• Phase 6 — IMDSv2 + encrypted EBS (Phase 6: secure defaults — IMDSv2, encrypted EBS, subnet/SG guidance #12): touches RunInstances params, separable.
• Phase 2 + 3 — OIDC and token docs (Phase 2: support GitHub OIDC for AWS credentials #8 + Phase 3: recommend GitHub App token / fine-grained PAT over classic PAT #9): pure documentation, any time.
• Phase 4.b — non-root retry (Phase 4.b: drop runner to non-root user — second attempt with bootstrap debuggability #20): blocked on Phase 7 landing so bootstraps are debuggable.

Moving ahead with the order 7 → 5 → 6 → 2 + 3 → 4.b instead of the original tracker ordering.