Self-hosted GitHub Actions runner (Hetzner) — cost reduction plan for resQ

[WomB0ComB0]

Context — billable Actions is ~$10.89/month, ~90% from resQ (private, tier=critical). Public repos already discounted; concurrency + caching already applied on resQ's heavy workflows. The remaining lever is running resQ's heavy jobs on a flat-fee runner instead of metered GitHub-hosted minutes.

---

Option matrix (prices current 2026-04-21, verified against Hetzner UI)

Option	Location	Specs	Monthly
Status quo (GitHub-hosted)	—	—	$11 billable + $20 Team = $31
Hetzner CX33 ✓ recommended	Ashburn VA / Hillsboro OR / Helsinki	4 vCPU Intel/AMD (x86), 8 GB, 80 GB SSD	~$28 ($7.99 + $20 Team, ~$0 billable) — saves ~$3/mo plus removes quota anxiety
Hetzner CX23	same	2 vCPU, 4 GB, 40 GB	~$25 ($4.99 + $20) — too tight for Rust workspace builds, don't pick this
Hetzner CX43	same	8 vCPU, 16 GB, 160 GB	~$34 ($13.99 + $20) — more headroom if you want fast parallel compiles
Hetzner CAX21 (ARM)	Helsinki only	4 vCPU Ampere ARM, 8 GB, 80 GB	~$29.49 ($9.49 + $20) — pick only if you specifically want ARM
Hetzner CPX31 (older AMD line)	Ashburn / Hillsboro	4 vCPU, 8 GB	~$33 ($13) — superseded by CX33; ignore
Blacksmith (drop-in SaaS)	US	4 vCPU x86	$23–28
Oracle Cloud Free Tier	US	4 ARM vCPU / 24 GB free	$20 (Team only)

Reading the UI right panel: if you're mid-provision with CX23 selected, bump to CX33 before clicking Create & Buy. Rust compile of a multi-crate workspace with release profile routinely spikes memory past 4 GB; 4 GB OOMs cc1, 8 GB sustains. Extra $3/month avoids weeks of flaky "mysteriously killed" CI runs.

Location: CX is available in Ashburn VA, Hillsboro OR, Helsinki, Falkenstein, Nuremberg, Singapore. Pick Ashburn if you want a dev box next to you; Helsinki if you want the same hardware that's shown in the pricing screenshots and don't mind ~100 ms latency for CI-only workloads (negligible — runs are minutes, not milliseconds).

---

Security constraint (important)

Register the runner only to private repos (resq-software/resQ, resq-software/resq-proto if needed). Never register it at org level with public repos visible, and never add it to resq-software/ardupilot, landing, crates, etc. Self-hosted runners on public repos let any drive-by PR execute arbitrary code on your machine — GitHub explicitly recommends against this.

---

Setup: Hetzner CX33

1. Provision the VM

• Hetzner Cloud → Servers → Add Server
• Location: Ashburn VA (US East) / Hillsboro OR (US West) / Helsinki (EU) — your call
• Image: Ubuntu 24.04 LTS
• Type: CX33 — 4 vCPU Intel/AMD, 8 GB, 80 GB SSD — $7.99/mo
• SSH key: yours
• Firewall: allow only outbound + SSH inbound

2. Install the runner on the VM

sudo useradd -m -s /bin/bash runner
sudo apt update && sudo apt install -y jq git curl build-essential

sudo -iu runner
mkdir actions-runner && cd actions-runner

# x64 runner binary (CX is x86_64 Intel/AMD)
curl -sSLO https://github.com/actions/runner/releases/download/v2.333.1/actions-runner-linux-x64-2.333.1.tar.gz
tar xzf actions-runner-linux-x64-2.333.1.tar.gz

3. Register against resq-software/resQ (NOT org-wide)

Get a registration token:

gh api --method POST /repos/resq-software/resQ/actions/runners/registration-token --jq '.token'

On the VM (as runner):

./config.sh \
  --url https://github.com/resq-software/resQ \
  --token $REG_TOKEN \
  --name hetzner-cx33-1 \
  --labels self-hosted,linux,x64,hetzner \
  --work _work \
  --unattended

4. Run as systemd service

sudo ./svc.sh install runner
sudo ./svc.sh start
sudo ./svc.sh status

5. Install Docker (if docker-publish.yml targets this runner)

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker runner
sudo systemctl restart actions.runner.*.service

6. Opt in per-workflow — don't swap all at once

Start with rust-ci.yml:

runs-on: ${{ vars.USE_SELF_HOSTED == 'true' && fromJSON('["self-hosted", "linux", "x64"]') || 'ubuntu-latest' }}

Set vars.USE_SELF_HOSTED at repo level (Settings → Secrets and variables → Actions → Variables). Flip to true to route runs to Hetzner; flip to false to instantly fall back if the VM has issues.

---

x64-specific notes for resQ's stack

• Rust: full native, every crate compiles. No inline-asm gotchas.
• Docker multi-arch: if docker-publish.yml builds linux/amd64,linux/arm64, amd64 is native; arm64 goes through qemu (~4× slower). Options:
  • Split into two parallel jobs (amd64 on self-hosted, arm64 on ubuntu-latest) and merge via buildx manifest
  • Keep as-is and accept the slower arm64 build
  • Drop linux/arm64 from the matrix if you don't publish ARM images to users
• Bun, uv, .NET SDK: native x64, all fine.

---

Monitoring & rollback

• Runner health: https://github.com/resq-software/resQ/settings/actions/runners
• systemd journal: sudo journalctl -u 'actions.runner.*' -f
• Disk cleanup cron (weekly): find ~runner/actions-runner/_work -type d -name '_temp' -mtime +3 -exec rm -rf {} +
• Rollback: set vars.USE_SELF_HOSTED=false → next PR runs on ubuntu-latest again. No workflow changes needed.

---

Validation checklist

• Provisioned Hetzner CX33 (not CX23) in Ashburn / Hillsboro / Helsinki
• Runner installed + registered to resq-software/resQ only
• systemd service running (sudo ./svc.sh status shows active)
• Docker installed (if targeting docker-publish.yml)
• USE_SELF_HOSTED repo variable added, set to true on rust-ci.yml first
• Verified a test PR routes to hetzner-cx33-1
• Rolled out to ci.yml
• Rolled out to docker-publish.yml (with multi-arch plan decided)
• Measured billable delta after 2 weeks vs April 2026 baseline

---

Alternative #1 — Blacksmith (zero-infra, instant)

runs-on: blacksmith-4vcpu-ubuntu-2404

Sign up at https://blacksmith.sh → install their GitHub App on resQ → swap the label. No VM to manage. ~25% of GHA per-minute cost. Est monthly: $3–8 for resQ. Best single-step alternative if you'd rather not operate infrastructure.

Alternative #2 — Oracle Cloud Free Tier

Free: 4 Ampere ARM vCPU / 24 GB RAM / 200 GB block storage. Setup is ~2 hours (OCI console is fiddly) but the monthly cost is $0. Trade-off: ARM-only (same Docker-multi-arch caveats as CAX21) and you depend on Oracle's free-tier policy not changing.

Alternative #3 — Hetzner CAX21 ARM (Helsinki only)

If you specifically want ARM and don't mind Helsinki latency, CAX21 is $9.49/mo for 4 vCPU Ampere / 8 GB. Swap labels to arm64 and use the actions-runner-linux-arm64-*.tar.gz binary. Compared to CX33, it's $1.50 more expensive and adds ARM-specific gotchas (some crypto crates need feature flags; Docker amd64 via qemu is slow). Not worth it over CX33 unless you have an ARM-specific reason.

---

Revision history:

• 2026-04-18 · initial version (CAX21, EU)
• 2026-04-21 · swapped CAX21 → CPX31 for US availability
• 2026-04-21 · updated to CX33 (Hetzner's current Intel/AMD line; newer and cheaper than CPX31), flagged "don't pick CX23 for Rust CI" warning, corrected all prices against the Hetzner UI pricing shown in the provisioning screenshots