feat: CI evals on Ubicloud — 12 parallel runners + Docker image (v0.11.10.0) by garrytan · Pull Request #360 · garrytan/gstack

garrytan · 2026-03-23T06:17:43Z

Summary

E2E evals run in CI on every PR. 12 parallel Ubicloud runners, each running one test suite. Docker image pre-bakes bun, node, Claude CLI, and deps for near-instant setup. Results posted as PR comment.
3x faster eval runs. Within-file test concurrency via testConcurrentIfSelected — wall clock ~6min vs ~18min.
Docker CI image auto-rebuilds on Dockerfile/package.json changes, cached by content hash in GHCR.
Routing test fix — skills installed at top-level .claude/skills/ for project-level discovery.

Test Coverage

All new code paths are CI config (YAML/Dockerfile) — not unit-testable. Test file changes are mechanical (test() → testConcurrentIfSelected()). Routing test fix is self-testing.

Pre-Landing Review

No issues found. Pure CI infra — no SQL, LLM, or security concerns.

Eval Results

No prompt-related files changed — evals skipped.

Test plan

bun test passes (exit 0)
Local eval concurrency verified: 2.5-2.8x speedup (browse: 53s→21s, routing: 500s→177s)
CI Docker image builds and runs successfully on Ubicloud
Eng review: CLEARED (0 issues, 0 critical gaps)

🤖 Generated with Claude Code

Switch all E2E tests from serial test() to testConcurrentIfSelected() so tests within each file run in parallel. Wall clock drops from ~18min to ~6min (limited by the longest single test, not sequential sum). The concurrent helper was already built in e2e-helpers.ts but never wired up. Each test runs in its own describe block with its own beforeAll/tmpdir — no shared state conflicts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Single-job GitHub Actions workflow that runs E2E evals on every PR using Ubicloud runners ($0.006/run — 10x cheaper than GitHub standard). Uses EVALS_CONCURRENCY=40 with the new within-file concurrency for ~6min wall clock. Downloads previous eval artifact from main for comparison, uploads results, and posts a PR comment with pass/fail + cost. Ubicloud setup required: connect GitHub repo via ubicloud.com dashboard, add ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY as repo secrets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…-duplicate Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…clock Matrix strategy spins up 12 ubicloud-standard-2 runners simultaneously, one per test file. Separate report job aggregates all artifacts into a single PR comment. Bun dependency cache cuts install from ~30s to ~3s. Runner cost: ~$0.048 (from $0.024) — negligible vs $3-4 API costs. Wall clock: ~3-4min (from ~8min). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Dockerfile.ci pre-installs bun, node, claude CLI, gh CLI, and node_modules so eval runners skip all setup. Image rebuilds weekly and on lockfile/Dockerfile changes via ci-image.yml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…clock Switch eval workflow to use Docker container image with pre-baked toolchain. Each of 12 matrix runners pulls the image, hardlinks cached node_modules, builds browse, and runs one test suite. Setup drops from ~70s to ~19s per runner. Wall clock is dominated by the slowest individual test, not sequential sum. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ntent hash Move Docker image build into the evals workflow as a dependency job. Image tag is keyed on hash of Dockerfile+lockfile+package.json — only rebuilds when those change. Eliminates chicken-and-egg problem where the image must exist before the first PR run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This project uses bun.lock (text format), not bun.lockb (binary). Also move Docker login before manifest inspect so GHCR auth works. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

bun.lock is in .gitignore so it doesn't exist after checkout. Dockerfile and workflows now use package.json only for deps caching. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Docker image layers and workspace are on different filesystems, so cp -al (hardlink) fails. Use ln -s (symlink) instead — zero copy overhead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…y-skip-permissions as root Claude Code CLI blocks --dangerously-skip-permissions when running as uid=0 for security. Add a 'runner' user to the Docker image and set --user runner on the container. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Claude CLI routing behavior changes when CI=true — it skips skill invocation and uses Bash directly. Unsetting these markers makes Claude behave like a local environment for consistent eval results. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Unsetting CI/GITHUB_ACTIONS didn't improve routing test results (still 1/11 in container). The issue is model behavior in containerized environments, not env vars. Routing tests will be tracked as a known CI gap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

In containerized CI, Claude lacks the project context (CLAUDE.md) that guides routing decisions locally. Without it, Claude answers directly with Bash/Agent instead of invoking specific skills. Copying CLAUDE.md gives Claude the same context it has locally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Routing tests now copy CLAUDE.md, README.md, package.json, ETHOS.md, and all SKILL.md files into each test tmpDir. This gives Claude the same project context it has locally, which is needed for correct skill routing decisions in containerized CI environments. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Claude Code discovers project skills from .claude/skills/<name>/SKILL.md at the top level only. Nesting under .claude/skills/gstack/<name>/ caused Claude to see only one "gstack" skill instead of individual skills like /ship, /qa, /review. This explains 10/11 routing failures in CI — Claude invoked "gstack" or used Bash directly instead of routing to specific skills. Also adds workflow_dispatch trigger and --user runner container option. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Two fixes: 1. Report job: add actions/checkout so `gh pr comment` has git context. Also add pull-requests:write permission for comment posting. 2. Routing tests: install skills to BOTH project-level (.claude/skills/) AND user-level (~/.claude/skills/) since Claude Code discovers from both locations. In CI containers, $HOME differs from workdir. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-actions · 2026-03-23T17:15:10Z

E2E Evals: ✅ PASS

22/22 tests passed | $3.94 total cost | 12 parallel runners

Suite	Result	Status	Cost
e2e-routing	11/11	✅	$1.97
e2e-routing	11/11	✅	$1.97

12x ubicloud-standard-2 (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite

PR garrytan#360 bumped VERSION to 0.11.10.0 but missed package.json. The gen-skill-docs test catches this: "package.json version matches VERSION file." Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

garrytan and others added 14 commits March 22, 2026 23:06

chore: bump version and changelog (v0.11.6.0)

4e5f71c

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: optimize CI eval PR comment — aggregate all suites, update-not…

9195b67

…-duplicate Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: bun.lockb → bun.lock + auth before manifest check

c44d8d4

This project uses bun.lock (text format), not bun.lockb (binary). Also move Docker login before manifest inspect so GHCR auth works. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: bun.lock is gitignored — use package.json only for Docker cache

ce31cb1

bun.lock is in .gitignore so it doesn't exist after checkout. Dockerfile and workflows now use package.json only for deps caching. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: symlink node_modules instead of hardlink (cross-device)

fe574f1

Docker image layers and workspace are on different filesystems, so cp -al (hardlink) fails. Use ln -s (symlink) instead — zero copy overhead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

debug: add claude CLI smoke test step to diagnose exit_code_1

57ca010

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ci: retrigger eval workflow

7598565

ci: add workflow_dispatch trigger for manual runs

50681f9

garrytan closed this Mar 23, 2026

garrytan reopened this Mar 23, 2026

garrytan and others added 10 commits March 23, 2026 07:15

debug: more verbose claude CLI diagnostics

4d3f8b4

fix: install bun to /usr/local so non-root runner user can access it

cb364e1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: merge origin/main (resolve CHANGELOG/VERSION/SKILL.md conflicts)

ab7b6d5

chore: bump version and changelog (v0.11.10.0)

4d91456

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

garrytan changed the title ~~feat: CI evals on Ubicloud + 3x faster test concurrency (v0.11.6.0)~~ feat: CI evals on Ubicloud — 12 parallel runners + Docker image (v0.11.10.0) Mar 23, 2026

garrytan merged commit f4bbfaa into main Mar 23, 2026
14 checks passed

This was referenced Mar 23, 2026

bug: package.json version out of sync with VERSION file on main #399

Open

fix: sync package.json version with VERSION file (0.11.10.0) #400

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: CI evals on Ubicloud — 12 parallel runners + Docker image (v0.11.10.0)#360

feat: CI evals on Ubicloud — 12 parallel runners + Docker image (v0.11.10.0)#360
garrytan merged 25 commits intomainfrom
garrytan/fast-ci-e2e-eval

garrytan commented Mar 23, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

garrytan commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Coverage

Pre-Landing Review

Eval Results

Test plan

Uh oh!

github-actions bot commented Mar 23, 2026

E2E Evals: ✅ PASS

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

garrytan commented Mar 23, 2026 •

edited

Loading