test: harden e2e infra, colocate helpers, and add tooling configs by wyattjoh · Pull Request #63 · clerk/cli

wyattjoh · 2026-03-26T15:29:11Z

Summary

This branch bundles four threads of work that grew out of the E2E test
infrastructure introduced in #59:

Test helper colocation

Move src/test/stubs.ts to src/test/lib/stubs.ts
Move src/test/integration/lib/setup.ts to
src/test/integration/lib/harness.ts
Update all unit and integration test imports to match

Claude rules and developer docs

New .claude/rules/{commands,errors,testing,e2e}.md covering command
authoring, error helpers, unit test conventions, and the full E2E flow
New "E2E tests" section in CONTRIBUTING.md documenting the local
1Password-driven workflow (bun run test:e2e:op)
Trim stale Bun/HTML scaffolding from CLAUDE.md and clarify that E2E
targets the production Clerk API with a dedicated test application

E2E test infrastructure hardening

scripts/refresh-e2e-fixtures.ts: error when --only is passed without
a value, instead of silently refreshing every fixture
scripts/run-e2e.ts: align default concurrency with the package script
test/e2e/lib/dev-server.ts: resolve the getAvailablePort TOCTOU race
by allocating the port inside startDevServer and retrying on
EADDRINUSE / "port X is in use" output, with early-bail on conflict
instead of waiting the full 60s readiness timeout. Up to three attempts
before surfacing a hard failure.

Dependabot

New .github/dependabot.yml with weekly bun and github-actions
updates
Fixture projects under test/e2e/fixtures/* are excluded both by not
being listed as a configured directory and via an explicit
exclude-paths glob as a defensive backstop, since those fixtures are
pinned snapshots regenerated by bun run e2e:refresh-fixtures

Test plan

bun run format:check
bun run lint
bun run test
bun run test:e2e:op exercises the port-conflict retry path under
concurrency 4

Summary by CodeRabbit

Tests
- E2E tests now execute 4 files concurrently by default, significantly improving test feedback cycles.
Reliability
- Dev server now intelligently detects port conflicts and automatically retries allocation, eliminating startup failures.
Chores
- Automated dependency updates now enabled via Dependabot configuration.
- Established CLI command authoring, error handling, and testing conventions documentation.

wyattjoh · 2026-03-31T14:34:21Z

Stack: e2e-stack

test: harden e2e infra, colocate helpers, and add tooling configs #63 👈 this PR
- refactor(test): consolidate unit and e2e test runners #100

Part of a stacked PR chain. Do not merge manually.

coderabbitai · 2026-03-31T15:10:57Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 9cb1161c-6874-4010-a263-0e2bc8da7152

📥 Commits

Reviewing files that changed from the base of the PR and between babe2c2 and 30caec2.

📒 Files selected for processing (49)

.claude/rules/commands.md
.claude/rules/e2e.md
.claude/rules/errors.md
.claude/rules/testing.md
.github/dependabot.yml
CLAUDE.md
CONTRIBUTING.md
package.json
packages/cli-core/src/commands/api/bapi.test.ts
packages/cli-core/src/commands/api/catalog.test.ts
packages/cli-core/src/commands/api/index.test.ts
packages/cli-core/src/commands/api/interactive.test.ts
packages/cli-core/src/commands/api/ls.test.ts
packages/cli-core/src/commands/apps/list.test.ts
packages/cli-core/src/commands/auth/login.test.ts
packages/cli-core/src/commands/auth/logout.test.ts
packages/cli-core/src/commands/config/pull.test.ts
packages/cli-core/src/commands/config/push.test.ts
packages/cli-core/src/commands/config/schema.test.ts
packages/cli-core/src/commands/deploy/index.test.ts
packages/cli-core/src/commands/doctor/context.test.ts
packages/cli-core/src/commands/doctor/doctor.test.ts
packages/cli-core/src/commands/env/pull.test.ts
packages/cli-core/src/commands/link/index.test.ts
packages/cli-core/src/commands/switch-env/index.test.ts
packages/cli-core/src/commands/unlink/index.test.ts
packages/cli-core/src/commands/whoami/index.test.ts
packages/cli-core/src/lib/autolink.test.ts
packages/cli-core/src/lib/credential-store.test.ts
packages/cli-core/src/lib/plapi.test.ts
packages/cli-core/src/test/integration/agent-mode.test.ts
packages/cli-core/src/test/integration/api-queries.test.ts
packages/cli-core/src/test/integration/auth-lifecycle.test.ts
packages/cli-core/src/test/integration/completion.test.ts
packages/cli-core/src/test/integration/config-management.test.ts
packages/cli-core/src/test/integration/config-put.test.ts
packages/cli-core/src/test/integration/deploy-to-prod.test.ts
packages/cli-core/src/test/integration/dry-run.test.ts
packages/cli-core/src/test/integration/env-merge.test.ts
packages/cli-core/src/test/integration/error-codes.test.ts
packages/cli-core/src/test/integration/error-recovery.test.ts
packages/cli-core/src/test/integration/lib/harness.ts
packages/cli-core/src/test/integration/onboard.test.ts
packages/cli-core/src/test/integration/switch-apps.test.ts
packages/cli-core/src/test/lib/stubs.ts
scripts/refresh-e2e-fixtures.ts
scripts/run-e2e.ts
test/e2e/lib/dev-server.ts
test/e2e/lib/fixture-test.ts

📝 Walkthrough

Walkthrough

This pull request adds comprehensive development documentation, reorganizes test infrastructure, and enhances E2E test execution. New convention files define CLI command structure, error handling patterns, unit/integration testing practices, and E2E fixture setup. Test stub imports are consolidated under test/lib/stubs.ts, and integration tests are refactored to use a new harness module at test/integration/lib/harness.ts. The E2E dev server now includes internal port allocation with retry logic on port conflicts, and the default concurrency for E2E tests is increased from 1 to 4. Dependabot configuration enables automated dependency updates. Several test scripts and documentation are updated to reflect these infrastructure changes.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: E2E infrastructure hardening, test helper colocation, and addition of tooling/documentation configs.
Docstring Coverage	✅ Passed	Docstring coverage is 85.71% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Move test/stubs.ts to test/lib/stubs.ts so it lives alongside other test helpers. Move test/lib/setup.ts to test/integration/lib/harness.ts so the integration test harness is colocated with the integration tests that use it. Update all import paths in unit and integration tests accordingly.

Add Claude rules for command authoring conventions, error handling patterns, and unit test conventions. Document CLERK_CLI_TOKEN non-interactive authentication in the auth command README. Update CLAUDE.md test command to exclude e2e tests by default.

Two test files were merged into main on PR #59 after the colocation refactor was originally drafted, so they kept their pre-refactor import paths and broke unit tests on this branch: - commands/apps/list.test.ts referenced ../../test/stubs.ts - test/integration/completion.test.ts referenced ../lib/setup.ts Both now match the convention used by every other file in their respective directories.

- Remove unimplemented CLERK_CLI_TOKEN section from auth README - Error in refresh-e2e-fixtures when --only is missing a value (previously refreshed every fixture silently) - Align e2e test concurrency default to 4 across runner, package.json, and .claude/rules/e2e.md - Narrow testing rule to permit import-time mock.module() registration - Fix .claude/rules path globs to match packages/cli-core/src layout - Correct CI secret name and clarify production API targeting in e2e rules and CLAUDE.md - Fix bun test -> bun run test drift in CONTRIBUTING.md

getAvailablePort binds to port 0 and closes the listener before the dev server starts, leaving a TOCTOU window where a sibling fixture (or anything on the host) can grab the same port. At concurrency 4 this is a real collision risk. Move port allocation inside startDevServer and retry on conflict. A new internal tryStart helper watches dev server stdout/stderr for EADDRINUSE / "port X is in use" patterns and bails early instead of waiting the full 60s readiness timeout, then the outer loop picks a fresh port. Max 3 attempts before surfacing a bind failure. getAvailablePort is now module-private. The caller in fixture-test.ts reads the chosen port from the start result.

Configures weekly dependency updates for the root Bun workspace and the GitHub Actions used in CI. Fixture projects under test/e2e/fixtures/* are intentionally not listed: they are pinned snapshots of scaffolded framework output and are regenerated via `bun run e2e:refresh-fixtures`. Dependabot only checks the directories it is told about, so leaving them out means they will not receive update PRs.

Belt-and-suspenders defense. Dependabot does not auto-discover manifests, so the fixtures are already excluded by virtue of not being listed under `directory`. The new `exclude-paths` entry makes the intent explicit in the config and protects against future expansions of the configured paths (e.g. switching to `directories` with a glob).

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from c56ed4d to ef155ef Compare March 26, 2026 15:31

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 7159fcb to 136d8c7 Compare March 26, 2026 15:31

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from ef155ef to c5cc804 Compare March 26, 2026 17:42

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 136d8c7 to 3b97b7b Compare March 26, 2026 17:43

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch 3 times, most recently from 1448c34 to e1599e3 Compare March 26, 2026 18:53

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 3b97b7b to 43a9f34 Compare March 26, 2026 18:53

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from e1599e3 to 39f7c6f Compare March 26, 2026 19:02

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 43a9f34 to f2cc487 Compare March 26, 2026 19:02

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from 39f7c6f to 22dcbcf Compare March 26, 2026 19:24

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from f2cc487 to 6b3844a Compare March 26, 2026 19:24

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from 22dcbcf to a662609 Compare March 26, 2026 19:30

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 6b3844a to 0e4adaf Compare March 26, 2026 19:30

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from a662609 to d8ac3fb Compare March 27, 2026 22:10

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch 2 times, most recently from 891752f to be5ec70 Compare March 31, 2026 14:32

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from d8ac3fb to 37eb8ca Compare March 31, 2026 14:32

wyattjoh mentioned this pull request Mar 31, 2026

feat(init): support CLERK_PLATFORM_API_KEY for non-interactive auth #67

Merged

4 tasks

wyattjoh mentioned this pull request Mar 31, 2026

test(e2e): add E2E test infrastructure and fixture projects #59

Merged

4 tasks

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from 37eb8ca to 8e483fd Compare March 31, 2026 15:06

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from be5ec70 to efc8c54 Compare March 31, 2026 15:06

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from efc8c54 to e9be78f Compare March 31, 2026 18:53

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from 8e483fd to a920dbe Compare March 31, 2026 18:53

wyattjoh mentioned this pull request Mar 31, 2026

fix(init): scaffolder and env pull fixes for all frameworks #84

Merged

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from e9be78f to 978c82d Compare March 31, 2026 19:02

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from a920dbe to 2931d00 Compare March 31, 2026 19:02

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 978c82d to a32b0de Compare March 31, 2026 19:04

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 09185c0 to 2828a73 Compare April 1, 2026 20:15

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from 8e57e2f to 05a9a08 Compare April 1, 2026 20:15

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 2828a73 to 5672569 Compare April 1, 2026 20:21

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from 05a9a08 to 5093deb Compare April 1, 2026 20:21

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 5672569 to 2a865ef Compare April 2, 2026 18:21

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch 2 times, most recently from df785fa to c720a34 Compare April 2, 2026 21:44

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 2a865ef to 8f56478 Compare April 2, 2026 21:44

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from c720a34 to 8e0811f Compare April 2, 2026 23:37

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 8f56478 to d0f376d Compare April 2, 2026 23:37

wyattjoh force-pushed the wyattjoh/e2e-tests-init branch from 8e0811f to 5a19141 Compare April 6, 2026 17:16

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch 3 times, most recently from 93ae2f2 to 2574d9a Compare April 6, 2026 19:02

Base automatically changed from wyattjoh/e2e-tests-init to main April 6, 2026 19:06

wyattjoh added 2 commits April 6, 2026 13:10

wyattjoh force-pushed the wyattjoh/test-infra-and-docs branch from 2574d9a to 5132a8d Compare April 6, 2026 19:11

wyattjoh added 5 commits April 6, 2026 13:17

wyattjoh changed the title ~~refactor(test): move test helpers and add documentation~~ test: harden e2e infra, colocate helpers, and add tooling configs Apr 6, 2026

wyattjoh marked this pull request as ready for review April 6, 2026 20:27

wyattjoh mentioned this pull request Apr 6, 2026

refactor(test): consolidate unit and e2e test runners #100

Merged

4 tasks

rafa-thayto approved these changes Apr 6, 2026

View reviewed changes

wyattjoh merged commit fc67a46 into main Apr 6, 2026
7 checks passed

wyattjoh deleted the wyattjoh/test-infra-and-docs branch April 6, 2026 21:32

coderabbitai Bot mentioned this pull request Apr 22, 2026

feat(auth): add OAuth refresh token session handling #205

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: harden e2e infra, colocate helpers, and add tooling configs#63

test: harden e2e infra, colocate helpers, and add tooling configs#63
wyattjoh merged 7 commits into
mainfrom
wyattjoh/test-infra-and-docs

wyattjoh commented Mar 26, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

wyattjoh commented Mar 31, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Mar 31, 2026 •

edited

Loading

Walkthrough

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

wyattjoh commented Mar 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test helper colocation

Claude rules and developer docs

E2E test infrastructure hardening

Dependabot

Test plan

Summary by CodeRabbit

Uh oh!

wyattjoh commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wyattjoh commented Mar 26, 2026 •

edited by coderabbitai Bot

Loading

wyattjoh commented Mar 31, 2026 •

edited

Loading

coderabbitai Bot commented Mar 31, 2026 •

edited

Loading