Skip to content

test(e2e): post-deploy MVP smoke suite (#126)#134

Merged
ssilvius merged 3 commits into
mainfrom
feat/126-post-deploy-smoke
May 16, 2026
Merged

test(e2e): post-deploy MVP smoke suite (#126)#134
ssilvius merged 3 commits into
mainfrom
feat/126-post-deploy-smoke

Conversation

@ssilvius

@ssilvius ssilvius commented May 16, 2026

Copy link
Copy Markdown
Contributor

Closes #126. Last child of epic #129. Stacked on #128/#130/#131/#132/#133.

What

Post-deploy smoke suite that runs against a real deployed worker, never local dev.

playwright.config.ts

  • testDir: tests/e2e (new convention; the old root-glob caught nothing)
  • baseURL: process.env.SMOKE_BASE_URL ?? \"https://rafters.studio\" -- override for staging
  • No webServer -- this suite never spins up local dev
  • Single smoke project, trace on first retry

tests/e2e/mvp-smoke.e2e.ts

Test Cost Notes
/api/health returns 200 OK none
color path validation rejects junk none 400
color adhoc=true returns full shape none math only, no AI
color sync=true real Sonnet call @cost Random OKLCH per run to avoid gateway cache, 60s timeout
GitHub sign-in redirects to github.com none 30x
OTP send-verification-otp @cost Real Resend send
Polar webhook rejects invalid signature none 4xx
Inbound mail manual test.skip with note -- trigger from CF Email Routing dashboard

@cost tests opt-in only:

pnpm test:e2e --grep @cost

Default run (5 tests) costs nothing.

Verification

  • pnpm exec playwright test --list --grep-invert @cost: 5 tests discovered
  • pnpm test (unit suite): 27 passed / 5 skipped (.e2e.ts doesn't match vitest's *.test.ts glob)
  • Root @playwright/test devDep added so pnpm test:e2e resolves the binary

Out of scope

  • CI workflow gating deploy on smoke (operator step)
  • Staging environment (single env for MVP; baseURL override leaves the door open)
  • Inbound mail programmatic trigger -- no public API, manual via CF Email Routing dashboard

Test plan

  • pnpm test:e2e --list --grep-invert @cost discovers 5 tests
  • pnpm test unit suite still 27 / 5 skip (no overlap)
  • After all 6 PRs land + secrets provisioned + wrangler deploy: pnpm test:e2e against prod -> 5 default tests green
  • --grep @cost run after deploy verifies real Sonnet + Resend paths

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

@ssilvius ssilvius mentioned this pull request May 16, 2026
Open
10 tasks
@ssilvius ssilvius force-pushed the feat/126-post-deploy-smoke branch 2 times, most recently from c61083b to 016fba4 Compare May 16, 2026 06:27
ssilvius added a commit that referenced this pull request May 16, 2026
@ssilvius @claude
chore: regen lockfile + bump @polar-sh/sdk for better-auth peer (#134)
2a1a84d 
Fresh pnpm install regenerated pnpm-lock.yaml from package.json + the
extended workspace + catalog state. Net effect:

- lockfile shrunk ~1900 lines (cleaner resolution after the cascade
  rebases through #128 -> #122 -> #123 -> #124 -> #125 -> #126)
- vitest catalog 4.1.0 -> 4.1.6 (latest patch, no source change needed)
- zod catalog 4.3.6 -> 4.4.3 (latest patch, no source change needed)
- @polar-sh/sdk bumped ^0.46.6 -> ^0.47.0 to satisfy the peer that
  @polar-sh/better-auth@1.8.4 wants. Without the bump pnpm peers check
  fails with "unmet peer @polar-sh/sdk Wanted ^0.47.0".

Verification
- pnpm peers check: no issues
- pnpm test: 27 passed, 5 skipped (unchanged)
- pnpm build (wrangler dry-run): clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ssilvius and others added 3 commits May 15, 2026 23:45
@ssilvius @claude
test(e2e): post-deploy MVP smoke suite (#126)
e3e5653 
playwright.config.ts
- testDir tests/e2e (new convention; the old root-glob caught nothing
  because the project has no .e2e.ts files yet).
- baseURL: SMOKE_BASE_URL env override, default https://rafters.studio.
  Tests run against a real deployed worker, never local dev.
- webServer block removed -- this suite never spins up local dev.
- Single "smoke" project. Trace on first retry only.

tests/e2e/mvp-smoke.e2e.ts (new)
- /api/health: 200 + {status: "ok"}
- /api/color/:oklch:
  - path validation: 400 on non-OKLCH input
  - adhoc=true: full ColorValue with 11-position scale + harmonies
    (math only, no AI cost)
  - @cost sync=true: real Anthropic call via gateway, asserts
    intelligence.label + reasoning. Random OKLCH per run to avoid the
    24h gateway cache. 60s timeout.
- /api/auth:
  - github sign-in: 30x redirect to github.com (no cost)
  - @cost OTP send-verification-otp: real Resend send, asserts <500
- /api/auth/polar/webhook: invalid signature rejected (4xx, no cost)
- inbound mail: test.skip with note -- triggered manually via CF Email
  Routing dashboard test send; no programmatic way to fire one.

@cost-tagged tests opt-in only:
  pnpm test:e2e --grep @cost

Default run (5 tests) covers: health, color path validation, color
adhoc, GitHub redirect, polar webhook rejection. Zero external charges.

Verification
- pnpm exec playwright test --list --grep-invert @cost: 5 tests
- pnpm test (vitest unit suite, *.test.ts): 27 passed, 5 skipped
  (.e2e.ts files do not match vitest's *.test.ts glob)
- root @playwright/test devDep added so pnpm test:e2e resolves the
  binary from root (was only present in apps/web/)

Out of scope
- CI workflow gating deploy on smoke (operator step)
- Staging environment (single env for MVP -- baseURL override leaves
  the door open)
- Inbound mail programmatic trigger (no public API; manual)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ssilvius @claude
ci: smoke workflow runs post-deploy mvp suite on push to main (#126)
4bf0407 
Was marked out-of-scope in the original #126 issue body, but it is the
operational link that closes the deploy loop -- without it, operators
have to remember to run pnpm test:e2e by hand after every deploy.

.github/workflows/smoke.yml
- Triggers: push to main, plus manual workflow_dispatch with an
  optional include_cost boolean for the @cost-tagged tests
  (Anthropic + Resend charges).
- 5-minute timeout. Default run skips @cost via --grep-invert.
- Uploads the playwright-report artifact on failure (7-day retention).
- Uses INCLUDE_COST as an env var rather than ${{ inputs.* }} inline,
  per the workflow-injection guidance.

Sequencing
- This workflow runs against the deployed production URL
  (default https://rafters.studio per playwright.config.ts), so it
  fires AFTER wrangler deploy. Order on push to main:
  1. CI workflow runs typecheck + unit tests
  2. (Operator) wrangler deploy from local
  3. Smoke workflow fires (also on push) and exercises the deployed
     worker
- If a smoke run fires before the deploy lands, it will fail against
  the prior version. That is correct behavior -- a deploy hasn't
  happened, so the new code isn't live, so the smoke should report
  the gap.

Out of scope
- Auto-deploy from CI (operator-controlled)
- Slack / PagerDuty notification on failure
- Smoke against staging (single env for MVP)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ssilvius @claude
chore: regen lockfile + bump @polar-sh/sdk for better-auth peer (#134)
ac762b1 
Fresh pnpm install regenerated pnpm-lock.yaml from package.json + the
extended workspace + catalog state. Net effect:

- lockfile shrunk ~1900 lines (cleaner resolution after the cascade
  rebases through #128 -> #122 -> #123 -> #124 -> #125 -> #126)
- vitest catalog 4.1.0 -> 4.1.6 (latest patch, no source change needed)
- zod catalog 4.3.6 -> 4.4.3 (latest patch, no source change needed)
- @polar-sh/sdk bumped ^0.46.6 -> ^0.47.0 to satisfy the peer that
  @polar-sh/better-auth@1.8.4 wants. Without the bump pnpm peers check
  fails with "unmet peer @polar-sh/sdk Wanted ^0.47.0".

Verification
- pnpm peers check: no issues
- pnpm test: 27 passed, 5 skipped (unchanged)
- pnpm build (wrangler dry-run): clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ssilvius ssilvius force-pushed the feat/126-post-deploy-smoke branch from 2a1a84d to ac762b1 Compare May 16, 2026 06:45
@ssilvius ssilvius merged commit 93a5171 into main May 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

test(e2e): post-deploy smoke tests for auth, color, polar, inbound mail

1 participant

@ssilvius