Run every slow e2e suite in CI, not just one - #78
Merged
Conversation
CI ran 3 of the 27 slow tests. The other 24 existed only on a maintainer's laptop, which meant three of the six supported runtimes — cloudflare, dbos-ts, inngest — had no automated proof at all. The fan_out regression fixed in #76 was caught by exactly one of those suites running locally; on CI alone it would have merged. The job's comment said the TS/Docker suites "stay opt-in (they need Node/Docker)". That premise was never checked: ubuntu-latest ships both. The real cost is minutes, and three runtimes' worth of live coverage is worth minutes. Three jobs now, split by toolchain rather than by whether anyone remembered: - **Python e2e** — every slow suite needing no external toolchain. Previously only test_mcp_e2e; now also park-on-auth, the DBOS Python runtime, the OAuth device flow, the MCP-trigger server and the empirical eval path. - **TypeScript e2e** — cloudflare, inngest, the TS agent loop, MCP over the TS SDK, and the cloudflare/inngest park-on-auth suites. - **DBOS-TS e2e (Docker)** — split out because the DBOS TypeScript SDK is Postgres-only, so these two start a throwaway container. npm's download cache is keyed on `_ts_common.py`, since each suite installs into its own emitted directory and there is no lockfile for setup-node to key on. `tests/test_ci_workflow.py` keeps this from rotting: it asks pytest which files carry slow tests and asserts each appears in a CI job, so adding a slow suite without wiring it up fails rather than silently going unrun. It also catches ci.yml referencing a test file that no longer exists. Verified by unwiring a suite and watching it fail with that suite named. Marking a suite slow is a statement about the toolchain it needs, not about whether it should run. 1172 fast pass; ruff, strict mypy, sanity-check clean. The new jobs are proven by this PR's own CI run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SBuSDL7Svm7iDzgmhTpDSW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI ran 3 of the 27 slow tests. The other 24 existed only on a maintainer's laptop — which meant three of the six supported runtimes (cloudflare, dbos-ts, inngest) had no automated proof at all.
The fan_out regression fixed in #76 was caught by exactly one of those suites, running locally. On CI alone it would have merged clean.
Why they were excluded
The job comment said the TS/Docker suites "stay opt-in (they need Node/Docker)". That premise was never checked —
ubuntu-latestships both. The real cost is minutes.Three jobs, split by toolchain
Split by what each suite needs rather than by whether anyone remembered it. The DBOS-TS pair is separate because the DBOS TypeScript SDK is Postgres-only (no SQLite parity with DBOS Python), so those tests start a throwaway container.
npm's download cache is keyed on
_ts_common.py— each suite installs into its own emitted directory, so there's no lockfile forsetup-nodeto key on.Keeping it from rotting
tests/test_ci_workflow.pyasks pytest which files carry slow tests and asserts each appears in a CI job. Adding a slow suite without wiring it up now fails instead of silently going unrun. It also catchesci.ymlpointing at a test file that no longer exists.Verified by unwiring a suite and watching it fail with that suite named — not by inspection.
This PR's own CI run is the proof the new jobs work. If a job proves flaky or slow on runners I'll adjust before merging.
🤖 Generated with Claude Code
https://claude.ai/code/session_01SBuSDL7Svm7iDzgmhTpDSW