Skip to content

chore: make smoketests faster via test.concurrent, no sharding, 800% workers#773

Merged
jason-rl merged 18 commits into
mainfrom
jason/faster_smoketests
Apr 6, 2026
Merged

chore: make smoketests faster via test.concurrent, no sharding, 800% workers#773
jason-rl merged 18 commits into
mainfrom
jason/faster_smoketests

Conversation

@jason-rl

@jason-rl jason-rl commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

User description

⚠️ PR Title Must Follow Conventional Commits

Format: feat[optional scope]: <description>

Examples: feat: add new SDK method · feat(storage): support file uploads · feat!: breaking API change

Description

Motivation

Changes

Testing

  • Unit tests added
  • Integration tests added
  • Smoke Tests added/updated
  • Tested locally

Breaking Changes

Checklist

  • PR title follows Conventional Commits format (feat: or feat(scope):)
  • Documentation updated (if needed)
  • Breaking changes documented (if applicable)

CodeAnt-AI Description

Speed up smoke tests by running them in parallel

What Changed

  • Smoke tests now run as one parallel job instead of four shards, which shortens CI time and removes shard log collection
  • Many smoke tests now run at the same time, including devbox, blueprint, storage object, scenario, gateway, and SDK checks
  • Coverage smoke-test runs now fail normally when tests fail, so failures are surfaced directly instead of being handled as a separate follow-up step

Impact

✅ Shorter smoke test runs
✅ Faster CI feedback
✅ Clearer failed test reporting

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

@github-actions

github-actions Bot commented Apr 4, 2026

Copy link
Copy Markdown

❌ Object Smoke Tests Failed

Test Results

❌ Some smoke tests failed

Failed Tests:

  • �[22m�[1msmoketest: object-oriented gateway config › comprehensive gateway proxying tests › GET request - list devboxes�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.exec - basic execution�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.exec - CWD persistence across commands�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.exec - environment variable persistence�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.exec - combined CWD and environment persistence�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.execAsync - basic async execution�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.execAsync - stateful async execution�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.exec - sequential execution (queuing)�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.execAsync - sequential execution with queuing�[39m�[22m
  • �[22m�[1msmoketest: object-oriented devbox › named shell - stateful command execution › shell.exec - with streaming callbacks�[39m�[22m

Please fix the failing tests before checking coverage.

📋 View full test logs

@jason-rl jason-rl changed the title Make smoketests faster via test.concurrent Make smoketests faster via test.concurrent, no sharding, 400% workers Apr 4, 2026
@jason-rl jason-rl changed the title Make smoketests faster via test.concurrent, no sharding, 400% workers chore: make smoketests faster via test.concurrent, no sharding, 400% workers Apr 4, 2026
@jason-rl
jason-rl requested review from alb-rl and dines-rl April 4, 2026 02:08
@jason-rl
jason-rl marked this pull request as ready for review April 4, 2026 02:09
@codeant-ai

codeant-ai Bot commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

CodeAnt AI is reviewing your PR.


Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@codeant-ai codeant-ai Bot added the size:L This PR changes 100-499 lines, ignoring generated files label Apr 4, 2026
Comment thread tests/smoketests/object-oriented/axon.test.ts Outdated
Comment thread jest.config.ts Outdated
],
// Add display name for smoke tests to make it clearer in output
...((runSmoketests || runBuiltPackageTest) && { displayName: 'Smoke Tests' }),
...(runSmoketests && { maxWorkers: 50 }),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Using a fixed maxWorkers: 50 for smoke tests can oversubscribe low-core CI/local machines and trigger worker spawn/resource exhaustion failures (for example spawn EAGAIN) or severe process thrashing. Use a CPU-relative value so worker count scales with available cores. [possible bug]

Severity Level: Major ⚠️
- ⚠️ Local `yarn test:smoke` can oversubscribe low-core machines.
- ⚠️ Heavy smoke suites may thrash or intermittently fail locally.
- ⚠️ Developer smoke-test feedback loop slowed or destabilized.
Suggested change
...(runSmoketests && { maxWorkers: 50 }),
...(runSmoketests && { maxWorkers: '50%' }),
Steps of Reproduction ✅
1. On a low-core developer machine (e.g., 2 CPUs), run `yarn test:smoke` from the
repository root, which executes the script defined in `package.json:21` as `"test:smoke":
"RUN_SMOKETESTS=1 jest --verbose tests/smoketests"`.

2. The Jest CLI starts without an explicit `--maxWorkers` flag, so it loads the default
configuration from `jest.config.ts:6-9`, where `runSmoketests` is set based on
`process.env['RUN_SMOKETESTS']` at `jest.config.ts:3`.

3. Because `RUN_SMOKETESTS=1` is set by the script, `runSmoketests` is `true` and the
spread at `jest.config.ts:38` applies, forcing `maxWorkers: 50` into the Jest
configuration for this run.

4. Jest now attempts to run the smoke test suite in `tests/smoketests/**` (triggered by
the script at `package.json:21`) with up to 50 workers on a machine with far fewer cores,
causing significant oversubscription; this can manifest as heavy CPU thrashing or, on
constrained environments with low process limits, worker-spawn failures such as `spawn
EAGAIN`, degrading or breaking local smoke-test runs.
Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** jest.config.ts
**Line:** 38:38
**Comment:**
	*Possible Bug: Using a fixed `maxWorkers: 50` for smoke tests can oversubscribe low-core CI/local machines and trigger worker spawn/resource exhaustion failures (for example `spawn EAGAIN`) or severe process thrashing. Use a CPU-relative value so worker count scales with available cores.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
👍 | 👎

Comment thread tests/smoketests/examples/examples.test.ts Outdated
Comment thread tests/smoketests/object-oriented/gateway-config.test.ts Outdated
Comment thread tests/smoketests/object-oriented/gateway-config.test.ts Outdated
Comment thread tests/smoketests/object-oriented/network-policy.test.ts Outdated
Comment thread tests/smoketests/object-oriented/network-policy.test.ts Outdated
Comment thread tests/smoketests/object-oriented/network-policy.test.ts Outdated
Comment thread tests/smoketests/object-oriented/network-policy.test.ts Outdated
Comment thread tests/smoketests/object-oriented/network-policy.test.ts Outdated
Comment thread tests/smoketests/object-oriented/devbox.test.ts Outdated
Comment thread tests/smoketests/object-oriented/devbox.test.ts Outdated
@codeant-ai

codeant-ai Bot commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

CodeAnt AI finished reviewing your PR.

@codeant-ai

codeant-ai Bot commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

CodeAnt AI is running Incremental review


Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@codeant-ai codeant-ai Bot added size:L This PR changes 100-499 lines, ignoring generated files and removed size:L This PR changes 100-499 lines, ignoring generated files labels Apr 6, 2026
@codeant-ai

codeant-ai Bot commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

CodeAnt AI Incremental review completed.

Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
jason-rl and others added 5 commits April 6, 2026 17:56
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
jason-rl and others added 7 commits April 6, 2026 17:57
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
Co-authored-by: codeant-ai[bot] <151821869+codeant-ai[bot]@users.noreply.github.com>
@jason-rl jason-rl changed the title chore: make smoketests faster via test.concurrent, no sharding, 400% workers chore: make smoketests faster via test.concurrent, no sharding, 800% workers Apr 6, 2026
@jason-rl
jason-rl merged commit d5d7a29 into main Apr 6, 2026
3 checks passed
@jason-rl
jason-rl deleted the jason/faster_smoketests branch April 6, 2026 18:07
@stainless-app stainless-app Bot mentioned this pull request Apr 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants