Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ You don't need to worry about asynchronous nature of NodeJS or about various API
- Also plays nice with TypeScript.
- </> Smart locators: use names, labels, matching text, CSS or XPath to locate elements.
- 🌐 Interactive debugging shell: pause test at any point and try different commands in a browser.
- ⚡ **Parallel testing** with dynamic test pooling for optimal load balancing and performance.
- Easily create tests, pageobjects, stepobjects with CLI generators.

## Installation
Expand Down
1 change: 1 addition & 0 deletions bin/codecept.js
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ program
.option('-i, --invert', 'inverts --grep matches')
.option('-o, --override [value]', 'override current config options')
.option('--suites', 'parallel execution of suites not single tests')
.option('--by <strategy>', 'test distribution strategy: "test" (pre-assign individual tests), "suite" (pre-assign test suites), or "pool" (dynamic distribution for optimal load balancing, recommended)')
.option(commandFlags.debug.flag, commandFlags.debug.description)
.option(commandFlags.verbose.flag, commandFlags.verbose.description)
.option('--features', 'run only *.feature files and skip tests')
Expand Down
24 changes: 22 additions & 2 deletions docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,12 +102,32 @@ DEBUG=codeceptjs:* npx codeceptjs run

## Run Workers

Run tests in parallel threads.
Run tests in parallel threads. CodeceptJS supports different distribution strategies for optimal performance.

```
```bash
# Run with 3 workers using default strategy (pre-assign tests)
npx codeceptjs run-workers 3

# Run with pool mode for dynamic test distribution (recommended)
npx codeceptjs run-workers 3 --by pool

# Run with suite distribution
npx codeceptjs run-workers 3 --by suite

# Pool mode with filtering
npx codeceptjs run-workers 4 --by pool --grep "@smoke"
```

**Test Distribution Strategies:**

- `--by test` (default): Pre-assigns individual tests to workers
- `--by suite`: Pre-assigns entire test suites to workers
- `--by pool`: Dynamic distribution for optimal load balancing (recommended for best performance)

The pool mode provides the best load balancing by maintaining tests in a shared pool and distributing them dynamically as workers become available. This prevents workers from sitting idle and ensures optimal CPU utilization, especially when tests have varying execution times.

See [Parallel Execution](/parallel) documentation for more details.

## Run Rerun <Badge text="Since 3.3.6" type="warning"/>

Run tests multiple times to detect and fix flaky tests.
Expand Down
82 changes: 82 additions & 0 deletions docs/parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,88 @@ By default, the tests are assigned one by one to the available workers this may
npx codeceptjs run-workers --suites 2
```

### Test Distribution Strategies

CodeceptJS supports three different strategies for distributing tests across workers:

#### Default Strategy (`--by test`)
Tests are pre-assigned to workers at startup, distributing them evenly across all workers. Each worker gets a predetermined set of tests to run.

```sh
npx codeceptjs run-workers 3 --by test
```

#### Suite Strategy (`--by suite`)
Test suites are pre-assigned to workers, with all tests in a suite running on the same worker. This ensures better test isolation but may lead to uneven load distribution.

```sh
npx codeceptjs run-workers 3 --by suite
```

#### Pool Strategy (`--by pool`) - **Recommended for optimal performance**
Tests are maintained in a shared pool and distributed dynamically to workers as they become available. This provides the best load balancing and resource utilization.

```sh
npx codeceptjs run-workers 3 --by pool
```

## Dynamic Test Pooling Mode

The pool mode enables dynamic test distribution for improved worker load balancing. Instead of pre-assigning tests to workers at startup, tests are stored in a shared pool and distributed on-demand as workers become available.

### Benefits of Pool Mode

* **Better load balancing**: Workers never sit idle while others are still running long tests
* **Improved performance**: Especially beneficial when tests have varying execution times
* **Optimal resource utilization**: All CPU cores stay busy until the entire test suite is complete
* **Automatic scaling**: Workers continuously process tests until the pool is empty

### When to Use Pool Mode

Pool mode is particularly effective in these scenarios:

* **Uneven test execution times**: When some tests take significantly longer than others
* **Large test suites**: With hundreds or thousands of tests where load balancing matters
* **Mixed test types**: When combining unit tests, integration tests, and end-to-end tests
* **CI/CD pipelines**: For consistent and predictable test execution times

### Usage Examples

```bash
# Basic pool mode with 4 workers
npx codeceptjs run-workers 4 --by pool

# Pool mode with grep filtering
npx codeceptjs run-workers 3 --by pool --grep "@smoke"

# Pool mode in debug mode
npx codeceptjs run-workers 2 --by pool --debug

# Pool mode with specific configuration
npx codeceptjs run-workers 3 --by pool -c codecept.conf.js
```

### How Pool Mode Works

1. **Pool Creation**: All tests are collected into a shared pool of test identifiers
2. **Worker Initialization**: The specified number of workers are spawned
3. **Dynamic Assignment**: Workers request tests from the pool when they're ready
4. **Continuous Processing**: Each worker runs one test, then immediately requests the next
5. **Automatic Completion**: Workers exit when the pool is empty and no more tests remain

### Performance Comparison

```bash
# Traditional mode - tests pre-assigned, some workers may finish early
npx codeceptjs run-workers 3 --by test # ✓ Good for uniform test times

# Suite mode - entire suites assigned to workers
npx codeceptjs run-workers 3 --by suite # ✓ Good for test isolation

# Pool mode - tests distributed dynamically
npx codeceptjs run-workers 3 --by pool # ✓ Best for mixed test execution times
```

## Test stats with Parallel Execution by Workers

```js
Expand Down
17 changes: 16 additions & 1 deletion lib/command/run-workers.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,22 @@ module.exports = async function (workerCount, selectedRuns, options) {

const { config: testConfig, override = '' } = options
const overrideConfigs = tryOrDefault(() => JSON.parse(override), {})
const by = options.suites ? 'suite' : 'test'

// Determine test split strategy
let by = 'test' // default
if (options.by) {
// Explicit --by option takes precedence
by = options.by
} else if (options.suites) {
// Legacy --suites option
by = 'suite'
}

// Validate the by option
const validStrategies = ['test', 'suite', 'pool']
if (!validStrategies.includes(by)) {
throw new Error(`Invalid --by strategy: ${by}. Valid options are: ${validStrategies.join(', ')}`)
}
delete options.parent
const config = {
by,
Expand Down
Loading