Skip to content

Improve integration test speed#1489

Merged
dcantah merged 1 commit intoapple:mainfrom
noah-thor:make-integration-fast
Apr 30, 2026
Merged

Improve integration test speed#1489
dcantah merged 1 commit intoapple:mainfrom
noah-thor:make-integration-fast

Conversation

@noah-thor
Copy link
Copy Markdown
Contributor

This improves the integration test suite speed by running all integration tests in a single swift test invocation. We maintain the one @Suite at a time behavior by introducing a new test trait that uses a single mutex to limit inter-suite parallelism.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

Motivation and Context

This is meant to improve the CI/CD experience by reducing execution time.

Testing

  • Tested locally
  • Added/updated tests
  • Added/updated docs

@noah-thor
Copy link
Copy Markdown
Contributor Author

Local execution time saw a decrease of ~60s

hyperfine \
                                                                               --min-runs 3 \
                                                                               --prepare 'make clean; make all' \
                                                                               --export-json integ-bench.json \
                                                                               --command-name 'per-suite (make integration)' \
                                                                                 'make integration' \
                                                                               --command-name 'single-process (make integration-fast)' \
                                                                                 'make integration-fast'
Benchmark 1: per-suite (make integration)
  Time (mean ± σ):     636.377 s ± 28.689 s    [User: 134.902 s, System: 76.158 s]
  Range (min … max):   617.463 s … 669.387 s    3 runs

Benchmark 2: single-process (make integration-fast)
  Time (mean ± σ):     582.090 s ± 24.498 s    [User: 115.609 s, System: 64.499 s]
  Range (min … max):   553.825 s … 597.187 s    3 runs

This improves the integration test suite speed by running all
integration tests in a single `swift test` invocation. We maintain the
one `@Suite` at a time behavior by introducing a new test trait that
uses a single mutex to limit inter-suite parallelism.
@noah-thor noah-thor force-pushed the make-integration-fast branch from 4b8a8de to 18ed141 Compare April 30, 2026 22:07
@dcantah dcantah merged commit a141002 into apple:main Apr 30, 2026
3 checks passed
katiewasnothere added a commit that referenced this pull request May 1, 2026
## Motivation and Context
The `SuiteGate` actor introduced for the serial test trait in
#1489 could be simplified using
containerization's
[AsyncLock](https://github.com/apple/containerization/blob/f2c42402e744df992fecb84eb3a82935c6fa01d3/Sources/ContainerizationExtras/AsyncLock.swift#L23).

## Testing
- [ ] Tested locally

Note: working on A/B performance testing to validate this approach does
not significantly impact test run performance.

Signed-off-by: Kathryn Baldauf <k_baldauf@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants