-
Notifications
You must be signed in to change notification settings - Fork 78
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compiling many Swift Testing tests is significantly slower than XCTest tests #502
Comments
Tracked internally as rdar://130478685 |
Thank you for this report @marcosgriselli! The example project and data are very helpful. I am investigating this along with @rintaro. |
Does this need to move over to swift-syntax or swift? |
At this point I do believe it should be transferred to the swiftlang/swift repo. I just tried doing that though, and it appears that you cannot transfer issues across organization boundaries, and due to the pending apple → swiftlang organization transition we'll need to wait until swift-testing makes the transition. We expect this to happen before too long. |
@stmontgomery happy to close and create the issue there unless you want the transfer to keep the issue history clean |
We can wait to transfer. The relevant people know about the issue already. |
I'm actively looking into this issue:
So far, these patches improves the |
Description
Compiling a large amount of Swift Testing tests is significantly slower than compiling the same amount of tests using XCTest on a clean build.
At 1000 tests in a single file building a test suite with Swift Testing is 24x slower than an almost identical suite using XCTest.
Here's a repo with a project to benchmark: https://github.com/marcosgriselli/swift-testing-performance. A few results from that project:
1000 Tests in a single file:
➜ swift-testing-performance git:(main) ✗ ./benchmark.py --test_count 1000 --runs 5 Benchmark 1: XCTest (1000 tests) Time (mean ± σ): 3.059 s ± 0.090 s [User: 0.658 s, System: 0.214 s] Range (min … max): 2.965 s … 3.183 s 5 runs Benchmark 2: SwiftTesting (1000 tests) Time (mean ± σ): 73.940 s ± 0.341 s [User: 0.747 s, System: 0.285 s] Range (min … max): 73.603 s … 74.427 s 5 runs Summary XCTest (1000 tests) ran 24.17 ± 0.72 times faster than SwiftTesting (1000 tests)
Performance significantly improves when splitting the tests into different files, still 10x slower than XCTest.
1000 tests split in 10 files:
I'm aware of the macros compilation build time issues outside of swift-syntax and that the team is working on addressing these, is there anything we can do in the meantime to reduce the impact while adopting Swift Testing?
Expected behavior
No response
Actual behavior
No response
Steps to reproduce
Run benchmarks on the shared repo to measure performance.
swift-testing version/commit hash
Xcode 16.0 (16A5171c)
Swift & OS version (output of
swift --version ; uname -a
)swift-driver version: 1.109.2 Apple Swift version 6.0 (swiftlang-6.0.0.3.300 clang-1600.0.20.10)
Target: arm64-apple-macosx14.0
Darwin Marcoss-MBP.attlocal.net 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64
The text was updated successfully, but these errors were encountered: