Build performance issue of swift-testing

[Kyle-Ye]

Description

Before migrating to use "swift-testing", one of my package's average CI time is as the following

XCTest	macOS 13 + Swift 5.9.2	Ubuntu + Swift 5.9.2
Swift Test(Debug+Covage)	45s	20s
Swift Test(Release)	34s	27s

Most the job will do its job less in 1 min.

However after fully migrate all my test case to swift-testing, the CI time exploded especially with Release configuration.

Expected behavior

It is expected the build time will increase due to the usage of swift macro. And 2~3 min looks normal and acceptable.

Increased to more than 10 min is definitely means something is wrong here IMO.

And my question is the following:

1. Is this a known issue?
2. Is this expected for Swift 5.9 (or 5.10) client only and the issue will not happen for Swift Snapshots toolchain?
3. Is there a workaround for this currently?

Actual behavior

Testing	macOS 13 + Swift 5.9.2	Ubuntu + Swift 5.9.2
Swift Test(Debug+Covage)	4min3s	3min24s
Swift Test(Release)	11min19s	12min37s

Steps to reproduce

No response

swift-testing version/commit hash

swift-testing 0.2

Swift & OS version (output of swift --version ; uname -a)

Swift 5.9.2

[grynspan]

Thanks for opening this issue. swift-testing does not support Swift 5.9, and Swift 5.10 support is temporary, so we'd want to see what your results look like in Swift 5.11 (i.e. using a main development toolchain.)

Is it possible the times you're seeing include the time to build the testing library itself? XCTest is built into Xcode and the Swift toolchain, so it doesn't need to be built with your package or test target; the testing library is an experimental Swift package and must be built from source along with swift-syntax, and those packages can take a noticeable amount of time to build before your package can start building.

If the times above do not include the time to build swift-syntax or swift-testing, then I think we need to confirm the issue persists in Swift 5.11. Since we don't have access to your test code, we can't reproduce the results you're seeing locally and are not able to profile the build process to see where time is being lost. Are you able to share access to your repository? If not, are you able to provide a reduced test function that is significantly slower to build with swift-testing than with XCTest?

[Kyle-Ye]

Is it possible the times you're seeing include the time to build the testing library itself?

Yes, I highly suspect it too.

XCTest is built into Xcode and the Swift toolchain, so it doesn't need to be built with your package or test target

Is there a Xcode Beta version available for us to try it? (I do not know whether the latest public beta version - Xcode 15.2 Beta is capable of it.)

Another thing I'd like to complain is that I can't run a single test with Xcode currently. I can only choose to run AllTests to literally run all tests. Hope we can get the Xcode support for it soon.

final class AllTests: XCTestCase {
    func testAll() async {
        await XCTestScaffold.runAllTests(hostedBy: self)
    }
}

Or we can only use the nightly Swift Toolchain via swift test to test it?

Anyway I'll give it a try with the nightly toolchain locally and report back the result later.

[grynspan]

swift-testing is an experimental open-source package. If you'd like to see Apple add support for it to Xcode, I would encourage you to file feedback with them. 😁

swift test has built-in support for swift-testing using a main-branch development toolchain when you pass the --enable-experimental-swift-testing option. The 5.9 and 5.10 toolchains will not be updated to add this support, so you'll need a main-branch development toolchain to try it out.

[Kyle-Ye]

swift test has built-in support for swift-testing using a main-branch development toolchain when you pass the --enable-experimental-swift-testing option.

Does this only supported on macOS or I have missed something else here?

I have tried it with the latest Swift nightly toolchain(2023-12-21-a) with Ubuntu 22.04. Still can't find Testing framework.

Note

On my test branch for Swift 5.11, I have commented the following line on my Pacakge.swift file as I thought they are no longer needed on Swift 5.11 toolchain
.package(url: "https://github.com/apple/swift-testing", from: "0.2.0")
.product(name: "Testing", package: "swift-testing")

> uname -a       
Linux ubuntu-2204 6.5.13-orbstack-00121-ge428743e4e98 apple/swift-testing#1 SMP Wed Dec 27 10:22:46 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

> swift --version                               
Swift version 5.11-dev (LLVM 13124099c3f0229, Swift d6871edc839adec)
Target: aarch64-unknown-linux-gnu

> swift test --enable-experimental-swift-testing
xx.swift:9:8: error: no such module 'Testing'
import Testing
       ^
error: fatalError

Update

XCTest is built into Xcode and the Swift toolchain, so it doesn't need to be built with your package or test target; the testing library is an experimental Swift package and must be built from source along with swift-syntax, and those packages can take a noticeable amount of time to build before your package can start building.

I thought swift-testing is already bundled into the toolchain just like XCTest. I added the dependency back and then have a successful build on 5.11. But the test release build time issue still remains the same probably due to the build time on swift-testing and swift-syntax.

[Kyle-Ye]

Time statistics on my local machine

> time swift test --enable-experimental-swift-testing
swift test --enable-experimental-swift-testing  171.38s user 9.40s system 238% cpu 1:15.85 total
> time swift test -c release --enable-experimental-swift-testing
swift test -c release --enable-experimental-swift-testing  527.19s user 10.92s system 150% cpu 5:56.43 total

Note

1. Not comparable with the initial data I provide since they are running on CI machine
2. Most of the release build is stuck in something on SwiftSyntax509 eg. [10/13] Compiling SwiftSyntax509 Empty.swift

The SwiftSyntax issue may exist for all macro package. But it is especially fatal for testing frameworks like swift-testing.

[grynspan]

swift-testing is not built into any toolchains. It is an experimental package. Sorry if my previous comments were confusing!

If you're seeing most of the time spent in swift-syntax, we can send this issue over to that repository and have our colleagues look at it. I don't know what's in Empty.swift that would cause long build times.

[ahoppen]

Empty.swift is a red herring here. SwiftSyntax509 depends on SwiftSyntax and you’re most likely seeing the compile times of the SwiftSyntax module while SwiftPM displays Empty.swift.

The underlying issue here is that swift-syntax needs to be built to use the macros in swift-testing and that release builds of swift-syntax take a while (#1194).

[ahoppen]

Tracked in Apple’s issue tracker as rdar://120672412

[Kyle-Ye]

The time has greatly improved with the new macOS 14 runner + Xcode 15.3 + Swift 5.10

It may be a combination of the upgrade of the GitHub macOS 14 image to a 4-core CPU and Swift-syntax optimization.

Testing	macOS 13 + Swift 5.9.2 + swift-testing 0.3.0	macOS 14 + Swift 5.10 + swift-testing 0.6.0
Swift Test(Debug+Covage)	4min3s	1min24s
Swift Test(Release)	11min19s	3min21s

Although there is still a gap with the original XCTest branch, the current time result has fallen back to our acceptable range.