Build and run Swift Testing's own tests in Swift CI

[finagolfin]

Description

Both the main CI scripts do not run the Testing tests yet, which mean it is only tested by some github workflows with the prebuilt snapshot toolchains in that repo. When a CI run or toolchain build is triggered here, the Testing tests are not run on these compiler or corelibs changes.

Additional information

I'm currently building the Oct. 2 trunk snapshot of the Swift toolchain natively on my Android phone using build-script and unlike most of the other swiftlang repos, the Testing tests are not run. Instead, I have been building the full toolchain first then manually building Testing again and running its tests using the new SwiftPM, but it would be better if this were scripted.

@grynspan and @stmontgomery, let us know if there are any technical obstacles to setting this up.

[grynspan]

Swift Testing's own tests need to be run after building Swift Test as a package because if you try to use the copy in the toolchain, it will bump up against the lack of testability. So I'm not sure if you'll be able to configure this to do what you want.

This is a problem specific to Swift Testing because it's testing itself, which is not something other packages/libraries/tools need to do. Note that test coverage in swift-corelibs-xctest is much lower.

[finagolfin]

You could just build Testing a second time as a SwiftPM package after SwiftPM is built and test that? That's what non-Darwin Foundation now does, #74594.

[grynspan]

Oh, sure. It's just that you won't be able to test the copy in the toolchain directly, so the signal we get back won't be all that much better than what we do in the Swift Testing repo.

It may be helpful in addition to doing that to set up a smoke test suite into the toolchain build that uses Swift Testing for a few basic operations, thus confirming that the macros etc. are working with the toolchain copy.

[finagolfin]

It won't be the exact library from the toolchain, but it will be the same source commit, which has proven good enough for non-Darwin Foundation to test on this CI that way already.

As for checking a few basic operations with the toolchain, I suppose you could always add more integration tests.

But the main advantages of adding running the Testing tests to these CI build scripts are that arbitrary pulls and commits from this compiler and the corelibs repos will be run through the Testing tests also while in development, rather than only later in the snapshot builds, and those using these official build scripts to produce toolchains, whether the FreeBSD toolchain or the various linux distros, won't be doing so without running the Testing tests also.

[stmontgomery]

Thanks for filing this, @finagolfin. The topic has come up before but I don't believe we have an issue tracking it. It's something I intend for us to begin working on in the near future.

The main benefit of doing this IMO would be to protect Swift Testing from regressions in the compiler.