Allow running XCTest bundle with multiple comma-separated tests #268
Conversation
|
Great find @egorzhdan! I forgot that Xcode’s version of the tool supported this. |
| /// Run tests or test cases, printing results to stdout and exiting with | ||
| /// a non-0 return code if any tests failed. The names of tests or test cases | ||
| /// may be provided to only run a subset of them. | ||
| case run(selectedTestNames: [String]?) |
There was a problem hiding this comment.
I guess we should prefer to use a Set for this for efficiency when filtering during execution?
There was a problem hiding this comment.
We will iterate over this array in TestFiltering.selectedTestFilter anyway for mapping into SelectedTests, but using a Set in the filter itself is indeed faster. Fixed.
|
@swift-ci please test |
|
Just fixed the recently arisen conflicts, @stmontgomery could you please take a look? |
|
@swift-ci please test |
This improves consistency with macOS version of XCTest which supports such behavior
|
@swift-ci Please test |
|
The failure on Linux is probably unrelated to this PR. I don't think the changes here could have caused a SwiftPM build script error (the build finished successfully a few days ago and I didn't make any changes since then) |
|
@swift-ci test linux |
|
@swift-ci Please test |
|
Hi, could you please share any ETA for these changes to be included into the Swift toolchain on Swift.org? It's essential to have this fix to run separate tests and as I can see they are still not included into the latest DEV. |
What specific toolchain are you using? This has been in master since before the 5.3 branch so it should be included in recent snapshots. |
|
@stmontgomery Sorry, it was my mistake during testing. Found it, everything works with 5.3 branch. |
This improves consistency with macOS version of XCTest which supports such behavior.
Right now any tool which runs a subset of tests is required to invoke the XCTest bundle N times, which forces the tool to spawn a lot of processes, and part of the work will be performed over and over again. This is especially painful if any expensive initialization is required (such as starting a web server), since
class func setUp()will be called every time a new test method is called instead of once per test case.From user's point of view statistics in this scenario will be shown in a more human-friendly format as
Executed 100 tests, with 0 failuresinstead ofExecuted 1 tests, with 0 failuresprinted 100 times.