[Tests] Add regression test suite for OS X and Linux

[modocache]

See the individual commit messages for details, and the instructions added to the README for an idea of how the test suite works.

This approach differs from the one suggested on the mailing list by @ddunbar, in which XCTest itself is used to test XCTest. I believe this approach is simpler to implement and maintain.

In addition, using XCTest to test itself is a dangerous proposition: one or more bugs in XCTest may cause regressions, which may then not be caught because of those same bugs.

[stepanhruda]

if I don't want to run the tests in Xcode, are steps 3, 4 & 5 needed?

[modocache]

Tests/test.py works by running each executable in the Tests/Fixtures/Products/ directory. Those need to be built before the script is run.

The SingleFailingTestCase Xcode target compiles Tests/Fixtures/Sources/SingleFailingTestCase/main.swift into an executable and installs it in the Products directory. It also ensures that SwiftXCTest.framework is linked to that executable properly.

So in order to run the test on OS X, adding a new Xcode target (steps 3 and 4) is necessary, because it is responsible for putting an executable in the Products directory. Adding the new target to the aggregate target (step 5) is necessary to eliminate the need to build the newly added target manually before running Tests/test.py.

Also, your comment reminded me: because the executables are compiled and placed in the Products directory every time the tests are run (whether on OS X or Linux), the contents of that directory should be added to the .gitignore. I'll amend that change to commit b52c686.

[stepanhruda]

Right, but since you can build the executables on Linux, isn't it possible to build them on OS X without adding them to *.xcodeproj.

(Just trying to understand this better, and perhaps wondering if those steps shouldn't be marked Xcode instead of OS X)

[modocache]

Yeah, no worries, thanks for all the feedback!! 💯

Right, but since you can build the executables on Linux, isn't it possible to build them on OS X without adding them to *.xcodeproj.

I actually would love to achieve all this without using Xcode projects, but I can't for the life of me figure out how to compile and link the SwiftXCTest.framework build products to the executables.

(Just trying to understand this better, and perhaps wondering if those steps shouldn't be marked Xcode instead of OS X)

They could be. In practice, however, the two are the same--the SwiftXCTest.framework build system is tied to Xcode on OS X at the moment. As far as I can tell, it's not a Swift package, and is not meant to be built using llbuild--but correct me if I'm wrong here! 😛

[stepanhruda]

It would be nice to include a tested binary of XCTest in the repo to run the tests instead of unittest. It can be periodically updated as newer versions are tested and trusted.

The main reason for this: once there is a better reporting API available (literally anything that isn't plain printing to stdout), it will likely be annoying to call into it and make expectations from Python.

But since the build script is already in Python and no such API exists right now, this seems like a good first step.

[modocache]

But since the build script is already in Python and no such API exists right now, this seems like a good first step.

Yup! My main motivation for this test runner is to be able to verify the current behavior without first changing the production code. This is an integration test suite, one that will hopefully become less important as swift-corelibs-xctest is refactored to be more unit-testable.

The main reason for this: once there is a better reporting API available (literally anything that isn't plain printing to stdout), it will likely be annoying to call into it and make expectations from Python.

I actually don't think this is the case. Imagine testing a dot reporter:

/// Tests/Fixtures/Sources/DotReporterTests/main.swift

// .F..
// %exit-status: 1

#if os(Linux)
    import XCTest
#else
    import SwiftXCTest
#endif

class OneFailingThreePassingTestCase: XCTestCase {
    // ...four test methods here. The second one fails.
}

XCTRegisterReporter(DotReporter()) // Imagine this is the reporter API
XCTMain([OneFailingThreePassingTestCase()])

And there you have it! This is all you need to test on Linux. (You'd have to add an Xcode target to get this tested on OS X.)

[stepanhruda]

I see, in that case the Python suite doesn't ever need to go away. It can instead stay very thin and only verify that the bare minimum reports back correctly, the rest of the module can be self-tested.

In self-tests only methods from the bare minimum can be used (no fancy matchers etc., nothing untested by the integration suite). I am not sure how to nicely systematically enforce this (splitting into 2 modules would work, but that's unacceptable for clients).

[ddunbar]

If we are going to use a non-XCTest based solution, then IMHO we should use lit, which would make this project consistent with LLVM/Clang/Swift/llbuild.

lit has a lot of useful features for doing this style of testing, including integration of other testing systems, and we have a lot of institutional knowledge about how to use it to solve testing problems.

It is also pretty conceptually similar to this approach, we shouldn't roll another custom alternative.

Of course I am biased. :)

[modocache]

I'd love to use lit! I hadn't realized it could be used outside of testing LLVM. I'll look into using it instead--it'll be a great opportunity for me to learn more about it.

Let's continue the discussion on the overall approach on the mailing list. I feel that a non-XCTest-based approach is the most effective way to catch regressions without modifying the production code in XCTest itself. It'd also provide value over time, even after we add a unit test suite. Would love to hear more debate on the topic, however.

In any case, expect a pull request for a lit-based approach in the coming days.

[ddunbar]

Sounds great, thanks Brian. I agree its worth discussing this on the mailing list to see what the core XCTest owners think.