Permalink
Find file
292 lines (207 sloc) 9.58 KB

Swift Testing

Introduction

Testing is an essential part of modern software development. Tight integration of testing into the Swift Package Manager will help ensure a stable and reliable packaging ecosystem.

SE Review Link, Second Review

Proposed Solution

We propose to extend our conventional package directory layout to accommodate test modules. Any subdirectory of the package root directory named "Tests" or any subdirectory of an existing module directory named "Tests" will comprise a test module. For example:

Package
├── Sources
│   └── Foo
│       └──Foo.swift
└── Tests
    └── Foo
        └── Test.swift

Or:

Package
└── Sources
    ├── Foo.swift
    └── Tests
        └── Test.swift

Or, for projects with a single module:

Package
├── Sources
│   └── Foo.swift
└── Tests
    └── TestFoo.swift

The filename: TestFoo.swift is arbitrary.

In the examples above a test case is created for the module Foo based on the sources in the relevant subdirectories.

A test-module is created per subdirectory of Tests, so:

Package
├── Sources
│   └── Foo
│       └──Foo.swift
└── Tests
    └── Foo
        └── Test.swift
    └── Bar
        └── Test.swift

Would create two test-modules. The modules in this example may test different aspects of the module Foo, it is entirely up to the package author.

Additionally, we propose that building a module also builds that module's corresponding tests. Although this would result in slightly increased build times, we believe that tests are important enough to justify this (one might even consider slow building tests to be a code smell). We would prefer to go even further by executing the tests each time a module is built as well, but we understand that this would impede debug cycles.

As an exception, when packages are built in release mode we will not build tests because for release builds we should not enable testability. However, considering the need for release-mode testing this will be a future direction.

This feature is controversial; many are worried always building tests will slow debug cycles. We understand these concerns but think knowing quickly tests have been broken by code changes is important. However we will provide an command line option to not build tests and will be open to reversing this decision in future should it be proven unwise.

Command-Line Interface

We propose the following syntax to execute all tests (though not the tests of any dependent packages):

$ swift test

The command line should accept the names of specific test cases to run:

swift test TestModule.FooTestCase

Or specific tests:

swift test TestModule.FooTestCase.test1

In the future we would like to support running specific kinds of tests:

swift test --kind=performance

swift test would forward any other arguments to the underlying testing framework and it would decide how to interpret them.


Sometimes test sources cannot compile and fixing them is not the most pressing priority. Thus it will be possible to skip building tests with an additional flag:

swift build --without-tests

It is desirable to sometimes specify to only build specific tests, the command line for this will fall out of future work that allows specification of targets that swift build should specifically build in isolation.


Users should be able to run the tests of all their dependencies. This is not the default behavior, but a flag will be provided.

Command Output

Executing a test from the terminal will produce user-readable output. This should incorporate colorization and other formatting similar to other testing tools to indicate the success and failure of different tests. For example:

$ swift test --output module
Running tests for PackageX (x/100)
.........x.....x...................

Completed
Elapsed time: 0.2s

98 Success
 2 Failure
 1 Warning

FAILURE: Tests/TestsA.swift:24 testFoo()
XCTAssertTrue expected true, got false

FAILURE: Tests/TestsB.swift:10 testBar()
XCTAssertEqual

WARNING: Tests/TestsC.swift:1
"Some Warning"

An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.

Running swift test will firstly trigger a build. We feel this is the most expected result considering tests must be built before they can be run and almost all other tools build before running tests. However we will provide a flag to not build first.

Test-only Dependencies

There is already a mechanism to specify test-only dependencies. It is very basic, but a new proposal should be made for more advanced Package.swift functionality.

This proposal also does not cover the need for utility code, ie. a module that is built for tests to consume that is provided as part of a package and is not desired to be an external package. This is something we would like to add as part of a future proposal.

Test-target configuration

This proposal does not allow a test-module to have module dependencies from its own package, and thus there is no provided mechanism to specify or configure tests in the Package.swift file.

This will be added, but as part of a broader proposal for the future of the Package.swift file.

Automatic Dependency Determination

Testing is important and it is important to make the barrier to testing as minimal as possible. Thus, by analyzing the names of test targets, we will automatically determine the most likely dependency of that test and accommodate accordingly. For example, a test for "Foo" will depend on compilation of the library target Foo. Any additional dependencies or dependencies that could not be automatically determined would need to be specified in a package manifest separately.

Debug / Release Configuration

Although tests built in debug configuration are generally run against modules also build in debug configuration, it is sometimes necessary to specify the build configuration for tests separately. It is also sometimes necessary to explicitly specify this information for every build, such as when building in a release configuration to execute performance tests. We would like to eventually support these use cases, however this will not be present in the initial implementation of this feature.

Testability

Swift can build modules with "testability", which allows tests to access entities with internal access control. Because it would be tedious for users to specify this requirement for tests, we intend to build debug builds with testability by default.

It is desirable that modules that are built for testing can identify this fact in their sources. Thus at a future time we will provide a define.

Test Frameworks

Initially, the Swift Package Manager will use XCTest as its underlying test framework.

However, testing is an evolving art form, so we'd like to support other approaches that might allow frameworks other than XCTest to be supported by the package manager. We expect that such an implementation would take the form of a Swift protocol that the package manager defines, which other testing frameworks can adopt.

Impact On Existing Code

Current releases of the package manager already exclude directories named "Tests" from target-determination. Directories named FooTests are not excluded, but as it stands this is a cause of compile failure, so in fact these changes will positively impact existing code.

Alternatives Considered

We considered supporting the following layout:

Package
└── Sources
│   └── Foo.swift
└── FooTests
    └── Test.swift

This was considered because of the vast number of existing Xcode projects out there that are laid out in the fashion. However it was decided that the rules for these layouts are inconsistent with our existing simple set. When users experience unexpected consequences of layouts with our convention approach it can be confusing and tricky to diagnose, so we should instead submit another proposal in the future that allows easy configuration of targets in Package.swift.


We considered decoupling testing from SwiftPM altogether.

However, since tests must be built and dependencies of tests must be managed complete decoupling is not possible. The coupling will be minimal, with a separate library and executable for tests, this is about as far as we think it is prudent to go.


We considered not baking in support for XCTest and only having a protocol for testing.

We would like to get testing up to speed as soon as possible. Using XCTest allows this to occur. We also think there is value in packages being able to depend on a testing framework being provided by the Swift system.

However nothing stops us eventually making the support for XCTest work via our protocol system.