Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal for testing support in SwiftPM #51

Merged
merged 5 commits into from Jan 2, 2016

Conversation

@mxcl
Copy link
Contributor

commented Dec 12, 2015

Our proposal for how testing infrastructure should operate and be implemented in the package manager.

└── Tests
└── TestFoo.swift

The filename: `TestFoo.swift` is arbituary.

This comment has been minimized.

Copy link
@pwightman

pwightman Dec 12, 2015

Contributor

arbitrary?

This comment has been minimized.

Copy link
@ReadmeCritic
However, any such decision would warrant extensive design consideration,
so as to avoid polluting or crowding the command-line interface.
Should there be sufficient demand and justification for it, though,
it would be straightforward to add this functionality.

This comment has been minimized.

Copy link
@kylef

kylef Dec 12, 2015

I would really like to see any arguments after swift build --test or swift test pass though to the test runner.

Example, the test framework Spectre allows command line flags to use different reporters. For example, --tap, --dot etc.

This comment has been minimized.

Copy link
@modocache

modocache Dec 12, 2015

Collaborator

Very much agree in terms of command-line arguments being used by test runners, and in terms of the necessity of custom test reporters.

Still, I believe Swift's Process.environment may be able to suffice for now, and would allow us to decouple that important discussion from this proposal.

to be supported by the package manager.
We expect that such an implementation would take the form of
a Swift protocol that the package manager defines,
which other testing frameworks can adopt.

This comment has been minimized.

Copy link
@kylef

kylef Dec 12, 2015

Perhaps it could just be simpler, and if the test module has a main.swift file it's treated as a command line tool and can be built and run.

This is the approach I've done in Spectre and you can find an example using this approach (Makefile, main.swift).

This comment has been minimized.

Copy link
@modocache

modocache Dec 12, 2015

Collaborator

Have the authors considered decoupling swiftpm's testing support from XCTest, and indeed any specific testing framework? The alternative I propose is to simply run a test package's main.swift file. A test failure would be any exit status that is not 0.

This would allow developers writing their tests to choose how they test their applications--something that is very difficult in Objective-C XCTest, which hides the entry point to the test program from developers. In this proposed ecosystem, swift-corelibs-xctest could be expanded to provide seamless integration with Objective-C XCTest. And meanwhile, developers who aren't interested in integration with Objective-C XCTest could use entirely different testing frameworks, like @kylef's Spectre framework. I consider this to be an improvement upon the current Objective-C XCTest ecosystem, which prevents developers from changing even the order in which their test case classes are run.

If coupling is indeed the intent of the authors, I'd like to flesh out what exactly the following sentence means:

Initially, the Swift Package Manager will use XCTest as its underlying test framework.

Does this mean swift-corelibs-xctest will be linked and available to use within packages' tests, and those tests are still expected to use a main.swift file to call XCTMain()? Does it mean that swiftpm will call XCTMain() somehow, perhaps automatically discovering the objects conforming to XCTestCaseProvider in the test package? Or does it mean something else?

This comment has been minimized.

Copy link
@anandabits

anandabits Dec 12, 2015

Contributor

Have the authors considered decoupling swiftpm's testing support from XCTest, and indeed any specific testing framework? The alternative I propose is to simply run a test package's main.swift file. A test failure would be any exit status that is not 0.

+1. I have been using Quick (https://github.com/Quick/Quick https://github.com/Quick/Quick) and Nimble (https://github.com/Quick/Nimble https://github.com/Quick/Nimble) and really like them. SwiftCheck (https://github.com/typelift/SwiftCheck https://github.com/typelift/SwiftCheck) is something I may use in the future as well. It takes a completely different approach to testing. It would be a shame to tie ourselves to one test framework.

Of course if SPM supports alternative test frameworks it would need to be able to resolve those dependencies in a similar way to the dependencies of the package itself.

This comment has been minimized.

Copy link
@heckj

heckj Dec 14, 2015

I would also like to advocate for support for different test runners in addition to the concepts of passing down test reports as additional modules. I very much like XCTest as a default, and in other languages I've found it immensely beneficial to utilize different test additions and libraries. That may be more appropriate determined from an ENV variable.

This comment has been minimized.

Copy link
@ghost

ghost Dec 15, 2015

+1 on making this more flexible here. Additionally being able to include modules/plugins for other aspects of testing - failing on regressions of code coverage, static code analysis for detecting known errors/bad practice, enforcing code styles and so on - will be beneficial for code quality.

This comment has been minimized.

Copy link
@modocache

modocache Dec 16, 2015

Collaborator

I think a protocol could be an interesting approach. Without knowing what sort of protocol that would be, however, it's hard to comment on what would or wouldn't be possible.

This comment has been minimized.

Copy link
@bppr

bppr Dec 16, 2015

^^ Agreed, but I think as long as that protocol isn't part of XCTest and is more of a part of the build-tool / package management infrastructure itself, my testing framework can adopt that protocol fairly easily, while XCTest can be integrated as a sensible default. But each package should have a measure of control over what testing tools they want to use.

As someone developing a third-party testing tool, I really just don't want my testing framework to have an XCTest dependency just so I can package it with the most common tools. As long as that happens, I'll be a happy camper.

(edit to add I think it sounds like what @mxcl is suggesting gets a 👍 from me -- just want to ensure we're of the same mind)

This comment has been minimized.

Copy link
@bppr

bppr Dec 16, 2015

I think, for me, the biggest question in terms of implementation is: how do we package or distribute those custom test running plugins / protocols that are going to be visible from the build process itself -- project-level runtime dependencies listed from Package.swift aren't going to be accessible at build-time, because they're being built at build-time.

npm solves this by allowing you to provide a test command for your module, which seems like the simplest implementation. Other build tools like leiningen have a way to get build-time dependencies set up using a separate field for plugins. This means that each project's build is run in a sandbox with an import path set up for various user-specified dependencies. From there we could just import our test protocol and give the user a place to configure it (probably in a Package initializer).

Curious to see what folks around here are thinking?

(edit: also, if this is not the appropriate place for this particular conversation, would it be appropriate to start a mailing list thread to chat about these ideas? On swift-build-dev perhaps? Bit of a new face around here, any advice is appreciated)

This comment has been minimized.

Copy link
@kylef

kylef Dec 16, 2015

@kylef’s proposal that the presence of a main.swift causes standalone test-binaries has the serious con IMO that any additional infrastructure people may want to use on top of swiftpm testing will be decoupled. If the minimum layer is a protocol, then we can build whatever test-runners the community wants on top of that.

@mxcl Can you elaborate on what the problem is with things being decoupled? In my opinion, being decoupled would be a huge improvement.

I think it's fine that there is the XCTest protocol, but I really would like to have a way to not use it, even if that option is not default. Having the flexibility to write tests without any limitations or constraints imposed by the XCTest protocol would be beneficial. There are many different preferences on how testing can be done and I would hate to see a certain design being tightly coupled into the language and ecosystem. Tight coupling to a particular testing design might hamper Swift adoption in as-yet unforeseen applications.

As a proof of concept. I've build a simple runner (and unfortunately also a simple build tool for tests) which allows Spectre to work with SPM (although not integrated). This allows a Tests/ directory where I can have a main.swift which runs the tests. https://github.com/kylef/spectre-build

This comment has been minimized.

Copy link
@rballard

rballard Dec 27, 2015

Contributor

@kylef Would you mind starting a thread on swift-build-dev@swift.org about your proposal here? I'd love for us to put together a concrete proposal for how we'll support other test frameworks sooner rather than later – it sounds like you and others would like to get that support added soon!

FWIW, the main concern I have with the lightweight "just use a main.swift that runs whatever test code you want" approach is that it doesn't lead to unified test behavior for swift packages. I'd really like package users to be able to rely on being able to run the tests for all their dependencies without needing to manually install other test support first (re: @bppr 's concern), and to get parseable test output in a uniform format, so that tools like a CI dashboard, or a package index that checks package health, can easily be built on top.

Let's discuss on-list what sort of protocol we'd need to define to let you easily get up and running with a different test framework while a) making it possible for swiftPM to get all the components it needs to run your tests automatically and b) providing the test output in a uniform format. Thanks!

└── Test.swift

Additionally, we propose that building a module
also builds that module's corresponding tests.

This comment has been minimized.

Copy link
@modocache

modocache Dec 12, 2015

Collaborator

Excellent idea! But would it be possible to explicitly disable this behavior by passing command-line options to swiftpm?

I ask because I can envision several situations in which this is not desirable behavior, such as when the tests are known to be broken. In this situation, a developer could pull the latest revision down because its tests are failing on a continuous integration suite, and needs to then build and run that revision in order to attach a debugger like LLDB.

This comment has been minimized.

Copy link
@danthorpe

danthorpe Dec 14, 2015

I think it makes a lot of sense to follow the directory pattern used for the Sources with the Tests. 👍

Would this also naturally extend to allow for multiple test bundles using names like TestN? Having a build system which can build multiple test bundles (just by creating multiple directories) would be a boon for distributing testing across modern CI infrastructure which typically uses multiple virtualized agents. Something like $ swift build --test --n 1

Granted, large test suites implies large libraries which is something I think the SPM authors aim to discourage. Just a thought.

This comment has been minimized.

Copy link
@mxcl

mxcl Dec 16, 2015

Author Contributor

I will add to the proposal detailing that a flag will be available that disables building tests.

@danthorpe multiple test modules/bundles will be built in the same manner as multiple source modules are currently.


An additional option may be passed to the testing command
to output JUnit-style XML or other formats that can be integrated
with continuous integration (CI) and other systems.

This comment has been minimized.

Copy link
@modocache

modocache Dec 12, 2015

Collaborator

Would this system use a plugin architecture to allow users to specify their own test reporters? For example, one that outputs the results in a JSON format? Or would each reporter need to be merged into the swiftpm codebase?

This comment has been minimized.

Copy link
@mxcl

mxcl Dec 16, 2015

Author Contributor

At first no, however this makes sense to implement at the same time as the feature whereby custom test frameworks can be used. Probably the same route would work well, ie. a protocol is defined and a dependency can be built into the test-runner that generates output.

@modocache

This comment has been minimized.

Copy link
Collaborator

commented Dec 12, 2015

Currently the test suites in swift-corelibs-foundation's and swift-package-manager use different scripts/Xcode projects for testing, but both do the following:

  • On OS X, Objective-C XCTest is executed via $(DEVELOPER_DIR)/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/xctest. This program gathers up all the test cases and runs them.
  • On Linux, XCTest is executed via XCTMain(), called from within a main.swift file. Both foundation and package-manager use different systems to ensure this file is not run on Linux.

How will this change as a result of this proposal, if at all?

@parkera

This comment has been minimized.

Copy link
Member

commented Dec 12, 2015

Ultimately we want swift-corelibs-foundation to be able to build, run, and test itself with swiftpm.

@modocache

This comment has been minimized.

Copy link
Collaborator

commented Dec 12, 2015

@parkera On Linux? Or on OS X as well?

@parkera

This comment has been minimized.

Copy link
Member

commented Dec 12, 2015

On Linux - the plan with the core libraries is to provide their features where we do not have the Darwin equivalents.

a test for "Foo" will depend on compilation of the library target `Foo`.
Any additional dependencies or dependencies that could not be automatically determined
would need to be specified in a package manifest separately.

This comment has been minimized.

Copy link
@danthorpe

danthorpe Dec 14, 2015

To clarify the author's intent for resolving additional dependencies - does this mean that in Package.swift there will be a "private" section for dependencies used by the tests, (things like DVR), or that the Test directory will include its own manifest?

This comment has been minimized.

Copy link
@kylef

kylef Dec 14, 2015

Just want to link this to apple/swift-package-manager#74

There is a pull request and discussion around "private dependencies" in this pull request.

This comment has been minimized.

Copy link
@mxcl

mxcl Dec 16, 2015

Author Contributor

As much as I like the idea of having another manifest and thus keeping it all separate, maintaining two manifests is an extra layer of tedium for packagers, so I think we'll (longterm) be learning towards keeping all of this in one manifest file. The linked PR is being reviewed today, but is not a firm decision, we may at a later time change the manifest format before Swift 3 drops.

@pcantrell

This comment has been minimized.

Copy link

commented Dec 14, 2015

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

├── Foo.swift
└── Tests
└── Test.swift

This comment has been minimized.

Copy link
@ghost

ghost Dec 15, 2015

Is there a reason for this variant? Will one format be recommended over the other? Having one recommended structure seems like it would be better in terms of consistency/providing guidance to developers.

This comment has been minimized.

Copy link
@mxcl

mxcl Dec 15, 2015

Author Contributor

One will be recommended. I believe in strong recommendations for flexible workflows.

We would prefer to go even further by executing the tests
each time a module is built as well,
but we understand that this would impede debug cycles.

This comment has been minimized.

Copy link
@ghost

ghost Dec 15, 2015

Often it is simpler/faster to completely disable testing when making changes/prototyping to then re-enable them just before submitting (to make sure nothing has been missed and everything passes). Would it make sense to provide this kind of workflow?

This comment has been minimized.

Copy link
@mxcl

mxcl Dec 15, 2015

Author Contributor

Yes agreed, via some command line flag. I’ll add this to the proposal.

@drewcrawford

This comment has been minimized.

Copy link

commented Dec 15, 2015

How are perf tests handled? Is that:

  1. A separate command that should be proposed and discussed totally separately? swift build --perftest?
  2. A flag to test? swift build --test --performance-only (or swift build --test --with-performance)
  3. Do they run as part of --test?
  4. Do they live in the test folder?
Detail the ability to specify specific tests to run.
@jpsim

This comment has been minimized.

Copy link
Contributor

commented Dec 16, 2015

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

This is a concern of mine as well. Tests that require to be run on a physical device or on a specific platform would also fall into this category.

Although I think it should be sufficient for the first version of this to allow opting out of certain test cases within this directory structure. Maybe by having a static var on XCTestCase subclasses:

class FooDeviceTests: XCTestCase {
  static var excludeFromPackageManagerTests = true
}
@modocache

This comment has been minimized.

Copy link
Collaborator

commented Dec 16, 2015

(As I commented above) I don't love the idea of coupling swiftpm to corelibs-xctest. Adding something like XCTestCaseProvider.excludeFromPackageManagerTests would then also couple corelibs-xctest to concepts that only exist in swiftpm, which I think is undesirable.

Theoretically, you could exclude certain test cases from swiftpm using existing concepts:

class FooDeviceTests: XCTestCase {
    var allTests: [(String -> () -> Void)] {
        var tests = [(String -> () -> Void)]()
        if !Process.arguments.contains("swift-build") {
            tests.append(("test_onPhysicalDevice", test_onPhysicalDevice))
        }
        return tests
    }
}

Going forward, we should have test frameworks like swift-corelibs-xctest and Spectre determine how to aggregate tests. When left to implement their own aggregation schemes, testing frameworks can innovate: the way JUnit 4 implements aggregation is different from how RSpec uses test filters; both have pros and cons. Asserting too much control in how tests are run in the package manager stifles these frameworks' opportunity to develop new concepts.

The command-line arguments part of the proposal, introduced in 68e618c, is great. It sounds like it would make filtering easy to implement, either in swift-corelibs-xctest or other frameworks:

// swift build -t --exclude broken

// MyTestingFramework parses command-line arguments to
// determine which tests to run or not run.
class MyBrokenTests: MyTestingFrameworkTestCase {
    // These tests don't work yet, so I tag them as 'broken'.
    var tags: [String] { ['broken'] }
    func test_myTestThatDoesntPassYet() { /* ... */ }    
}
@mxcl

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2015

Currently the test suites in swift-corelibs-foundation's and swift-package-manager use different scripts/Xcode projects for testing, but both do the following:

  • On OS X, Objective-C XCTest is executed via $(DEVELOPER_DIR)/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/xctest. This program gathers up all the test cases and runs them.
  • On Linux, XCTest is executed via XCTMain(), called from within a main.swift file. Both foundation and package-manager use different systems to ensure this file is not run on Linux.

How will this change as a result of this proposal, if at all?

Long term a goal is to remove the divergence here, but there is no timeline for it.

@mxcl

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2015

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

There is no proposal for this at this time. However, we intend to support other test frameworks that conform to a protocol vended by swiftpm, it seems easy enough to design the protocol with these considerations in mind (it could signify the end of a test with a closure to allow asynchronous running mechanisms, and start-end functions could be provided to allow control of external test processes).

@mxcl

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2015

How are perf tests handled? Is that:

  1. A separate command that should be proposed and discussed totally separately? swift build --perftest?

From the top of my head, I don’t see why it would need to be separated.

  1. A flag to test? swift build --test --performance-only (or swift build --test --with-performance)

We could certainly allow this kind of specification. Currently the proposal says it will accept the name of the test case on the command line and that seems sufficient initially.

  1. Do they run as part of --test?

I would think: yes.

  1. Do they live in the test folder?

I would think: yes.

@mxcl

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2015

Although I think it should be sufficient for the first version of this to allow opting out of certain test cases within this directory structure. Maybe by having a static var on XCTestCase subclasses:

class FooDeviceTests: XCTestCase {
  static var excludeFromPackageManagerTests = true
}

It seems to me that #ifs in the Package.swift would be a better way to exclude targets for different target platforms.

@mxcl

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2015

Going forward, we should have test frameworks like swift-corelibs-xctest and Spectre determine how to aggregate tests. When left to implement their own aggregation schemes, testing frameworks can innovate: the way JUnit 4 implements aggregation is different from how RSpec uses test filters; both have pros and cons. Asserting too much control in how tests are run in the package manager stifles these frameworks' opportunity to develop new concepts.

If you feel we are missing any understandings here, please feel free to enlighten us. I am certainly less of an expert in this area than I'd like.

@DougGregor

This comment has been minimized.

Copy link
Member

commented Dec 16, 2015

Can we take this discussion over to a mailing list? Presumable the package manager mailing list, since it's not really in the purview of the general swift-evolution list?

@modocache

This comment has been minimized.

Copy link
Collaborator

commented Dec 16, 2015

If you feel we are missing any understandings here, please feel free to enlighten us.

Oops, sorry if my comment came across as sanctimonious here! 😝

My intention was simply to point out that baking into the package manager the ability to include or exclude performance tests, depending on how that was implemented, may make it harder for test frameworks to build cool new features. I don't think anyone's missing any understandings; based on your comments above I think we're all on the same page! 👍

Can we take this discussion over to a mailing list?

Sounds fine to me!

@pcantrell

This comment has been minimized.

Copy link

commented Dec 16, 2015

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

There is no proposal for this at this time. However, we intend to support other test frameworks that conform to a protocol vended by swiftpm, it seems easy enough to design the protocol with these considerations in mind

I’m less concerned about the details of how the test framework is invoked, and more about being about to put all the things in Tests that belong there, without worrying about that tripping up SwiftPM. That means:

  1. being able to have multiple modules under Tests, some of which are not run by SwiftPM by default,
  2. having the ability to run those non-default test modules via SwiftPM, and
  3. having the ability to put directory structures under Tests that SwiftPM does not know how to deal with at all, and run through some different mechanism altogether.

It’s unclear to me from the wording of the current proposal whether this is covered. It sounds like SwiftPM would find everything named *.swift (or maybe *Test.swift) under the test directory and run it as a test?

@mxcl

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2015

If anyone has any further comments then we should take them to swift-build-dev@swift.org. I will finish by replying to the last two comments. The intent for comments here is more to correct things, it was my mistake to direct comments here initially.

Oops, sorry if my comment came across as sanctimonious here! 😝

Not at all, I seek merely to ensure we have considered all aspects.

I’m less concerned about the details of how the test framework is invoked, and more about being about to put all the things in Tests that belong there, without worrying about that tripping up SwiftPM. That means:

  1. being able to have multiple modules under Tests, some of which are not run by SwiftPM by default,
  2. having the ability to run those non-default test modules via SwiftPM, and
  3. having the ability to put directory structures under Tests that SwiftPM does not know how to deal with at all, and run through some different mechanism altogether.
    It’s unclear to me from the wording of the current proposal whether this is covered. It sounds like SwiftPM would find everything named *.swift (or maybe *Test.swift) under the test directory and run it as a test?

Excluding targets will be possible as a result of future work on Package.swift.

@keith keith referenced this pull request Dec 17, 2015
rballard added a commit that referenced this pull request Jan 2, 2016
Proposal for testing support in SwiftPM
@rballard rballard merged commit 58adcc1 into apple:master Jan 2, 2016
@mxcl mxcl deleted the mxcl:swiftpm-testing branch Mar 15, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.