Add test for Test module. #143

kostiakoval · 2016-02-23T08:10:09Z

Check that Package.xctest are generated.

ankitspd · 2016-02-23T08:21:29Z

Tests/Functional/TestMiscellaneous.swift

+    func testModuleWithTests() {
+        fixture(name: "SwiftTesting/SingleTarget", file: #file, line: #line) { prefix in
+            XCTAssertBuilds(prefix)
+            print(prefix)


should print be there in tests?

👍 Thanks. Fixed

modocache · 2016-02-23T17:42:55Z

Awesome! Would love to see this and more tests for the test module merged.

kostiakoval · 2016-02-23T17:49:25Z

Me to :) Not sure if this test, and other, related to tests module, should be in a separate file?

mxcl · 2016-02-23T17:53:12Z

On Linux it has a different name. Also needs the other Linux machinery.

kostiakoval · 2016-02-24T07:58:43Z

Like that? I haven't tested it on linux, hope it will work :)

mxcl · 2016-02-24T18:07:55Z

Not quite, there is only one LinuxMain.swift.

kostiakoval · 2016-03-02T18:31:21Z

LinuxMain.swift is for a Fixture, so I can run test manually on the Linux.
I've update it to latest testing syntax and tested on Linux, it works.

Is there anything I should with it?

ankitspd · 2016-03-02T18:32:46Z

👌

mxcl · 2016-03-08T03:22:55Z

Tests/Functional/TestMiscellaneous.swift

+                XCTAssertDirectoryExists(prefix, ".build", "debug", "Package.xctest")
+            #else
+                //FIXME: test-Package not generated during swift-build on linux so this will fail.
+                //XCTAssertFileExists(prefix, ".build", "debug", "test-Package")


This should work, I'm confused, something doesn't work?

XCTAssertBuilds calls executeSwiftBuild which runs swift build,
test-Package executable is generated when "test" target is built

Of course. Well, we should run swift test.

Ok, I will fix it. Thanks for feedback

kostiakoval · 2016-03-09T19:36:54Z

@mxcl I've added XCTAssertTests to run swift-test. Works both on Mac and Linux.

mxcl · 2016-03-09T20:12:38Z

@swift-ci Please test

kostiakoval · 2016-03-09T21:01:27Z

@modocache Can I get any help why tests are failing, have any ideas or tips?

tmp/../SingleTarget/Tests/Foo/Test.swift:2:8: error: no such module 'XCTest'
import XCTest

mxcl · 2016-03-09T21:11:31Z

@kostiakoval platform? On OS X don't build swift with xctest, on Linux you also have to build foundation now and depending on how you invoke the tests, tell the system where to find it.

kostiakoval · 2016-03-10T08:00:14Z

I'm not sure how to pass those parameters. From what I see we can't pass any arguments to swift-test on linux. Should I send some environment variables?

I would appreciate any help please .

modocache · 2016-03-10T14:20:32Z

Thanks again for working on this, @kostiakoval! I think it'll be really cool to not only test corelibs-xctest's behavior (which we do in corelibs-xctest), but also the behavior of the swift test command.

First off as a basic sanity check I pulled down the latest revision on master for all repositories onto my Ubuntu 15.10 machine and ran:

$ utils/build-script -R --swiftpm --xctest --test -- \
    --skip-test-cmark --skip-test-swift --skip-test-foundation

This succeeded. I then tried to pull in the changes from your pull request locally:

$ cd ../swiftpm
$ git pull https://github.com/kostiakoval/swift-package-manager.git TestTarget-tests

Re-running the Swift utils/build-script from before, I get the following build error:

Compiling Swift Module 'Gettest' (4 sources)
/home/modocache/GitHub/apple/swiftpm/Tests/Get/Utilities.swift:136:20: error: use of unresolved identifier 'Resources'
    let toolPath = Resources.findExecutable("swift-test")
                   ^~~~~~~~~
/home/modocache/GitHub/apple/swiftpm/Tests/Transmute/Utilities.swift:136:20: error: use of unresolved identifier 'Resources'
    let toolPath = Resources.findExecutable("swift-test")
                   ^~~~~~~~~
/home/modocache/GitHub/apple/swiftpm/Tests/Functional/Utilities.swift:136:20: error: use of unresolved identifier 'Resources'
    let toolPath = Resources.findExecutable("swift-test")
                   ^~~~~~~~~
/home/modocache/GitHub/apple/swiftpm/Tests/ManifestParser/Utilities.swift:136:20: error: use of unresolved identifier 'Resources'
    let toolPath = Resources.findExecutable("swift-test")
                   ^~~~~~~~~
<unknown>:0: error: build had 4 command failures
error: exit(1): /home/modocache/GitHub/apple/build/Ninja-ReleaseAssert/llbuild-linux-x86_64/bin/swift-build-tool -f /home/modocache/GitHub/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug.yaml test
bootstrap: error: tests failed with exit status 1
utils/build-script: command terminated with a non-zero exit status 1, aborting

I seem to recall that Resources was renamed or removed in a recent commit (I think it may have been Xcodeproj support), so your commit probably needs to be updated to reflect that.

Anyway, silly me, I guess I probably should have tested exactly your changes, and not them merged into master! 😅 I tried just that:

$ git remote add kostiakoval https://github.com/kostiakoval/swift-package-manager.git
$ git fetch kostiakoval 
$ git checkout TestTarget-tests

But unfortunately this time I get Swift 3-related errors.

I'll try to checkout pre-Swift 3 versions of all the Swift libraries and re-run your pull request. In the meantime, if you could rebase your changes onto the latest swiftpm master branch and fix up the build errors, that'd be super helpful! 🙇

modocache · 2016-03-10T14:32:44Z

Because of the above I'm not able to test this out myself, but my current theory is that you'll need to pass flags to executeSwiftTest(), as the test utilities in SwiftPM already do for executeSwiftBuild() and XCTAssertBuilds().

In the ModuleMapsTestCase, for example, I see that -L /some/output/directory is passed to XCTAssertBuilds(). You'll probably need to do the same with executeSwiftTest(), but passing some subset of the flags I pass here.

A corelibs-foundation framework is included with Swift snapshot releases. If you're interested in having the tests use the currently installed version, then you can link that one. But I imagine that in order for this test to be robust, you'd want to allow a path to a Foundation framework to be specified somehow...

kostiakoval · 2016-03-11T09:28:56Z

@modocache Thanks for you amazing feedback!
I've fixed the Resources.findExecutable("swift-test") build error. Tests are working again on MacOS.

@mxcl Can we run CI to see if we still have that error on Linux?

mxcl · 2016-03-11T18:08:37Z

@swift-ci Please test

kostiakoval · 2016-03-11T19:07:55Z

When I try to run test on Linux, even from the master repo, I get this error

./Utilities/bootstrap test

/home/kkoval/apple/swiftpm/Tests/PackageType/PackageNameTests.swift:12:8: error: no such module 'XCTest'
import XCTest
...

mxcl · 2016-03-11T19:40:36Z

On linux you have to specify the path to: --xctest

See Utilities/bootstrap --help.

modocache · 2016-03-11T19:48:58Z

On linux you have to specify the path to: --xctest

You now must also specify the path to Foundation, using --foundation. I've been meaning to add this to the README. Those instructions are also in Utilities/bootstrap --help.

mxcl · 2016-03-11T19:50:08Z

Honestly it is better to run the tests via the swift build-script. I use:

swift/utils/build-script \
    --swiftpm --llbuild --xctest --foundation --test --release -- \
    --skip-test-swift --skip-test-cmark --skip-test-xctest --skip-test-foundation

Though obviously this builds the whole of swift. So…

kostiakoval · 2016-03-12T08:18:32Z

Yes that's right. But how can I specify --xctest path when I call swift-test.
Right now when we call swift-test on Linux we don't pass any arguments to swift-corelibs-xctest.
https://github.com/apple/swift-package-manager/blob/master/Sources/swift-test/test.swift#L26

Does swift-corelibs-xctest support it now?

ankitspd · 2016-03-12T11:37:31Z

@kostiakoval I was able to fix by just changing this ankitspd@772335d
I built toolchain on my Linux VM with

$ ./swift/utils/build-script --preset="buildbot_linux" install_destdir=/home/aciid/hdd/toolchain/swift-nightly installable_package=/home/aciid/hdd/toolchain/swift.tar.gz

mxcl · 2016-03-13T21:05:19Z

But how can I specify --xctest path when I call swift-test.

You can't, this is not a supported execution model. swift-test must be part of a complete toolchain or it must use a complete toolchain via the TOOLCHAINS environment variable.

kostiakoval · 2016-03-20T17:18:41Z

The problem was in wrong naming. I've tested it local and it works.
@mxcl Can we invoke CI and get it merged :)

mxcl · 2016-03-22T19:47:45Z

Conflicts.

kostiakoval · 2016-03-22T20:05:17Z

@mxcl Fixed. And cleaned up commits

kostiakoval · 2016-03-23T18:26:12Z

@mxcl Could you please invoke "CI test" and merge it please? Also PR #122 is ready

ankitspd · 2016-03-23T18:43:33Z

👍

mxcl · 2016-03-31T17:37:31Z

@swift-ci Please test

czechboy0 · 2016-04-01T11:25:51Z

@kostiakoval Tests failed on Linux, can you please take a 👀 ?

kostiakoval · 2016-04-03T17:57:15Z

I found how to reproduce that bug. It works fine if you run ./Utils/bootstrap test with no extra parameters.
I happens when you run it and provide --swiftc --xctext --foundation --sbt options.
Now I need to find a way to fix it :)

mxcl · 2016-04-04T03:18:01Z

I've tried to make this work on Linux and my conclusion is we shouldn't bother with this test:

It is hard to make this work in the context of a full build of swift since the paths for XCTest and Foundation are not in the standard "toolchain" shape.
We are always testing swift-test because we use swift-test to test ourselves.

Instead we could add an integration test, but really we don't even need to because we test swift-test when we test ourselves.

So I'd like to close this, if there are no objections.

kostiakoval · 2016-04-04T06:36:48Z

I agree with point 1. I've opened that PR because some Tests folder layout didn't work
#162

I agree to close that PR and find a better way to test it.

czechboy0 · 2016-04-04T07:28:55Z

@mxcl If we can find a way to still implement something like XCTAssertBuilds but for tests, I'm fine with closing this one. But I definitely think we need to validate layouts not just with swift-build, but also with swift-test, because completely valid layouts have (twice already) been broken on a released snapshot. I believe we could have prevented the second breakage if we had XCTAssertTests, because in that case swift-build worked but swift-test failed to compile tests. So I'm only OK with closing this PR if there's a strategy you have in mind that can give us the ability to validate a layout for swift-test. I'm afraid that if we don't start testing out layouts completely, sooner or later we'll break swift-test again for projects in the wild.

@kostiakoval Exactly why I'm following this PR - I had my projects break on a snapshot bc swift-test didn't compile test on a layout that was only being tested by running swift-build.

mxcl · 2016-04-04T17:08:17Z

This PR doesn't test layouts. But I agree we should test layouts. We don't have to do it functionally. Would be better to have unit tests for that component.

…endency-cycles Make cycle information available to the client

[pull] swiftwasm from master

ankitspd reviewed Feb 23, 2016
View reviewed changes

kostiakoval force-pushed the TestTarget-tests branch from 6f8dd63 to 4b04f73 Compare February 23, 2016 08:30

kostiakoval force-pushed the TestTarget-tests branch from 3785e92 to 4b04f73 Compare February 24, 2016 07:40

kostiakoval force-pushed the TestTarget-tests branch from 375aa4d to 8dd9fe8 Compare March 2, 2016 17:10

mxcl reviewed Mar 8, 2016
View reviewed changes

kostiakoval force-pushed the TestTarget-tests branch from 8879598 to 6690557 Compare March 9, 2016 16:08

kostiakoval force-pushed the TestTarget-tests branch 2 times, most recently from b813132 to 82a562f Compare March 11, 2016 08:28

kostiakoval force-pushed the TestTarget-tests branch from c373ff9 to 33ea130 Compare March 12, 2016 15:28

kostiakoval mentioned this pull request Mar 21, 2016

[swift-test] [BLOCKED by #143] Disambiguate Package.test targets #213

Closed

kostiakoval force-pushed the TestTarget-tests branch from 33ea130 to d760714 Compare March 22, 2016 19:58

kostiakoval and others added 3 commits March 22, 2016 21:03

Add test for Test module.

ceba4f0

Add files for linux tests

e7ef2ee

run and verify that swift-test are passing

414729a

kostiakoval force-pushed the TestTarget-tests branch from d760714 to 414729a Compare March 22, 2016 20:03

mxcl closed this Apr 4, 2016

kostiakoval mentioned this pull request Apr 24, 2016

Parse specifer in swift-test #266

Merged

ankitspd pushed a commit to ankitspd/swift-package-manager that referenced this pull request Jan 11, 2019

Merge pull request apple#143 from neonichu/better-diagnostics-for-dep…

7853943

…endency-cycles Make cycle information available to the client

MaxDesiatov pushed a commit to MaxDesiatov/swift-package-manager that referenced this pull request Jul 6, 2020

Merge pull request apple#143 from swiftwasm/master

bc7b4c4

[pull] swiftwasm from master

ddunbar mentioned this pull request May 20, 2016

[SR-1351] Add a way to run swift-test on a fixture in tests #5453

Open

Add test for Test module. #143

Add test for Test module. #143

Conversation

kostiakoval commented Feb 23, 2016

ankitspd Feb 23, 2016

Choose a reason for hiding this comment

kostiakoval Feb 23, 2016

Choose a reason for hiding this comment

modocache commented Feb 23, 2016

kostiakoval commented Feb 23, 2016

mxcl commented Feb 23, 2016

kostiakoval commented Feb 24, 2016

mxcl commented Feb 24, 2016

kostiakoval commented Mar 2, 2016

ankitspd commented Mar 2, 2016

mxcl Mar 8, 2016

Choose a reason for hiding this comment

ankitspd Mar 8, 2016

Choose a reason for hiding this comment

mxcl Mar 8, 2016

Choose a reason for hiding this comment

kostiakoval Mar 9, 2016

Choose a reason for hiding this comment

kostiakoval commented Mar 9, 2016

mxcl commented Mar 9, 2016

kostiakoval commented Mar 9, 2016

mxcl commented Mar 9, 2016

kostiakoval commented Mar 10, 2016

modocache commented Mar 10, 2016

modocache commented Mar 10, 2016

kostiakoval commented Mar 11, 2016

mxcl commented Mar 11, 2016

kostiakoval commented Mar 11, 2016

mxcl commented Mar 11, 2016

modocache commented Mar 11, 2016

mxcl commented Mar 11, 2016

kostiakoval commented Mar 12, 2016

ankitspd commented Mar 12, 2016

mxcl commented Mar 13, 2016

kostiakoval commented Mar 20, 2016

mxcl commented Mar 22, 2016

kostiakoval commented Mar 22, 2016

kostiakoval commented Mar 23, 2016

ankitspd commented Mar 23, 2016

mxcl commented Mar 31, 2016

czechboy0 commented Apr 1, 2016

kostiakoval commented Apr 3, 2016

mxcl commented Apr 4, 2016

kostiakoval commented Apr 4, 2016

czechboy0 commented Apr 4, 2016

mxcl commented Apr 4, 2016