ParallelTestExecution causes sbt to think tests always pass.

[danmbyrd]

Hi,

I think I may have discovered a bug with the integration between sbt and scalatest. I've attempted to reproduce the issue in this dummy project - https://gist.github.com/danmbyrd/b4d9ec4168a19de59264. It seems that mixing in ParallelTestExection causes SBT to think that the tests within that spec always pass.

With the project as shown in the gist I get this output:

localhost:scalatest-repro dan$ sbt "show test:execute-tests"
[info] Loading global plugins from /Users/dan/.sbt/0.13/plugins
[info] Loading project definition from /Users/dan/src/scalatest-repro/project
[info] Set current project to dummy (in build file:/Users/dan/src/scalatest-repro/)
[info] DummySpec:
[info] numbers
[info] - should add
[info] numbers
[info] - should negate *** FAILED ***
[info]   0 was not equal to 1 (DummySpec.scala:13)
[info] Output(Passed,Map(org.acme.dummy.DummySpec -> sbt.SuiteResult@203a16),List(Summary(ScalaTest,Run completed in 263 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***)))
[success] Total time: 0 s, completed Nov 7, 2014 3:51:23 PM

The exit code seems to also reflect that sbt does not think any of the tests are failing (despite the logging):

localhost:scalatest-repro dan$ sbt test; echo $?
[info] Loading global plugins from /Users/dan/.sbt/0.13/plugins
[info] Loading project definition from /Users/dan/src/scalatest-repro/project
[info] Set current project to dummy (in build file:/Users/dan/src/scalatest-repro/)
[info] Updating {file:/Users/dan/src/scalatest-repro/}dummy...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/dan/src/scalatest-repro/target/scala-2.10/test-classes...
[info] DummySpec:
[info] numbers
[info] - should add
[info] numbers
[info] - should negate *** FAILED ***
[info]   0 was not equal to 1 (DummySpec.scala:13)
[info] Run completed in 277 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[success] Total time: 5 s, completed Nov 7, 2014 3:55:32 PM
0

Where as if I remove the mixing in of ParallelTestExecution I get the following:

localhost:scalatest-repro dan$ sbt "show test:execute-tests"
[info] Loading global plugins from /Users/dan/.sbt/0.13/plugins
[info] Loading project definition from /Users/dan/src/scalatest-repro/project
[info] Set current project to dummy (in build file:/Users/dan/src/scalatest-repro/)
[info] Compiling 1 Scala source to /Users/dan/src/scalatest-repro/target/scala-2.10/test-classes...
[info] DummySpec:
[info] numbers
[info] - should add
[info] numbers
[info] - should negate *** FAILED ***
[info]   0 was not equal to 1 (DummySpec.scala:13)
[info] Output(Failed,Map(org.acme.dummy.DummySpec -> sbt.SuiteResult@7bfc07a7),List(Summary(ScalaTest,Run completed in 235 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***)))
[success] Total time: 4 s, completed Nov 7, 2014 3:56:16 PM

localhost:scalatest-repro dan$ sbt test; echo $?
[info] Loading global plugins from /Users/dan/.sbt/0.13/plugins
[info] Loading project definition from /Users/dan/src/scalatest-repro/project
[info] Set current project to dummy (in build file:/Users/dan/src/scalatest-repro/)
[info] DummySpec:
[info] numbers
[info] - should add
[info] numbers
[info] - should negate *** FAILED ***
[info]   0 was not equal to 1 (DummySpec.scala:13)
[info] Run completed in 279 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error]     org.acme.dummy.DummySpec
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1 s, completed Nov 7, 2014 3:56:55 PM
1

I'm using scalatest 2.2.2, sbt 0.13.6 and scala 2.10.4 as shown in the project.

Let me know if theres any additional information that could be helpful, and apologies if this is an issue that is more on the SBT side.

[fommil]

we're seeing this with sbt 0.13.7 in https://github.com/ensime/ensime-server/pull/843 https://travis-ci.org/ensime/ensime-server/builds/52514899

weirdly, I thought we had disabled parallel testing when running on travis.

[aakoshh]

Hi!
I ran into the same issue and noticed that the output which comes form ScalaTest has the right error count but the last line which is from SBT is wrong:

[info] ScalaTest
[info] Run completed in 15 seconds, 384 milliseconds.
[info] Total number of tests run: 4
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 3, failed 1, canceled 0, ignored 21, pending 0
[info] *** 1 TEST FAILED ***
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0, Ignored 21
[success] Total time: 36 s, completed 01-Apr-2015 07:26:30
0

I inspected the results using a custom SBT TestResultLogger and none of the events had any non-zero counts so I couldn't return a different exit code.

As a workaround for Travis CI builds, I'm invoking the ScalaTest Runner instead of sbt test with something like the following in .travis.yml:

script:
  - sbt ++$TRAVIS_SCALA_VERSION test:compile
  - CP=$(sbt "export test:dependency-classpath" --error)
  - java -cp $CP org.scalatest.tools.Runner -R target/scala-2.11/test-classes -o -PS3

[fommil]

@aakoshh great! That is strong evidence that this might be an sbt bug

[brunon]

This is impacting us too - using ParallelTestExecution seems the only way to make the Akka Actor tests work, which makes it difficult to pick up the test failure status in SBT as it's ignored

[jawshooah]

I'm seeing the same thing:

[info] ScalaTest
[info] Run completed in 15 seconds, 627 milliseconds.
[info] Total number of tests run: 204
[info] Suites: completed 23, aborted 0
[info] Tests: succeeded 199, failed 5, canceled 0, ignored 1, pending 0
[info] *** 5 TESTS FAILED ***
[info] Passed: Total 31, Failed 0, Errors 0, Passed 31, Ignored 1
[success] Total time: 19 s, completed Jul 10, 2015 6:11:53 PM

Would really love a resolution to this issue. We're having to parse the log output directly in our Jenkins build to reliably halt on test failure.

ScalaTest 2.2.0
SBT 0.13.5

[xanderdunn]

This also became a problem for me. Thank you very much to @aakoshh for your solution! It works very well for me.

[cheeseng]

FYI, the fix is merged into 3.0.x branch already (unfortunately didn't make it into 3.0.0-M8), and shall be back-ported into 2.2.x as well.

[fommil]

Wow! Great news!

[fommil]

so was this an sbt or a scalatest bug?

[fommil]

@cheeseng is this in a 2.2.x release yet? Also, where is the commit that fixes it?

[bvenners]

Hi Sam,

Not sure. I'll investigate with Chee Seng tonight and if it didn't get released yet I'll make a 2.2.x release tonight. Sorry we've been focused on getting 3.0 out the door.

Bill

[fommil]

ironically, it turns out that parallel tests are slower in ENSIME because we already push the machine to the limit in terms of parallel JVMs to run the suites. Oh well 😄

[bvenners]

Hi Sam,

Do you mean running suites in parallel or actually running tests within a suite in parallel, which requires ParallelTestExecution? Oh, probably the latter since this ticket is about PTE. Assuming that, then yes, that's why we don't do that by default. Usually running suites (test classes) in parallel will give sufficient parallelism that running tests in parallel won't help much to improve performance. In cases where it does help performance to run the tests in parallel, that's when PTE should be mixed in. (And it is a performance enhancement, so really people should measure and if it doesn't help performance, I'd just "unmix" it back out.)

Bill

[bvenners]

We did some digging and I released 2.2.6-M1 on Sep 10, and it includes the fix. Can you try that and let me know if it fixes the issue?

[fommil]

OK, I'll put together a test project if there isn't already one.

[fommil]

@danmbyrd's dummy project correctly fails the tests in 2.2.6-M1 and 3.0.0-M14.

BTW, have you ever looked into sbts scripted plugin? It's great for writing integration tests for exactly this sort of thing!

[bvenners]

Awesome. Thanks for trying it out. I will get a 2.2.6 release out this week in conjunction with a 3.0 milestone that with luck will be the last milestone before RC1. In the meantime you can use 2.2.6-M1.