failing parallel tests always succeed

[fommil]

This is the dual of the bug reported at scalatest scalatest/scalatest#432 which has a reproducible project.

I'm seeing this in 0.13.7 in ensime-server and it is really quite a serious bug that tests are being reported as successful when they have actually failed!

[fommil]

(sorry, that wasn't a workaround... referenced the wrong ticket)

Note that this happens with or without the parallel forking.

[eed3si9n]

Is this a sbt issue or scalatest issue?

[fommil]

I have no idea, nor any way to tell.

[fommil]

I'm still seeing this in 0.13.8

[info] Run completed in 4 minutes, 56 seconds.
[info] Total number of tests run: 69
[info] Suites: completed 9, aborted 0
[info] Tests: succeeded 66, failed 3, canceled 0, ignored 1, pending 0
[info] *** 3 TESTS FAILED ***
[info] Passed: Total 22, Failed 0, Errors 0, Passed 22

even with

parallelExecution in IntegrationTest := false,
testForkedParallel in IntegrationTest := false,

there is definitely something very wrong, and I don't know if it's sbt or scalatest @bvenners

[fommil]

@eed3si9n this is marked "needs info" but I have given you a test case. Just do an assert(false) in any of the ensime-server integration tests and watch the suite pass...

[eed3si9n]

By NeedInfo, I meant needs-further-research-into-what's-going-on.

[fommil]

aah ok. It's extremely concerning actually.

[fommil]

@eed3si9n see the corresponding scalatest ticket. Information posted there suggests that this is an sbt bug rather than a scalatest runner bug.

[fommil]

I just seen this even with

parallelExecution in IntegrationTest := false,
testForkedParallel in IntegrationTest := false,

this is starting to become a really serious defect in sbt. There is a lot more chatter going on in the sister bug report at scalatest/scalatest#432 where some reporters have investigated and shown that it is an sbt, not a scalatest, bug.

[fommil]

with ParallelTestExecution may be the culprit! The tests seems to stabilise when I remove it. So maybe parallelExecution in IntegrationTest is ignored.

[fommil]

@eed3si9n responding to a query you had on gitter after I'd left, for me this is with fork=true. (Although I'm having a hell of a time trying to fork more than one JVM as a way of getting some parallelism in my tests)

[fommil]

I'm happy to report that this is a bug in scalatest and it will be fixed in 2.2.6!