ScalaTest providing incorrect results to SBT

[japgolly]

SBT's results differ from ScalaTests's results when using nested suites.
I'm making a big assumption here that ScalaTest is providing incorrect results but I don't know that that's fact.

My setup:

class VanillaSuite extends Suites(new A, new B, new C)

class ParalellisableDbTests extends Suites(new X, new Y, new Z)
class DbSuite extends Stepwise(
  new ParalellisableDbTests,
  new F, new G, new H
)

In SBT I have: testOptions in Test := Seq(Tests.Filter(_.matches(".*Suite$")))

In my console, when I run tests, ScalaTest says:

[info] ScalaTest
[info] Total number of tests run: 416
[info] Suites: completed 38, aborted 0
[info] Tests: succeeded 416, failed 0, canceled 0, ignored 0, pending 0

SBT says:

[info] Passed: Total 7, Failed 0, Errors 0, Passed 7

I expect SBT to say:

[info] Passed: Total 416, Failed 0, Errors 0, Passed 416

[bvenners]

Hi japgolly,

This is caused because sbt uses the fully qualified suite name as a suite identifier, instead of the suiteId. This should really eventually be fixed in a new version of the Framework API we think, but in the meantime we are going to submit a PR to sbt to have it look for repeated suite class names and just accumulate the number of tests for that suite instead of replacing it. I won't close this until we submit the PR to sbt.

Bill

[cheeseng]

Pull request submitted to SBT:

sbt/sbt#1364

[bvenners]

I just verified this is now working as desired in sbt 0.13.8, so closing this issue.

Given:

import org.scalatest._

class A extends FunSuite {
test("one") {}
test("two") {}
}
class B extends FunSuite {
test("one") {}
test("two") {}
}
class C extends FunSuite {
test("one") {}
test("two") {}
}
class D extends FunSuite {
test("one") {}
test("two") {}
}
class E extends FunSuite {
test("one") {}
test("two") {}
}
class F extends FunSuite {
test("one") {}
test("two") {}
}
class G extends FunSuite {
test("one") {}
test("two") {}
}
class H extends FunSuite {
test("one") {}
test("two") {}
}
class X extends FunSuite {
test("one") {}
test("two") {}
}
class Y extends FunSuite {
test("one") {}
test("two") {}
}
class Z extends FunSuite {
test("one") {}
test("two") {}
}

class VanillaSuite extends Suites(new A, new B, new C)

class ParalellisableDbTests extends Suites(new X, new Y, new Z)
class DbSuite extends Stepwise(
new ParalellisableDbTests,
new F, new G, new H
)

You get:

[info] B:
[info] C:
[info] A:
[info] - one
[info] - one
[info] - one
[info] - two
[info] - two
[info] - two
[info] F:
[info] - one
[info] - two
[info] G:
[info] - one
[info] - two
[info] H:
[info] - one
[info] - two
[info] Y:
[info] X:
[info] Z:
[info] - one
[info] - one
[info] - one
[info] - two
[info] - two
[info] - two
[info] ScalaTest
[info] Run completed in 218 milliseconds.
[info] Total number of tests run: 18
[info] Suites: completed 12, aborted 0
[info] Tests: succeeded 18, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 18, Failed 0, Errors 0, Passed 18