don't disable tests build-wide, just for specific subprojects where the failures are

[SethTisue]

there are a number of projects where we don't run any tests just because of (perhaps as little as) one failing test in one subproject

I hadn't previously tried to find a way to disable tests just for one subproject, but I know one now:

set executeTests in thisParticularSubproject in Test :=
  Tests.Output(TestResult.Passed, Map(), Iterable())

we should do a sweep through community.build, look for disabled test suites, re-enable them, see what fails, and then more selectively re-disable

we might even find that all the tests actually pass once enabled. when we disable running tests, they normally just stay disabled, perhaps for years

[SethTisue]

test run with all test suites re-enabled: https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/2450/consoleFull (404 til Jenkins finishes wondering what the difference between a tea "bag" and a tea "sachet" really is, anyway)

[SethTisue]

results from this round:

• succeeded: shapeless, scala-stm, scala-async, scala-refactoring
• failed: specs2, genjavadoc, scoverage, parboiled, scalacheck-shapeless, sbt
• blocked: akka-more, akka-http, breeze, ssl-config, scalikejdbc, scala-ssh, monix, unfiltered, akka-contrib-extra, scala-gopher, zinc, ammonite, http4s, github4s

[SethTisue]

69878ce re-enables tests on those four projects 🎉

most of the blocked projects are blocked by specs2 and/or genjavadoc

the genjavadoc failure looks like an overly sensitive test (only ordering differs, I think?)

what's going on with specs2? the single failing test in core (after which the other subprojects don't get to run) is org.specs2.specification.IsolatedExecutionSpec in core/test:test. we track their specs2-3.x branch. if I check out that branch locally and do ++2.12.4 and core/test:testOnly *IsolatedExecutionSpec, it fails. it also fails with ++2.12.3, but passes with ++2.12.2, so it seems this is a regression (of some sort, perhaps just a too-fragile test? idk) that the community build would have caught. I've reported it at etorreborre/specs2#622

[SethTisue]

as for genjavadoc, the failure is reproducible outside dbuild with ++2.12.5-bin-3792784 followed by genjavadoc-plugin/testOnly *BasicSpec; it passes with ++2.12.4

it looks like I just need to submit an upstream PR that adds an appropriate plugin/src/test/resources/patches/2.12.5.patch file (hopefully the Scala version number handling is smart enough to select that file when a Scala version number such as 2.12.5-bin-3792784 is involved)

[SethTisue]

lightbend/genjavadoc#123 fixes the genjavadoc issue, and a445762 re-enables the tests

[SethTisue]

let's try again but with SethTisue/community-build@ca8faf3 which disables running the tests in specs2 core only: https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/2457/consoleFull (404 til Jenkins stops feeling he can't even and decides that actually he can even)

EDIT: again, with another specs2 project's tests disabled: https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/2458/consoleFull

[SethTisue]

the part of this that enables specs2 tests has now been merged onto 2.12.x as its own thing.

I rebased my run-more-tests branch, here's a new test run: https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/2507/consoleFull (404 til Jenkins gets back from the optometrist)

[SethTisue]

the specs2 failure in run 2507 seems mysterious, and 2506 (not on this branch) had mysterious failures as well, so let's just run it again: https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/2509/consoleFull

[SethTisue]

run 2509: failed: scoverage, scalacheck-shapeless, scala-gopher

I'm pretty happy with the other results, so I'm going to declare victory and go with what we have