Forward test events to listener as they're emitted

[Duhemm]

Current situation

sbt provides an API to register test event listeners which help better understand test execution. At the moment, it's impossible to rely on the timing of these events, because:

• they're not passed on to the listeners at the time they're emitted
• startGroup and endGroup events are received by listeners at different times depending on whether the tests run in-process or in a forked JVM.

What I'd like to do

I would like to fix this by:

• changing the in-process test runner so that events are immediately sent to the listeners as they're produced by the test frameworks.
• slightly change the communication protocol between sbt and the forked JVM:
  • introduce new StartGroup and EndGroup messages, which sbt will map to the eponymous events of the test listeners.
  • send test events immediately as they're produced by the test frameworks.
  • Increase the priority of the thread reading messages from the forked JVM, so that they're handled as early as possible.

No API changes are envisioned for this work.

Benefits

This change will make it possible to measure how much time is spent doing test class setup and cleanup. Currently, it's not possible to get this information, because the startGroup, testEvent and endGroup callbacks are called together and their payloads don't contain this information.

Limitations

The test events have no timestamp, so it's impossible for test events listeners to know accurately when an event was emitted. The proposed changes improves the situation, but the results cannot be 100% accurate. It'd be good to look into this for sbt 2.

Details of current implementation

When forking

• All events are collected for a given test group:

  sbt/testing/agent/src/main/java/sbt/ForkMain.java

  Lines 407 to 412 in fb8ca26

  	final EventHandler handler =
  	new EventHandler() {
  	public void handle(final Event e) {
  	eventList.add(new ForkEvent(e));
  	}
  	};

• The events are sent back together to sbt:

  sbt/testing/agent/src/main/java/sbt/ForkMain.java

  Line 438 in fb8ca26

  writeEvents(os, taskDef, events);

• sbt reads the events and emits the startGroup, testEvent and endGroup events all together:

  sbt/main-actions/src/main/scala/sbt/ForkTests.scala

  Lines 215 to 221 in fb8ca26

  	case Array(group: String, tEvents: Array[Event]) =>
  	listeners.foreach(_ startGroup group)
  	val event = TestEvent(tEvents)
  	listeners.foreach(_ testEvent event)
  	val suiteResult = SuiteResult(tEvents)
  	results += group -> suiteResult
  	listeners.foreach(_.endGroup(group, suiteResult.result))

When running in-process

• The start and end group events are produced as early as possible:

  sbt/testing/src/main/scala/sbt/TestFramework.scala

  Lines 166 to 170 in fb8ca26

  	safeListenersCall(_.startGroup(name))
  	try {
  	val (suiteResult, nestedTasks) = runTest()
  	safeListenersCall(_.endGroup(name, suiteResult.result))
  	(suiteResult, nestedTasks)

• The test events are collected and sent together when a test group finishes executing: https://github.com/sbt/sbt/blob/fb8ca26e77d495cf206a45025a63c9a6d0754a9f/testing/src/main/scala/sbt/TestFramework.scala#L137C14-L137C14