Proposal: command progress events

[dragos]

This proposal is implemented in #7350. Below is the context and motivation for this change.

Motivation

Sbt defines the ExecuteProgress interface that offers fine-grained observability into task execution. However, users interact with sbt via commands, which in the most common case trigger one or more root tasks to execute. As an implementer of ExecuteProgress, it is currently impossible to infer what command is executing or if the current command was canceled by the user, or even if a command failed to parse. Moreover, a single command may trigger several “lifecycles” of execute progress (delimited by initial() and stop() calls), which makes it even harder to delimit logical workloads and correctly attribute task execution to user-facing actions. Examples are cross-compilation and sequential composition via semicolons (i.e. compile;test;publish).

Imagine a developer productivity team that would like to collect data about successful and failed builds across an organization. This data can then be used to diagnose errors (“20% of builds on the CI failed with the same error”, etc). Correctly determining the start, end, and failure status of each command is fundamental for such a service.

This proposal stems from our experience implementing Build Scans® for sbt, released as part of Gradle Enterprise (available for free at https://scans.gradle.com).

Goals

• Allow sbt plugins to receive notifications of what command starts executing and when it finishes
  • Command status should correctly reflect success, cancellation, and error (including parse error, if the command name is mistyped)
• Allow access to sbt State at the beginning and the end of each command
• The mechanism should be both source compatible (not break existing build definitions) and binary compatible (not break existing plugins).

Non-goals

The proposal shouldn’t change how tasks and commands are executed nor the order and content of existing events.

Proposal

We propose extending the existing ExecuteProgress trait with two additional methods:

/**
 * Tracks command execution progress. In addition to ExecuteProgress, this interface
 * adds command start and end events, and gives access to the sbt.State at the beginning
 * and the end of each command.
 */
trait CommandProgress extends ExecuteProgress[Task] {

  /**
   * Called before a command starts processing. The command has not yet been parsed.
   *
   * @param cmd The command string
   * @param state The sbt.State before the command starts executing.
   */
  def beforeCommand(cmd: String, state: State): Unit

  /**
   * Called after a command finished execution.
   *
   * @param cmd    The command string.
   * @param result Left in case of an error. If the command cannot be parsed, it will be
   *               signaled as a ParseException with a detailed message. If the command
   *               was canceled by the user, as sbt.Cancelled.
   */
  def afterCommand(cmd: String, result: Either[Throwable, State]): Unit
}

This new interface extends the existing ExecuteProgress interface. The same events that are published through the old one will also be published to implementers of CommandProgress. This new interface is exposed to clients as a new sbt setting.

  val commandProgress = settingKey[Seq[CommandProgress]](
"Command progress listeners receive events when commands start and end, in addition to task progress events.")

Use case: Composite commands

Assuming the user types ++compile, a client of this interface would now receive the following events:

Finished: shell. Remaining commands: [++compile, shell]
Starting: ++compile
Finished: ++compile. Remaining commands: [++  2.12.13!, +compile, ++  2.13.10!, +compile, +-, shell]
Starting: ++  2.12.13!
Finished: ++  2.12.13!. Remaining commands: [+compile, ++  2.13.10!, +compile, +-, shell]
Starting: +compile
Finished: +compile. Remaining commands: [++  2.12.13 test-project/Compile/compile, ++  2.13.10 test-project/Compile/compile, +-, ++  2.13.10!, +compile, +-, shell]
...
Starting: shell

Notice that ++compile finishes very quickly, but inserts a few extra commands in State.remainingCommands. This is a pattern that other commands follow, and without access to the State before and after each command clients would not be able to figure out what work belongs to this command.

Alternatives

A list of alternative solutions that we considered and why we didn’t pursue them.

Add methods directly to ExecuteProgress

The most natural extension would be to add these two methods with no-op defaults in the existing interface. This would ensure both binary and source compatibility. However, the interface is defined inside the tasks subproject, while sbt.State is in the main subproject. Without access to State before and after command execution, it would be impossible to attribute work to the correct user command.

Add methods to ExecuteProgress, and abstract over the State type

The full type signature of ExecuteProgress is ExecuteProgress[F[_]]. We could add a second type parameter for State and define before/afterCommand in terms of an abstract type for State. We then instantiate it as sbt.State in the main sbt project. However, this would not be source compatible.

Move the definition of ExecuteProgress in main

This alternative was briefly considered without a POC. It was dismissed because it would be a much larger code change and likely introduce incompatibilities in other projects. The tasks subproject is a self-contained execution engine written in a very generic way. It would go against the design philosophy of sbt to make it more concrete or cripple its progress interface in this way.