scripted tests and their design

[seroperson]

Hello! Recently, I had to write some scripted tests, and here I want to discuss their design and how they work.

To begin with, I have to say that they look like a non-native solution right from the start.

• You have to manage a special sbt-test directory, which isn’t supported by Metals or IntelliJ IDEA. It’s a completely separate directory and it's a pain to write tests there.
• Your plugin’s code is passed using the default dependency resolution mechanism and local plugin publishing, meaning your plugin is re-published each time scripted runs. You’re also responsible for passing the correct version, which must be done with something like scriptedLaunchOpts += version.apply(v => s"-Dproject.version=$v").value.
• The file containing test instructions has its own mini-DSL, which is quite limited and not extensible.
• If you have custom code that needs to run during a test, it must be implemented as a new task in the test’s build.sbt.
• If you want to reuse this code across multiple tests, your only options are copy-pasting or creating another plugin with shared utilities.

In Gradle’s implementation, they suggest an approach that doesn’t really differ from writing standard unit tests. In short, you write test cases that run a Gradle build using the GradleRunner class in a temporary directory, which you populate with the required contents directly within the test case.

This approach allows you to:

• Reuse all the existing testing logic that already runs unit tests. It comes with built-in support for timeouts, parallelization, afterEach, beforeEach, and more features currently missing from the scripted tests implementation.
• Avoid writing test logic inside build.sbt tasks, since you can now write it directly in your test case file. This also makes it easy to share code between tests. And no more mini-DSL.
• Enjoy full support from Metals and IntelliJ IDEA.

As for me current scripted tests implementation feels very flaky. Gradle's approach feels much more like a native solution. What do you think?

[eed3si9n]

As for me current scripted tests implementation feels very flaky. Gradle's approach feels much more like a native solution. What do you think?

scripted test existed ever since I was one of plugin authors (I wrote https://eed3si9n.com/testing-sbt-plugins/ in 2011 as a user), long before I became the maintainer, and I sort of took for granted that "this is the way it's done", so your perspective is refreshing. So thanks for the input.

Re: "native". I don't know if discussing what is and isn't "native" is all that productive. We can easily flip your argument saying that JUnit or ScalaTest style unit testing is (albeit popular) a generic solution, meanwhile scripted test is a purpose-built, integration testing framework that is native to sbt and the incremental compiler Zinc to emulate file changes interspersed with human command invocations.

If you've worked with Scala compiler on scala/scala, you might also discover that they also have a bespoke integration testing called Partest (which I also documented in https://github.com/scala/scala/blob/2.13.x/CONTRIBUTING.md#partest). Similar to scripted, basically you create some example files, and expected positive or expected-negative outputs from the compilation.

These file-based integration ("functional") testing is useful for tooling maintainers (like sbt, Zinc, and Scala compiler core devs and contributors) because we often get bug reports in the form of files, and we communicate with each other in the form of files with varying degree of minimization. And there are probably hundreds of tests in these repos captured this way. For example:

$ find sbt-app -name test | wc -l
     528

So some plugin authors, including myself, have started to advocate for its usage for plugin testing, riding on the shoulder of existing investment. I think reevaluating the usage of unit test framework for integration testing is an interesting idea, and something interested party can explore.

server-test

Probably the closest thing in sbt/sbt already might be the way we test sbt server, since we launch an instance of sbt server in a temp directory, initialized using files in server-test/src/server-test/ like scripted, but we establish a network connection against it, and run sbt server commands and test from outside:

  test("build/initialize") {
    val id = initializeRequest()
    assert(svr.waitForString(10.seconds) { s =>
      s.contains(s""""id":"$id"""") &&
      s.contains(""""resourcesProvider":true""") &&
      s.contains(""""outputPathsProvider":true""")
    })
  }

I don't know if it has the best developer experience, and it could be just as flaky as scripted test if not more.

miscellaneous concerns

1. File creation and update would need to be coded, as opposed to creating the files in some directory.
2. scripted test is extensible by implementing check task, that runs in the context of the build, which has access to arbitrary type created by the plugin tasks. A unit test running outside has no access to the task value, e.g. Analysis produced by compile task.

[seroperson]

Thank you for your quick and detailed response! I don’t think we need to completely replace scripted, as both approaches can coexist. Later, it might even be possible to implement scripted-like behavior on top of a unit testing framework, as demonstrated here (see the resources directory), so we wouldn’t have to maintain a separate mechanism for running scripted tests.

By "native", I meant that everyone is already familiar with writing tests using frameworks like JUnit or TestNG, which look and run the same way across most JVM ecosystems, and using a similar approach for testing plugins would feel more natural and easier for people to get started with.

Besides implementing some kind of SbtRunner (like GradleRunner in Gradle) we can also provide some utility methods, like loadBuildFromDirectory (to not code file creation), runScenario (to simplify possible migration from scripted approach) and more.