Enable defining and running tests inside same Scala file

[jvican]

Is your feature request related to a problem? Please describe.

Languages such as Rust enable you to define tests (usually unit tests) together with the code one is testing. This is useful and a productivity booster:

• It allows readers to see how an API is immediately used and what tests are present; and,
• It allows writers to avoid the tax to create an independent test file for everything you want to test and benefit from the locality of having the tests right next to the tested source.

All in all, the following a.scala source should be valid:

//> using scala "3.1"
//> using lib "org.scalameta::munit::0.7.27"

class NumAPI:
  def foo(): Int = 3

class NumAPISpec extends munit.FunSuite:
  test("check foo returns 3") {
    assertEquals(NumAPI().foo(), 3)
  }

Describe the solution you'd like

scala-cli could enable us to define and run tests in the same source code. This would involve making scala-cli test a.scala compile and run the tests defined in a.scala, regardless of whether //> using target.scope "test" has been defined or not.

Additional context

Here's a few design considerations to take into account just from the user and semantics POV (not discussing implementation details).

Should test code defined in a.scala be available/importable (when #948 is implemented)? I don't think it should. Tests defined with the tested code should be treated as private. The benefit of this is that we can also drop the test dependencies from being depended on.

What should we do with test dependencies? The unit tests might need to import additional test dependencies aside from the test framework. We should have a way that these test dependencies are not leaked to the classpath of other modules depending on the source.

[alexarchambault]

I like this idea.

Implementation-wise, I think we would likely compile everything together (main scope + tests), then split things in a second time. So we would have to:

• split test-specific classes from the rest (test suites, that we can detect like the test command does, would automatically go in the test scope, plus… maybe classes with a specific annotation, like @scala.testOnly, would be put there too)
• check that no class in the main scope references a class in the test scope
• split transitive dependencies (resulting in main ones and test ones), and maybe warn if some dependencies in the main scope were bumped by dependencies in test
• check that no class in the main scope references test-specific dependencies

[Gedochao]

@antholeole if you're indeed interested in implementing this, let me know if you need any help with setting up.
I think having this would be cool, Markdown sources could also benefit from this.

[antholeole]

I'll break ground on this when I get free time - if it's blocking anyone I can try to prioritize this and / or relinquish

[antholeole]

Sat down to get started on this today and it seems the request is actually working more or less without any modification?

//> using dep "org.scalameta::munit::0.7.29"

object NumAPI:
  def foo(): Int = 4

  @main
  def main() = print(foo())

class NumAPISpec extends munit.FunSuite:
  test("check foo returns 3") {
    assertEquals(NumAPI.foo(), 3)
  }

scala-cli test foo.scala runs the test in the source; scala-cli foo.scala runs the main block, as the RFC requested. Not sure if this is intended behavior (in the sense that someone wrote it for this purpose) but it does what I want it to. I think this is enough for my use case, so I will unassign this from myself.

EDIT: I found some places where enchancement is needed. I'm still going to take a look

[lbialy]

Are you fine with test dependencies polluting the classpath?