Migrate the simplest system test to scalatest (as a unit test)

[cyrille-artho]

For performance and ease of use, we would like to run upcoming tests under scalatest.
To make sure we have a working test harness and an example, I would like to migrate Choose00, the simplest test that currently exists.

For this, we need to have a harness ModbatTestHarness, which is similar to ConfigTest, to run Main.main:

package modbat.mbt

import java.io.ByteArrayOutputStream
import java.io.FileDescriptor
import java.io.FileOutputStream
import java.io.PrintStream

import org.scalatest._

object ModbatTestHarness {
  def testMain(args: Array[String]): (Int, List[String], List[String]) = {
    val out: ByteArrayOutputStream = new ByteArrayOutputStream() 
    val err: ByteArrayOutputStream = new ByteArrayOutputStream()
    val ret = 0

    Console.withErr(err) {
      Console.withOut(out) {
        ret = Main.run(args)
      }
    }
    (ret, scala.io.Source.fromString(out.toString).getLines().toList, scala.io.Source.fromString(err.toString).getLines().toList)
  }
}

Another class should be used to test different features, e.g., ChooseTest. For now, we'll have only one test there, but we can migrate more tests in the future and add new tests:

class ChooseTest extends FlatSpec with Matchers {
  "Choose00" should "pass with one transition" in {
    val result = ConfigTest.testCtor(Array("-s=1", "-n=1", "--no-redirect-out", "modbat.test.Choose00"))
    result._1 should be(0)
    result._2 should contain ... // three checks, probably in three lines
    result._3 shouldBe empty
  }
}

In the output, there are three lines that are relevant. I'd like to check them against the actual content without the "preamble" [INFO], and without looking at the order of the output. So please add three checks, to verify that each substring occurs (e.g., "1 tests executed, 1 ok, 0 failed."):

[INFO] 1 tests executed, 1 ok, 0 failed.
[INFO] 2 states covered (100 % out of 2),
[INFO] 1 transitions covered (100 % out of 1).

[cyrille-artho]

Note that this will only work for successful tests for now. Any test that results in System.exit(1) will terminate the VM and unit tests. A full migration therefore requires some refactoring of error handling.

[cyrille-artho]

I have refactored parts of the code (see issue #27) so that Main.run can now be used, which returns the exit code used by main. Therefore, the test harness should return a triple: return code, stdout, stderr. I've updated the entry above accordingly.

[cyrille-artho]

I think we have to add a fixture for these tests, which sets the environment variables.
I think a LOT of unit tests will share this part:
CLASSPATH=build/modbat-test.jar
So it is probably best to add this to ModbatTestHarness, but use it from each class that declares tests.

We could have a simple helper function for this:

  def setTestJar {
    System.setProperty("CLASSPATH", "build/modbat-test.jar")
  }

Probably testMain should be parametrized with another parameter, a function that sets the environment. This can be a function that returns "Unit" (void in Scala). Each test should specify this, as tests can be reordered (prioritized), and updating the classpath affects future tests; so they all should set the right setting.

Probably the syntax for the second parameter is: ...testMain(args: Array[String], env: () => Unit): ...", and env can be called simply by using env(), in the first line of testMain` after the variable declarations.

[cyrille-artho]

Once this works, this also needs a comment in the harness (I will add that).

[cyrille-artho]

I have reorganized the sources in branch unit-test in your repo. The build dependency could thus be resolved. There are two issues left:

1. The two JAR files modbat-examples.jar and modbat-test.jar are currently not built. Their specs have to be added back to build.gradle (the top-level one).

2. The output of one test case still appears on the console, even after uncommenting "withOut" and "withErr". Probably the reason is that System.out and System.err still have to be redirected, being different from Console.out and Console.err.

The easiest way to resolve point 2 is to try changing System.out to Console.out in "src/main/scala/modbat/log/Log.scala. That may fix the currently broken unit test.

Point 1 is needed for the system tests to pass again. This requires more work (modbat-examples.jar should have the sources as well, modbat-test.jar only needs the class files as it is not a release archive).

[cyrille-artho]

The pull request above fixes part 2, but part 1 still needs to be fixed.
If you have time, please try to find out how to manually specify what folders (class files and source files) to add to a specific JAR file. In that case, we can build the old modbat-test.jar (only class files) and modbat-examples.jar (class files + sources).

[cyrille-artho]

This issue can be closed after a review and removal of redundant tests that are still run with bin/test.sh.

[cyrille-artho]

Log files that are no longer needed should also be removed.