Convert test executions in bin/test.sh to ScalaTest using ModbatTestHarness

[cyrille-artho]

All the executions of the test cases can be converted from a list of command-line arguments to a list of strings and calls to testMain in ModbatTestHarness.

We still want to keep bin/test.sh for reference, but we also want a version that (1) does not execute any tests (similar to option -n) but (2) still checks all the outputs (which -n does not do).

First, we want to convert a couple of test cases that share similar settings, and then adapt a part of bin/test.sh as (for example) bin/diff.sh, to check the log/err outputs. Once that works, we will convert all 250+ tests.

[cyrille-artho]

As a first step, please write a parser for the output of bin/test.sh -n. The parser should build a data structure for each of the 250+ test cases, and pretty-print the data in tab-separated format as follows:
Test number CLASSPATH [possibly empty] APP [scala ....jar] PARAM1 [-s=...] PARAM2 [-n=...] PARAM3... PARAMN

[cyrille-artho]

In the main loop of the parser, the logic would be as follows:
You always maintain one entry, with the necessary fields: classpath, appString, params[]

1. When you see an empty line: print the current data (which is empty at the beginning), as tab-separated entries. After that, you clear all data.
2. In the main parse loop, whenever you recognize a particular item, store it in that field (and extend the parameter list by one item if it is a parameter).

At the end, you have a program that processes one item at a time, and outputs all of them on the terminal. This allows us to visually inspect if most of the entries get parsed correctly. We can later redirect the output to a file and use that in Google sheets, or with grep -c to count items, etc.

[cyrille-artho]

Please commit the script under a new branch, and commit + push it.
As a next step, I would like to work on two things:

1. Get an overview of common parameters. For this, you can adapt the script from here: https://www.youtube.com/watch?time_continue=94&v=Pcyuaj5lCtA
   (Remove the hard-coded limit of "10", i.e., the parameter to most_common).
2. We want to convert the output of your script to commands that run one scalatest unit test each. Example:

  "SetWeight2" should "pass with two transitions" in {
    val result = ModbatTestHarness.testMain(Array("-s=1", "-n=1", "--no-redirect-out", "-s", "modbat.test.SetWeight2"), ModbatTestHarness.setTestJar)
    result._1 should be(0)
    result._2.filter(_.contains("1 tests executed, 1 ok, 0 failed.")) should not be empty
    result._2.filter(_.contains("2 states covered (100 % out of 2),")) should not be empty
    result._2.filter(_.contains("2 transitions covered (100 % out of 2).")) should not be empty
    result._3 shouldBe empty
  }

For this, we need to

1. find out how many class files to generate (based on what);
2. know if we can (sometimes) find a way to create meaningful test names;
3. check if we can import the right return code from bin/test.sh, or else fix that manually later.

We will also have to write the output to .log/.err files, and use an adapted version of bin/test.sh to diff them, later on.

[cyrille-artho]

I am writing this in anticipation of the push/PR that will follow after the network connection problems are solved.

As a next step, focus on those tests that use modbat.jar as the application string (about 189 tests). Convert them to the unit test framework as used by, e.g., SetWeightTest. In the conversion, check result._1 and result._3, but not result._2 as of now.

Try to create the output such that all tests for the same class are collected in one class with a number of unit tests. You can use the class name plus a number as the test name.

[cyrille-artho]

This task can now be carried out for most unit tests; the only exceptions are tests producing multiple output files, for which the current test harness will have to be extended. Over 80 % of the existing tests can be ported now, thanks to PR #101.
This will be done over the coming weeks.