Run testsuite directly

[dyfer]

Purpose and Motivation

This PR changes the procedure of running the test suite by replacing QPM with direct calls to sclang.

The need to set up QPM makes it harder to recreate testsuite environment locally. IMO by running tests directly with sclang it's easier to debug tests as they might behave within the whole testsuite. Additionally QPM uses Python 2 which is deprecated.

Details

test runner

The new test runner is very similar to the test runner used previously in QPM. The changes include removing some try {} statements (trying to avoid #232), as well as adjusting data format (see below).

QPM vs direct calls to sclang

QPM had the following roles for the purpose of running tests IIUC:

1. adding testsuite/classlibrary to the compile paths
   1a. adding API quark to the compile paths
2. running tests
3. printing the results
4. capturing output of sclang and parsing for error messages

(1), (2), and (3) is now done as 3 separate calls to sclang. (1a) is not needed. (4) is not implemented.

data format for storing test results

Previously json format was used to store test results. This required using the API quark when running testsuite. This has been replaced by native SC objects. Additionally, the "proto" file that indicates which tests to run is now not overwritten with the results; instead a separate file is created to store the results.

testing procedure

The testing procedure is largely the same as before, and its reliability hasn't improved that much, but I believe this paves the way for further improvements. One outcome of the proposed changes is that the same script used in the CI can be run directly in the IDE with virtually no changes.

reliability

The reliability of the tests themselves is similar to the previous state. However, I believe the test environment is less likely to do "empty passes", like here, which I think was related to running array operations within try {} (i.e. the error reported in #232, IIUC). Still, occasional failures do happen: some tests occasionally fail, sclang segfaults, and mysterious crashes occur when trying to run the whole testsuite on Windows.

sclang scripts

This PR adopts the main test runner script from qpm and adds two other scripts: one to add testsuite/classlibrary to the compiled paths, and one to post the results.

The scripts, if run outside of the IDE, uses ANSI color escape codes. This might be problematic if the scripts are run in a terminal that doesn't support these, but I'd argue this is an issue to be addressed if it actually proves problematic to anyone.

Posting results is done using a separate script and is triggered in a separate "step" in GHA. This makes it easier to see the summary of the results. However, in certain cases, if sclang (erroneously) doesn't compile the class library, the posting fails (I've only seen that on Windows). IMO this is something to investigate later.

updated build matrix

As a part of this update, I've moved the Linux tests to run on a build done under Ubuntu 22.04 and GCC 12. NOTE: This also addresses the problem of the ubuntu-18.04 images being deprecated.

EDIT: fixed tests

I initially didn't intend to include any actual fixes to the UnitTests in this PR, but I've found both TestThreadReschedule and TestSerialPort failing/hanging on multiple runs; the included commits updating these two seem to make test suite more reliable.

Outstanding issues

test runner script vs UnitTest.runAll

One could argue that we shouldn't need a separate test runner script and that it should be enough to run UnitTest.runAll. The test runner provides the following functionality, not provided by UnitTest.runAll:

• store test results incrementally in a file
• catch unhandled errors thrown in the testsuite
• print the results after running the tests

We could consider migrating/integrating that functionality into UnitTest.runAll, but I believe that would be something for another PR. In any case, it would be useful to have everything directly in the main repository, as opposed to running with QPM.

documentation

There's only minimal documentation on qpm and running testsuite in the CI (1, 2); no documentation in the helpfiles AFAICT. This PR does not add any documentation... partially because I think this is still a work in progress - we might move the runner script into the UnitTest class, or change it considerably.

If you feel that some documentation should be added to this PR, please let me know. Otherwise I'd just update the two linked wiki entries indicating that we are now running testsuite directly in sclang after this PR is merged.

tests on Windows

Testing should now be possible on all platforms using essentially the same code.

Testing environment on Windows was added and the pipeline seems to work, but it's currently disabled. There seem to be additional issues on Windows that result in sclang's unexpected behavior, which prevents the testuite from ever finishing. I made some progress with that, but I feel it should be saved for another PR.

You can see a small subset of the testsuite running on Windows here.

Since GHA runners offer no audio hardware, the Windows runner installs Jack2 with JackRouter (virtual ASIO soundcard), which is then used by scsynth during tests.

running time of the whole suite

I believe that the running time of the whole suite is similar to previous implementation. However, the overall running time is not being reported next to the number of passed tests, as it has been with qpm implementation. This is something I could add if desired, but IMO it's enough that the time of the testing step is reported in GitHub Actions UI.

Types of changes

• Documentation
• Bug fix
• New feature

To-do list

• Code is tested
• All tests are passing (as much as they had before)
• Updated documentation (n/a?)
• This PR is ready for review

[dyfer]

@jrsurge @joshpar just to confirm: I've run these scripts in both Windows Command Prompt and in PowerShell. In both cases they print output properly and in color (i.e. properly interpreting ANSI escape codes). Moreover, the shells are fully operational afterwards.