Parallel execution of unit tests

[vkarak]

Originally opened by @teojgo and abandoned due to errors. We were simply missing an option!

./test_reframe.py --process-timeout=300 --processes=4
................................................................................SSS...S...................................................SSSSSS.......................................................S...........................................................................S..S................S.S..SS..S.S..SS.........................................................
----------------------------------------------------------------------
Ran 368 tests in 18.155s

OK (SKIP=21)

vs single-threaded:

....S........................................................SSS...S...................................................................................................SSSSSS..............................S..S.........................................................................................................................S.S..SS..S.S..SS.......................
----------------------------------------------------------------------
Ran 368 tests in 47.144s

OK (SKIP=21)

The only thing is what is a reasonable timeout? I would say to put set it to quite a high number, since for the submission unit tests, we might get stuck in the job scheduler for a long time if the target system is too loaded.

Internal issue: https://madra.cscs.ch/scs/reframe/issues/521

[teojgo]

Tested it and I get the following error sometimes:

FAIL: test_runall (unittests.test_policies.TestSerialExecutionPolicy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/manitart/reframe_dev_public/unittests/test_policies.py", line 43, in test_runall
    self.assertEqual(5, stats.num_failures())
AssertionError: 5 != 6

Maybe it is related to #78

[vkarak]

It could be. I've tested only on the branch with the fix of #78.

[vkarak]

With minor changes in the unit tests we can migrate to pytest, which is widely supported and with plenty of plugins.