RFE: randomize test execution order

[flx42]

I don't think this is possible right now. It would be useful to be able to add a CLI argument to randomize the execution order of all the tests.

Ideally it would randomize the tests across all files passed to Reframe, e.g. with -c test1.py -c test2.py -c test3.py it would randomize and interleave tests between all 3 files, not just within each file. To replay particular sequence of execution, perhaps the argument could accept an optional seed for the RNG.

The rationale is that some test A might change the state of the system like making the system very hot and throttling clocks, that could cause a failure in a later test C only if you run A -> C -> B instead of always executing A -> B -> C, for example if test B is long but does not stress the system (giving the system the time to cool down).
Similarly, if the same Reframe test starts in parallel on 10 machines, they might all reach a network test at the same time, creating heavy congestion (which is useful in itself, but might not be the target of the test).

If possible, it would also be great if the performance report printed would always stay in the same order to make it easier to compare runs despite the randomized execution order.

[vkarak]

Hi @flx42 your request makes sense. Technically, I would go as far as saying that this could be implemented right away without an option even, since there is no guarantee from the framework that your tests will execute in a specific order unless your tests define dependencies. So randomising the order of execution in any case to avoid conflicts and potentially also increase the overall execution throughput of the whole suite (especially if you have a mixture of build-and-run and run-only tests) could make sense. Just to find counter arguments to this, are you now taking any advantage of the implementation-specific behaviour of running tests in order?

[flx42]

Just to find counter arguments to this, are you now taking any advantage of the implementation-specific behaviour of running tests in order?

Right now, I don't.

But if it's not possible to have the performance report listing tests in a canonical order, then I suppose I would run some tests as randomized and some tests as non-randomized.

I imagine it would not be possible to have a fixed ordering for perflog handlers as to my knowledge the handlers are immediately at the end of each test and thus depend on the order of execution. Perhaps some folks depend on the execution order of the perflog handlers to write to a log file and expect the same order every time.

[vkarak]

I imagine it would not be possible to have a fixed ordering for perflog handlers as to my knowledge the handlers are immediately at the end of each test and thus depend on the order of execution. Perhaps some folks depend on the execution order of the perflog handlers to write to a log file and expect the same order every time.

It's not possible to impose an order at this level. If you need this, you must define dependencies between the tests. Same for the run and performance report. I am not sure they have any implicit order. In any case, the order is implementation defined, so I will suggest to go on with the randomisation step. Another scenario that this will be useful is when you try to distribute two-node tests on all pairs of a set of nodes. If you don't randomise, then you will end up submitting lots of jobs that can't proceed until you reach the point that one of the nodes in the pair is not occupied. If job submission happens also to not be instantaneous then you lose time and the overall throughput drops.

[flx42]

For the performance report, I thought reordering tests would be possible since it waits until all tests are finished before printing, right?

[vkarak]

Indeed, but imho the sorting criterion must be better defined than being just the load order. ;-) Perhaps, we could just sort and present them in alphabetical order.

[vkarak]

@flx42 With #2575, you should be able to achieve the desired effect by passing --exec-order=random.

[flx42]

Awesome, thanks! I will try that next week.

[flx42]

It seems to work as intended, but I got tripped by the new test naming scheme.