Heavy-test job should compile one test at a time with all available cores, not just one

[casella]

While checking the latest test reports, I figured that there must have been a misunderstanding in my specifications for the heavy-test job.

This is the regular new backend ScalableTestSuite job report:

and this is the corresponding heavy-test new backend ScalableTestSuite job report:

All the code generation tasks take 40% less time in the heavy-test job, I guess because you don't have many cores competing for memory access on one or two DMA memory channels. However, the C-compilation task takes 7X times, I guess because we are not running many clang instances in parallel on the available cores. BTW, this explains why the heavy-tests job really takes forever to run.

My intention was that the simulations in the heavy-tests job run as if you are running them standalone on the workstation: one by one, but using all the available memory and cores. For some reason, this probably ended up with just one core being used.

Can you please fix that?

[casella]

This is the chain of calls:

OpenModelicaLibraryTesting/.CI/Jenkinsfile

Line 638 in 4f2c30f

runRegressiontest('master', 'heavy_tests', '', '', 'ripper1', 'LibraryTestingRipper1DB', false, '', false, false, 'configs/heavy_tests.json', 1)

OpenModelicaLibraryTesting/.CI/Jenkinsfile

Line 967 in 4f2c30f

def runRegressiontest(branch, name, extraFlags, omsHash, dbPrefix, sshConfig, omcompiler, extrasimflags, boolean removePackageOrder, boolean conversionScript, libs_config_file = 'configs/conf.json', int jobs=0) {

in this call jobs = 1. It calls the test.py script here:

OpenModelicaLibraryTesting/.CI/Jenkinsfile

Line 1161 in 4f2c30f

stdbuf -oL -eL time ./test.py --ompython_omhome=/usr ${FMI_TESTING_FLAG} --extraflags='${extraFlags}' --extrasimflags='${extrasimflags}' --branch="${name}" --output="libraries.openmodelica.org:/var/www/libraries.openmodelica.org/branches/${name}/" --libraries='${libraryPath}/.openmodelica/libraries/' --jobs=${jobs} ${libs_config_file} ${params.OLDLIBS ? "configs/conf-old.json configs/conf-nonstandard.json" : ""} || (killall omc ; false) || exit 1

so --jobs gets 1 as well. Then, in test.py

OpenModelicaLibraryTesting/test.py

Line 60 in 4f2c30f

n_jobs = int(args.jobs)

n_jobs gets a value of 1. Later on, single_thread gets a value of ""

OpenModelicaLibraryTesting/test.py

Lines 212 to 216 in 4f2c30f

	if n_jobs == 1:
	single_thread="" # Alternative: single_thread="-n=0"
	else:
	single_thread="-n=1"

which is passed into conf["single_thread_cmd"]

OpenModelicaLibraryTesting/test.py

Line 578 in 4f2c30f

conf["single_thread_cmd"] = single_thread

Finally, that configuration (empty string) is used to set the command line options:

OpenModelicaLibraryTesting/testmodel.py

Line 206 in 4f2c30f

cmd = 'setCommandLineOptions("%s")' % conf["single_thread_cmd"]

But I it is used for the code generation only, which is in fact faster, since some of it is multi-threaded, not for the C compilation. That is handled afterwards by starting make:

OpenModelicaLibraryTesting/testmodel.py

Line 433 in 4f2c30f

res = checkOutputTimeout("make -j1 -f %s.makefile" % conf["fileName"], conf["ulimitOmc"], conf)

which is run with -j1, that I understand means one job at a time, regardless of what the value of jobs was in the call of runRegressiontest()

So I guess if single_tread=="", we should use -j instead.

What I don't get is why the C compile time is 5-7 times smaller with the regular newInst-newBackend job than with the heavy-test job. If both run make with a single thread, the time should be the same.

[adrpo]

We'll see if this is fixed via #140, I started a heavy test.

[casella]

Hopefully it should take much less time

[adrpo]

PR #142 should fix the run in serial.

[casella]

I checked the results of LargeTestSuite_NB and they look really good now, though this task doesn't have much C compilation going on due to the NB generating compact code.

I'll run a job with all libraries to double-check, but I think this issue is fixed. The only missing problem is the handling of codegen timeouts, but that's covered by #135

Thanks @adrpo!