Problem with scheduler

[laraPPr]

With the newer versions of ReFrame it cannot keep track of the submitted jobs with SLURM on our system at Ghent University.

With ReFrame version 4.5.1 everything runs with 'scheduler': 'slurm'

If i use the same sheduler with ReFrame 4.6.2 than the jobs get submitted but ReFrame does not see when they are completed. So ReFrame is endlessly waiting for the jobs to finish

if i use squeue as the scheduler with ReFrame 4.6.2 than the jobs get submitted but reframe already does the sanity check of the output before the jobs are finished and so all the tests fail while the submitted jobs keep running on the system.

[teojgo]

@laraPPr could you try adding check=True to

reframe/reframe/core/schedulers/slurm.py

Line 495 in fe588bd

completed = osext.run_command('squeue -h -j %s -o %%r' % job.jobid)

to see if the problem comes from this?

[laraPPr]

@laraPPr could you try adding check=True to

reframe/reframe/core/schedulers/slurm.py

Line 495 in fe588bd

completed = osext.run_command('squeue -h -j %s -o %%r' % job.jobid)

to see if the problem comes from this?

like this? completed = osext.run_command('squeue -h -j %s -o %%r' % job.jobid, check=True)

Tried it with both squeue and slurm and saw the problem that I had before.

[teojgo]

@laraPPr could you try adding check=True to

reframe/reframe/core/schedulers/slurm.py

Line 495 in fe588bd

completed = osext.run_command('squeue -h -j %s -o %%r' % job.jobid)

to see if the problem comes from this?

like this? completed = osext.run_command('squeue -h -j %s -o %%r' % job.jobid, check=True)

Tried it with both squeue and slurm and saw the problem that I had before.

Yes, like you did. Then the issue is coming from a different place...

[teojgo]

@laraPPr can you try also with 4.6.0 and see it it happens?

[laraPPr]

No it does not than it works with slurm and squeue as scheduler

[teojgo]

No it does not than it works with slurm and squeue as scheduler

I cannot understand if it works fine in 4.6.0 from the above.

[vkarak]

So it works for 4.6.0 and not for 4.6.2? We could bisect the changes and find the offending commit.

[laraPPr]

Also does not work in 4.6.1.
Is their a way to get the output of the sacct command in the run stage?

From what I understand it seems to change the environment when entering stage: run. And from that environment it is no longer able to successfully run 'sacct' and scancel on our system. But the first time it does the entering stage: run it does work for the sbatch commando.

And excerpt from the log

Entering stage: compile
Entering stage: compile
Entering stage: compile_wait
Entering stage: run
Copying PATH_TO/EESSI/test-suite/eessi/testsuite/tests/apps/lammps/src/rhodo to stage directory
Symlinking files: ['data.rhodo']
Generating the run script
[CMD] 'sbatch --export=None --clusters=skitty rfm_job.sh'
Spawned run job (id=7661032)
Entering stage: run
Copying PATH_TO/EESSI/test-suite/eessi/testsuite/tests/apps/lammps/src/lj to stage directory
Symlinking files: []
Generating the run script
[CMD] 'sbatch --export=None --clusters=skitty rfm_job.sh'
Spawned run job (id=7661033)
[CMD] 'sacct -S 2024-10-09 -P -j 7661032,7661033 -o jobid,state,exitcode,end,nodelist'
Entering stage: run
Entering stage: run
[CMD] 'sacct -S 2024-10-09 -P -j 7661032,7661033 -o jobid,state,exitcode,end,nodelist'
Entering stage: run
Entering stage: run
[CMD] 'scancel 7661032'

[laraPPr]

And yes it works for 4.6.0

[vkarak]

This is very strange, the only change in the Slurm backends between these two versions is #3184 and it should affect the squeue backend not the slurm that you are using.