ReFrame fails to detect job status of Slurm job arrays

[bcfriesen]

When running a Slurm job array using #SBATCH --array=<array_specs>, ReFrame fails to detect the status of the job from its sacct polling. The logs include things like:

[2019-06-20T12:18:49] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: entering stage: poll
[2019-06-20T12:18:49] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: executing OS command: sacct -S 2019-06-20 -P -j 2242
3095 -o jobid,state,exitcode,nodelist
[2019-06-20T12:18:49] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: job state not matched (stdout follows)
JobID|State|ExitCode|NodeList
22423095_1|RUNNING|0:0|nid0[2544-2545]
22423095_1.extern|RUNNING|0:0|nid0[2544-2545]
22423095_0|RUNNING|0:0|nid0[2517-2518]
22423095_0.extern|RUNNING|0:0|nid0[2517-2518]

and

[2019-06-20T12:19:04] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: entering stage: poll
[2019-06-20T12:19:04] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: executing OS command: sacct -S 2019-06-20 -P -j 22423095 -o jobid,state,exitcode,nodelist
[2019-06-20T12:19:04] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: job state not matched (stdout follows)
JobID|State|ExitCode|NodeList
22423095_1|COMPLETED|0:0|nid0[2544-2545]
22423095_1.batch|COMPLETED|0:0|nid02544
22423095_1.extern|COMPLETED|0:0|nid0[2544-2545]
22423095_1.0|COMPLETED|0:0|nid0[2544-2545]
22423095_0|COMPLETED|0:0|nid0[2517-2518]
22423095_0.batch|COMPLETED|0:0|nid02517
22423095_0.extern|COMPLETED|0:0|nid0[2517-2518]
22423095_0.0|COMPLETED|0:0|nid0[2517-2518]

I wonder if ReFrame gets confused by the output having the form <job_id>_<array_task_num> as opposed to just job_id?

[vkarak]

@bcfriesen Thanks for reporting this. Yes, it fails to detect it... Unfortunately, I don't think there is currently a workaround to this, but the fix should be easy.

[teojgo]

@bcfiesen Can you share the test that creates the problem?

[bcfriesen]

Hi @teojgo,

@bcfiesen Can you share the test that creates the problem?

Sure - the relevant part of the Python script is here:

@rfm.simple_test
class slurm_job_arrays(rfm.RunOnlyRegressionTest):
    def __init__(self):
        super().__init__()
        self.descr = ('Test Slurm job arrays')
        self.valid_systems = ['cori:knl']
        self.valid_prog_environs = ['PrgEnv-intel']
        self.sourcesdir = './src'

        self.executable = './job_array.sh'
        self.sanity_patterns = sn.assert_found(r'my job array task ID = 1', self.stdout)
        self.maintainers = ['bfriesen@lbl.gov']
        self.tags = {'slurm'}
        self.extra_resources = {'job_array': {'array_arg': '0-1'}}

        self.num_tasks = 16
        self.num_nodes = 2
        self.num_tasks_per_node = 8
        self.num_cpus_per_task = 4
        self.time_limit = (0, 1, 0)

and here is the job_array.sh part:

#!/bin/bash

printf "%s %d\n" "my job array task ID =" "${SLURM_ARRAY_TASK_ID}"

The extra_resources part of the compute partition is defined like this:

                        'resources': {
                          'job_array': ['--array={array_arg}'],
                        },

[teojgo]

@bcfriesen thank you for the update. I needed it to test the fix that I want to implement.

[teojgo]

@bcfriesen Can you check if #840 works for you?

[bcfriesen]

@bcfriesen Can you check if #840 works for you?

Hi @teojgo - yes this fix works! Thanks for the quick response. Feel free to close this issue.

[teojgo]

@bcfriesen #840 was merged. Can you please confirm that it works fine?

[bcfriesen]

Hi @teojgo,

I tested commit cdd4501 and it failed with my job arrays test. The log says:

[2019-07-10T13:38:45] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: spawned job finished
[2019-07-10T13:38:45] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: entering stage: sanity
[2019-07-10T13:38:45] debug: slurm_job_arrays on cori:knl using PrgEnv-intel: caught reframe.core.exceptions.SanityError: pattern `my job array task ID = 1' not found in `rfm_slurm_job_arrays_job.out'
[2019-07-10T13:38:45] info: reframe: [     FAIL ] slurm_job_arrays on cori:knl using PrgEnv-intel

It looks like the generated job script has this:

#SBATCH --output=rfm_slurm_job_arrays_job.out_%a
#SBATCH --error=rfm_slurm_job_arrays_job.err_%a

which produces separate output files for each task in the job array. In the staging directory I see

rfm_slurm_job_arrays_job.out_0
rfm_slurm_job_arrays_job.out_1

so ReFrame is looking in the wrong output file (rfm_slurm_job_arrays_job.out; see above). Did I do something wrong?

[teojgo]

Hi @bcfriesen,

The %a is appended on purpose during the generation of the sbatch script so that each job array task produces its own output. After the job finishes, the above outputs are merged at the .out file so that and ReFrame is looking in this merged file. It seems that in your case, although ReFrame correctly detects that you are submitting a job array, it fails to merge the files. Can you check if the .out file is produced in your stage directory? If not please paste the .log file here.