Support for multiple steps in job scripts

[vkarak]

Internal issue: https://madra.cscs.ch/scs/reframe/issues/127

This will allow a better implementation of the HDF5 check.

Things to consider:

• Different launchers? Ability to run local commands before or after a job step.
• Pass different options to the launcher?

[vkarak]

I was thinking the following for this. We could allow users to define a custom job script template as follows (either at the RegressionTest level or at the Job level):

# in setup
self.job.script_template = '''{rfm_job_preamble}
{rfm_environ}
any_cmd_you_like
for i in $(ls /myfiles); do
    echo $i
done
{rfm_run} {rfm_executable} {rfm_executable_opts}

# second job step
{rfm_run} myprog --foo
'''

The default job script template is the following:

self.job.script_template = '''{rfm_job_preamble}
{rfm_environ}
{rfm_prerun}
{rfm_run} {rfm_executable} {rfm_executable_opts}
{rfm_postrun}
'''

The advantages of this approach are

1. It gives you the maximum flexibility in generating the job script file
2. It still hides the system-level details
3. It makes clear what is generated and where
4. It doesn't bloat the RegressionTest API.
5. If we adopt the f-strings introduced in Python 3.6, this mechanism can become really powerful and will give us a nice integration of the python code and the generated shell script code.

The disadvantages are

1. You have to write a script template if you want to specify multiple job steps and you might forget some placeholders
2. It is perhaps too early to require Python 3.6.
3. It gives you perhaps too much power?

What do you think?

[vkarak]

Regarding the template, it is not perhaps absolutely necessary. Currently, one could use pre_run as follows:

self.pre_run = """
any_cmd_you_like
for i in $(ls /myfiles); do
    echo $i
done
""".splitlines()

Of course, this doesn't solve the multiple job steps issue, but it weakens the major argument in favour of the script template, which is the ability to easily manipulate the job script.

[vkarak]

We could nicely support multiple job steps with what we have now, if we had kept public the pre_run and post_run attributes of the Job. Check this out:

    def setup(self, partition, environ, **job_opts):
        super().setup(partition, environ, **job_opts)
        launch_cmd = ' '.join(self.job.launcher.command(self.job))
        self.job._pre_run += ['%s cat /proc/cpuinfo' % launch_cmd,
                              '%s nvidia-smi' % launch_cmd]

There is no way currently that we can achieve this using the pre_run attribute of RegressionTest. This is copied inside the Job's constructor and never referenced thereafter. Perhaps, a solution would be not to copy RegressionTest.pre_run but pass a reference to it.

[teojgo]

@vkarak why not pass a list as a command and a list of lists as executable options associated to the commands?

[vkarak]

@teojgo I think this is a bit restrictive. What if somebody wants to run sth else in between? What would be the meaning of pre_run and post_run?

self.executable is currently already restrictive and I don't want to extend the API in another restrictive way.

[vkarak]

I think my suggestion here can be implemented without the need to use the private fields of Job by moving the job descriptor creation as late as possible.

[teojgo]

Do you suggest the template one or the one with the pre_run? It is not quite obvious from the comment.

[vkarak]

@teojgo No the solution with the pre_run.

[teojgo]

Then we would need to make it a list to be able to mutate it.

[vkarak]

No, I am not talking about the Job._pre_run attribute. The problem comes from the fact the we construct the job descriptor, i.e., the Job instance inside the setup() method. When constructing the Job we pass it the pre_run arguments.

Ideally, we could move the creation of the Job inside the run() just before submitting the job. Then we solve this problem, but we create another one: we cannot set custom job options (self.job.options) in the setup()...