add support for checkpoint/restart of mtt logfiles

[hppritcha]

This feature adds support for checkpointing MTT logfiles as part of a MTT workflow
wherein some stages of an MTT workflow must be done on one set of system resources -
in particular login nodes to typical HPC DoE systems which have internet access -
but for other stages, such as running tests, MTT has to be run on backend nodes
with no internet access.

Pickle is used to serialize/deserialize the logfile irrespective of its structure.

Signed-off-by: Howard Pritchard howardp@lanl.gov

[mallove79]

Hi Howard -- This checkpoint/restart feature sounds handy. FWIW, we used to use mtt-relay for submitting test results from an offline network.

[ribab]

Can you describe how to use this feature, to help me understand how it works?

[hppritcha]

looks like this needs a rebase and I should add an example.

I use this in the following scenario. To the best of my ability I've not been able to use the existing ASIS keyword support to do what I need to do. Many of our DOE systems are set up as follows:

• front end nodes that can access the internet, aka cherry-py server running at mtt.open-mpi.org at AWS, middleware can be cloned from github, etc, tests can get cloned from github, etc.
• middleware and tests can be built on the frontend nodes
• frontends can't run the tests
• compute nodes can't reach the internet, so no talk with github, mtt.open-mpi.org etc.
• compute nodes may be managed by any number of resource managers, not just slurm
• compute nodes are obtained by requesting an allocation from the resource manager
• tests can be run on the compute nodes

We tried for while at NERSC to use the slurm plugin in MTT but it was a frustrating disaster. We were trying to get an allocation with --no-shell which would keep MTT running on the front end, and we used srun's for each individual test.
This approach was completely unreliable. slurm would sometimes just drop the allocation out from under us, so then all our sruns would just hang for some long time then come back with an error, etc. etc. nersc consult never were able to figure out why this wasn't working. Then srun support dropped out with master for that platform since we can't control how slurm is built there and at the time the install did not support pmix.

Then there was ANL's theta, which uses a batch system which MTT doesn't even know about so I came up with this approach.

Here's how I use it:

• main script runs on head node and runs an initial MTT script that downloads Open MPI, builds it, downloads ompi-tests and builds them.
• the last stage of this MTT script stores the log data via pickle into a restart file.
• the main script submits a job in a blocking fashion to the backend compute nodes which runs another script (backend script)
• backendscript runs on the head node of the allocation (or in the case of ANL theta, on a mom node)
• backendscript submits the second stage of the MTT run, this MTT reads in the restart file from above and has only sections for running the tests and storing a checkpoint of the logfile (which includes the first logfile) as the final step
• batch job ends, hopefully successfully and the main script running on the front end resumes (it was blocked waiting for the batch job to finish)
• this main script runs the final MTT stage
• this final MTT stage reads up the logfile (pickle format) created at the end of the batch job and reports results back to the cherry-py server running at mtt.open-mpi.org

[hppritcha]

added WIP to PR name as this PR should come with some examples.

[hppritcha]

@ribab finally got around to adding some example scripts and ini files showing a way to use this feature.

[hppritcha]

Any issues with merging this PR into master next week?

[ribab]

I'm going to test this with our test content to ensure everything still works

[hppritcha]

Thanks very much Ricky!

[ribab]

this line needs to be commented out, it is causing a syntax error.

[hppritcha]

think I fixed this.

[ribab]

This looks good now -- My test of existing content passed.