RunSpacesDesign

Design for the structure of the 'run' folder on hemelb systems

This is where input and output files of hemelb get put during runs.
The local run folder is a temporary copy of the corresponding run subfolder on blood.
Files get copied to and from blood as part of run scripts.

blood/hemelb-runs/date-based-name : input/output folders for all historical runs
blood/hemelb-configurations/meaningful-name : input.dat and input.xml files for potential runs.

fab legion run meaningful-name

or locally

hemelb-system/run meaningful-name

This will:

Generate a unique, date-based, stable name for the run
Create folders on blood and on run machine for the run
Copy the input.dat and input.xml from blood:hemelb-configurations to those folders
Run the job on run machine
[The run machine run script, after completion then:]
Drops a file in the run-machine and blood run folders to signify completion
Copies the output data back to blood.
Delete completed and backed up historical runs to keep the number of historical run folders below some limit.

One key feature of this should be lots of output to terminal because there are likely to be lots of things that go slightly wrong in early versions [JH: Scripts should have a --verbose switch, defaulting to highly verbose to begin.]
Have you considered using sshfs to make the file movement that little bit easier? [JH: Fabric has robust wrappers for scp, but it might be nice to have an option to use a mount if available.]
I'm assuming that you really mean "The local run folder is a temporary copy of a subfolder of the master run folder on blood" [JH: yes, amended.]
I'd prefer to have historical runs separated into those that were successfully backed up and those that weren't, then delete only those that were successfully backed up to blood [JH: Yes. Amended design.]