Automated Calculation Submission Protocol

[GabrielBram]

This protocol seeks to automate the transfer and submission of calculations from local machines to the desired HPC. The long term aim of this subproject is to enable users to submit and run calculations entirely from a local Python interface, and call back the files for analysis. This should enable the simple generation of systematic and high-throughput without some of the more fiddly aspects of transferring files and parameters through the terminal.

I currently have a crude version in Jupyter Notebook which just about works. This version simply calls bash commands from the IPython shell. Furthermore, an interface is implemented to automatically generate the submission script, with function calls to change appropriate parameters. Sensible defaults for the most commonly used packages will be added in future. The intended calculation (in the form of a Python script) is transferred from the Notebook to a .py script, which in turn is copied to the desired file and transferred via. SCP. The submission can then be invoked by a call function, which submits the calculation via. SSH. In future, a suite of functions designed to recall files to the local machine will be implemented, which should make data analysis and backing up data more convenient.

However, this version is not as automated as desired - setting a range of calculations which require subtly different inputs but different initialisations still involves manually changing the parameters rather than something more sophisticated (like looping through a list of options). Furthermore, the code needs to be outfitted with appropriate warnings and crashes, and I imagine there are a substantial number of pitfalls one can fall into without them.

Furthermore, where initial structures need reorienting (especially relevant for adsorption studies), there is still the rather laborious process of manipulating the structure to give an appropriate starting guess. As such, structures still need to be supplied on an individual basis if they cannot be generated through ASE or other Python interfaces. Automating this is beyond the scope of this subproject, but this may be remedied by future work in adsorbate rotator, or another tool I am not yet aware of.

[ikowalec]

@GabrielBram I'd be keen on seeing the current solutions to this, I know we spoke about this in the past. I have been playing around with Django quite a lot and wonder if we could put together an app which could handle this via dynamically generated jupyter notebooks or other browser-based interface?

First step: Rotating a structure in jupyter view --> input gen -->submission --> receiving a number back at the local machine?