Automating Job Submission and Analysis to a Supercomputer in Python using Slurm
Solar is a program written in Python that automates the submission of jobs to Slurm, a Supercomputer queueing system, and compiles the results in an easily readable format. Solar has the unique ability of automatically generating additonal Python code, ontop of Solar itself, from a simple configuration text file. This makes Solar very useful to those who need to submit many jobs to a Supercomputer and those with limited coding experience. Additionally, Solar also takes away the hassle of manually having to change each jobs parameters before submission to a Supercomputer. Solar makes use of Slurm for job submissions to our Supercomputer but can be easily adapted to your choice of queueing system.
- Numpy
- Slurm - or your choice of queueing system
pip install numpy
https://www.schedmd.com/downloads.php
Solar, as stated above, automatically generates code and automatically sends jobs to a Supercomputer. Additionally, Solar creates and curates log files, results files and raw data that is taken right from the Supercomputer (see Solar Examples).
- Step 1 and 2
- Step 2 in progress
- Step 3 in progress
The configuration file ultimately dictates what Solar does. The picture below showcases a couple lines from the config file that show a comment line, #, that is ignored by Solar followed by a line of three comma separated numbers.
Solar interprets this as a parameter we want to loop and sets up a for loop for it. In this case a for loop starting at 0, ending at 120, in steps of 40 (in the second picture the end is set to 160 (end + step size), this is how range needs to be set up to work properly). If there are multiple lines like this in the config file, Solar will automatically start nesting them.
Run Solar with the example config file to be able to see this in action.
- Mentor : Thomas Carroll @ Ursinus College
-
Logo made with DesignEvo