JobHelper helps you run Python scripts on high-performance computing (HPC) clusters more easily. It solves two main problems:
When running scientific code, you often need to handle many input parameters. Typing them all on the command line is tedious and error-prone. JobHelper lets you store parameters in configuration files (YAML, JSON, or TOML) and manage them easily.
Many workflows need multiple programs to run in a specific order, where some jobs depend on others finishing first. Job schedulers like Slurm support this, but you need to track job IDs manually. JobHelper automates this by managing the dependency graph for you.
Convert your Python scripts into a JobArgBase class. This gives you:
- Automatic saving/loading of parameters to configuration files
- Instant command-line interface (using
python-fire) - Easy parameter management
Put all your job configurations in one file, including which jobs depend on others. Then use jh project to submit all jobs to the cluster in the correct order automatically.
To get started with JobHelper, first create a new project: Run it with:
mkdir my_project
cd my_project
jh initThen you will see a sample project structure created for you. It includes the following files:
cli.py
log/
project.yaml
pyproject.toml
You can use the following command to take a look at the jobs to be run:
jh project from-config project.yaml - runIf everything looks good, submit the jobs to the cluster with:
jh project from-config project.yaml - run --nodryTake a look at tutorial/tutorial.md to learn how to scale existing programs / scripts in HPC with
JobHelper.
This project includes a SKILL.md file that provides a structured guide for Large Language Models (LLMs). You can use this file to help an LLM automate the conversion of your plain Python scripts into JobArgBase classes and generate the corresponding project configurations.
If your LLM interface supports "skills" or "system prompts," you can provide the content of SKILL.md to ensure the model follows the library's best practices.
For a complete hands-on tutorial that covers all the details, see tutorial/tutorial.md.
pip install git+https://github.com/szsdk/jobhelper.gitjh init- Set up a new project directoryjh project from-config <file>- Load and run jobs from a configuration filejh server- Start a web server to view project resultsjh viewer- View a project file interactivelyjh config- Display JobHelper configurationjh tools- Additional utility commands
For detailed help on any command, run jh <command> --help.
MIT