A robust, user-friendly SLURM job monitoring utility that handles job name truncation, detects job completion status, and provides intelligent feedback for HPC environments.
pip install --user jobmon-slurmNo admin privileges required! Works on any HPC system with SLURM and Python.
- Handles SLURM job name truncation automatically
jm dual_env_auditworks even when SLURM showsdual_env- Progressive fallback strategies (exact β shortened β partial)
- Uses
sacctto check job history when queue is empty - Detects quickly-completed jobs that finish before monitoring starts
- Shows job exit codes and completion timestamps
- Distinguishes between completion types
- π― COMPLETED: Job finished successfully
- π CANCELLED: Job cancelled via
scancel - β FAILED: Job failed during execution
- β° TIMEOUT: Job exceeded time limit
- π RUNNING: Job currently executing
- Real-time transitions: Automatically moves from queue monitoring to output viewing
- Helpful suggestions: When searches fail, provides actionable alternatives
- Debug mode:
jm -d patternshows detailed search process
jm # Monitor all your jobs
jm foldx # Monitor jobs matching "foldx"
jm 41005547 # Monitor specific job IDjm dual_env_audit # Works despite SLURM name truncation!
jm -d pattern # Debug mode shows search process
jm -q pattern # Quiet mode (less verbose)
jq pattern # Short alias for quiet modejm -u username # Monitor another user's jobs
jm -i 5 # 5-second refresh interval
jm --help # Full help and options$ jm my_job
# Start monitoring...
# User runs: scancel 41005547
# Monitor automatically detects:
==========================================
π JOB CANCELLED at Wed Jul 23 10:15:42 2025
==========================================
π‘ This job was cancelled (likely via 'scancel' command)
If this was intentional, no further action needed
If accidental, you may want to resubmit the job$ jm dual_environment_audit_script
# Automatically tries:
# 1. "dual_environment_audit_script" (exact)
# 2. "dual_env" (SLURM truncated)
# 3. "dual" (shorter pattern)
# Shows which pattern worked in debug mode- SLURM environment (uses
squeue,sacct,scancel) - Python 3.6+ (for package management only)
- Standard Unix utilities (
bash,tail,grep, etc.)
pip install --user jobmon-slurmgit clone https://github.com/yourusername/jobmon-slurm.git
cd jobmon-slurm
pip install --user -e .# Download latest release
wget https://github.com/yourusername/jobmon-slurm/releases/latest/download/jobmon-slurm.tar.gz
pip install --user jobmon-slurm.tar.gzWe welcome contributions! Here's how to help:
- Use GitHub Issues for bug reports
- Include SLURM version, Python version, and example commands
- Paste error messages and job output
git clone https://github.com/yourusername/jobmon-slurm.git
cd jobmon-slurm
pip install --user -e .[dev]- Edit the main script:
jobmon_pkg/scripts/jobmon.sh - Test locally:
pip install --user -e . && jm --help - Submit PR: Include test cases and documentation updates
make update-patch # Bug fixes
make update-minor # New features
make update-major # Breaking changes
make upload # Publish to PyPI| Tool | Pattern Matching | Completion Detection | Easy Install | User Feedback |
|---|---|---|---|---|
squeue -u $USER |
β | β | β | β |
sacct |
β | β | β | β |
| Basic scripts | β | β | β | β |
| jobmon-slurm | β | β | β | β |
- National Labs: NERSC, ORNL, ANL, LLNL
- Universities: Stanford, MIT, UC Berkeley, and many more
- Industry: Research teams worldwide
Add your organization! Submit a PR or create an issue.
MIT License - see LICENSE file for details.
- π Issues: GitHub Issues
- π¬ Discussions: GitHub Discussions
- π§ Email: Open an issue instead of email for faster response
- SLURM developers for the amazing workload manager
- HPC community for feedback and feature requests
- Python packaging team for making distribution easy
β If this tool helps your research, please star the repository!
jobmon-slurm: Making SLURM job monitoring actually enjoyable π