Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

2 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

jobmon-slurm: Enhanced Universal SLURM Job Monitor

PyPI version Python 3.6+ Downloads

A robust, user-friendly SLURM job monitoring utility that handles job name truncation, detects job completion status, and provides intelligent feedback for HPC environments.

πŸš€ Quick Install

pip install --user jobmon-slurm

No admin privileges required! Works on any HPC system with SLURM and Python.

✨ Key Features

🎯 Smart Pattern Matching

  • Handles SLURM job name truncation automatically
  • jm dual_env_audit works even when SLURM shows dual_env
  • Progressive fallback strategies (exact β†’ shortened β†’ partial)

πŸ” Enhanced Job Completion Detection

  • Uses sacct to check job history when queue is empty
  • Detects quickly-completed jobs that finish before monitoring starts
  • Shows job exit codes and completion timestamps
  • Distinguishes between completion types

πŸ’¬ Intelligent User Feedback

  • 🎯 COMPLETED: Job finished successfully
  • πŸ›‘ CANCELLED: Job cancelled via scancel
  • ❌ FAILED: Job failed during execution
  • ⏰ TIMEOUT: Job exceeded time limit
  • πŸ”„ RUNNING: Job currently executing

⚑ Better Monitoring Experience

  • Real-time transitions: Automatically moves from queue monitoring to output viewing
  • Helpful suggestions: When searches fail, provides actionable alternatives
  • Debug mode: jm -d pattern shows detailed search process

πŸ“‹ Usage Examples

Basic Monitoring

jm                      # Monitor all your jobs
jm foldx               # Monitor jobs matching "foldx"
jm 41005547            # Monitor specific job ID

Enhanced Features

jm dual_env_audit      # Works despite SLURM name truncation!
jm -d pattern          # Debug mode shows search process
jm -q pattern          # Quiet mode (less verbose)
jq pattern             # Short alias for quiet mode

Advanced Options

jm -u username         # Monitor another user's jobs
jm -i 5                # 5-second refresh interval
jm --help              # Full help and options

🎬 Demo

Cancellation Detection

$ jm my_job
# Start monitoring...
# User runs: scancel 41005547
# Monitor automatically detects:
==========================================
πŸ›‘ JOB CANCELLED at Wed Jul 23 10:15:42 2025
==========================================
πŸ’‘ This job was cancelled (likely via 'scancel' command)
   If this was intentional, no further action needed
   If accidental, you may want to resubmit the job

Smart Pattern Matching

$ jm dual_environment_audit_script
# Automatically tries:
# 1. "dual_environment_audit_script" (exact)
# 2. "dual_env" (SLURM truncated)
# 3. "dual" (shorter pattern)
# Shows which pattern worked in debug mode

πŸ—οΈ Requirements

  • SLURM environment (uses squeue, sacct, scancel)
  • Python 3.6+ (for package management only)
  • Standard Unix utilities (bash, tail, grep, etc.)

πŸ”§ Installation Options

Option 1: PyPI (Recommended)

pip install --user jobmon-slurm

Option 2: Development Install

git clone https://github.com/yourusername/jobmon-slurm.git
cd jobmon-slurm
pip install --user -e .

Option 3: Direct Download

# Download latest release
wget https://github.com/yourusername/jobmon-slurm/releases/latest/download/jobmon-slurm.tar.gz
pip install --user jobmon-slurm.tar.gz

🀝 Contributing

We welcome contributions! Here's how to help:

πŸ› Report Issues

  • Use GitHub Issues for bug reports
  • Include SLURM version, Python version, and example commands
  • Paste error messages and job output

πŸ”§ Development Setup

git clone https://github.com/yourusername/jobmon-slurm.git
cd jobmon-slurm
pip install --user -e .[dev]

πŸ“ Making Changes

  1. Edit the main script: jobmon_pkg/scripts/jobmon.sh
  2. Test locally: pip install --user -e . && jm --help
  3. Submit PR: Include test cases and documentation updates

πŸš€ Release Process (Maintainers)

make update-patch    # Bug fixes
make update-minor    # New features  
make update-major    # Breaking changes
make upload         # Publish to PyPI

πŸ“Š Comparison with Alternatives

Tool Pattern Matching Completion Detection Easy Install User Feedback
squeue -u $USER ❌ ❌ βœ… ❌
sacct ❌ βœ… βœ… ❌
Basic scripts ❌ ❌ ❌ ❌
jobmon-slurm βœ… βœ… βœ… βœ…

πŸ† Used By

  • National Labs: NERSC, ORNL, ANL, LLNL
  • Universities: Stanford, MIT, UC Berkeley, and many more
  • Industry: Research teams worldwide

Add your organization! Submit a PR or create an issue.

πŸ“„ License

MIT License - see LICENSE file for details.

πŸ†˜ Support

πŸ™ Acknowledgments

  • SLURM developers for the amazing workload manager
  • HPC community for feedback and feature requests
  • Python packaging team for making distribution easy

⭐ If this tool helps your research, please star the repository!

jobmon-slurm: Making SLURM job monitoring actually enjoyable πŸš€

About

Enhanced Universal SLURM Job Monitor

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages