Skip to content

V2.0

Choose a tag to compare

@ld32 ld32 released this 01 May 21:09
1d95cf2

New features:
1 simplified log messages,
2 auto save log message from runAsPipeline into file for future review,
3 supporting snakemake and nextflow, # still need work on cromwell workflow
4 more reliable OOT/OOM rerun,
5 add checkRun command to check run status for all the jobs submitted from the current working directory
6 add cancelAllJobs command to forcefully cancel all the jobs submitted from the current working directory
7 supporing both while loop and for loop

Old features:
ssbatch features:
1 Auto adjust memory and run-time according to statistics from earlier jobs
2 Auto choose partition according to run-time request
3 Auto re-run failed OOM (out of memory) and OOT (out of run-time) jobs
4 (Optional) Generate checkpoint before job runs out of time or memory, and use the checkpoint to re-run jobs.
5 Get good emails: ssbatch attaches the content of the sbatch script, the output and error log to email

smart pipeline features:
1 Submit each step as a cluster job using ssbatch, which auto adjusts memory and run-time according to statistics from earlier jobs
2 Re-run OOM/OOT jobs with doubled memory/run-time
3 Automatically arrange dependencies among jobs
4 Email notifications are sent when each job fails or succeeds
5 If a job fails, all its downstream jobs automatically are killed
6 When re-running the pipeline on the same data folder, if there are any unfinished jobs, the user is asked to kill them or not
7 When re-running the pipeline on the same data folder, user is asked to confirm to re-run or not if a job or a step was done successfully earlier
8 For re-run, if the script is not changed, runAsPipeline does not re-process the bash script and directly use old one
9 If user has more than one Slurm account, adding -A or —account= to command line to let all jobs to use that Slurm account
10 When adding new input data and re-run the workflow, affected successfully finished jobs will be auto re-run.Run bash script as smart slurm pipeline