LSF Reference
Jobs are submitted to the scheduler as shell scripts using the bsub command.
bsub < myjob.sh
Notice the < redirect. bsub takes commands from stdin, which is why this syntax is required
Resources are requested using the bsub parameters or using 'directives' which live at the top of your script. See the simple example below.
#! /usr/bin/env bash
#BSUB -J test_synergy
#BSUB -n 12
#BSUB -R "span[hosts=1]"
#BSUB -R "rusage[mem=4500]"
#BSUB -M 5000
#BSUB -We 01:00
#BSUB -W 02:00
#BSUB -o test_%J.out
#BSUB -e test_%J.err
sleep 10
echo "finished with exit code $? at: `date`"
Let's walk through this example. All bsub directives start with #BSUB so bsub knows what they are. Each of these directives can alternatively be provided at the command line but for the sake of reproduceability it is recommended that you put these inside you script.
-
-J test_synergy: name of the job (optional, pick something sensible) -
-n 12: number of cores (note that this is total number of cores required across all nodes, should never be more than 56 on this system) -
-R "span[hosts=1]": run job on a single node (most jobs will not be able to run on multiple nodes unless you are using MPI, which you're probably not, so be sure to always include this) -
-R "rusage[mem=4000]": requests 4 GB (listed as MB) of memory for this job, a good default if you're unsure is ~4GB / core requested -
-M 5000: memory limit of 5 GB (listed as MB), your job will not be allowed to use more memory than this -
-We 01:00: runtime estimate for 1 hour, units in HH:MM (effects scheduling, pay attention to this!) -
-W 02:00: runtime limit for 2 hours, same format as runtime estimate, job is killed if it exceeds this value -
-o test_%J.out: file forstdout,%Jis the job id -
-e test_%J.err: file forstderr
Everything else is just like a normal bash script. If you're coming from pbs you'll notice this is very similar. Your environment should be preserved (i.e. $PATH) and all your commands will be run relative to the submit directory.
Note: Use the above example as a template and keep a copy in your home directory. Then simply copy to a new directory for a new job.
Jobs arrays are multiple jobs that are run in a single script. The script runs use the same command but with different inputs. The following is an example script:
#! /usr/bin/env bash
#BSUB -J "test_synergy[1-10]"
#BSUB -n 12
#BSUB -R "span[hosts=1]"
#BSUB -R "rusage[mem=4500]"
#BSUB -We 01:00
#BSUB -W 02:00
#BSUB -o test_%J.out
#BSUB -e test_%J.err
sleep 10
echo $LSB_JOBINDEX >>job_indices.txt
echo "finished with exit code $? at: `date`"
This script will run 10 times and add a number each time (numbers 1-10) to the file job_indices.txt. You can use job arrays to run multiple input files through the same script by numbering your files. Indicate the array indices in square brackets:
- "[1-10]" Runs numbers 1 to 10
- "[1,2,4-10]" Runs numbers 1, 2, and 4 to 10
The %I will be replaced with the array id in the output and error files.
When your job is created, it makes some variables that you can use in your scripts.
-
$LSB_DJOB_NUMPROCThis is the number of cores that you are running the script with. This is set by-n -
$LSB_JOBNAMEName of the job -
$LSB_JOBIDJob Id
Find more here
Example Usage:
Run diamond with the numbers of cores specified by -n
diamond -p $LSB_DJOB_NUMPROC
If you have complicated pipelines (or simple ones too) snakemake is a very good option to help you organize and run them. It can automatically submit your jobs to the cluster for you and can make handling multiple samples very easy.
In some cases it can be handy to run an interactive job for testing and development purposes. bsub -Is -n 1 -W 01:00 bash will put you in the queue with a request for 1 core and 1 hour runtime and when a slot become available it will drop into a node where you can run commands interactivaly like you would on the login node. When you are finished type exit to leave.
❗ Important ❗
Interactive jobs are to be used strictly for testing purposes only. Always set a runtime limit with -W in case you forget to exit when you're done.
An important reason for taking the time and effort to build our own compute system is to remove barriers that can exist with large national systems like Compute Canada. These include storage limits as well as runtime and memory limits and defaults. And so we have tried to remove these barriers as much as possible. So that means we have no runtime or memory limits or defaults and a very large storage system without hard quotas. But if this is to work then we need you on board and to be conscientious user! That means 2 things:
- Be aware of your storage useage.
- Pay attention to the runtime and memory that your job uses and request the correct amount when you submit.
Please read through the details below to learn how to do this effectively.
Runtime estimate is set with the -We parameter. As with all bsub parameters this can be on the commandline or in your batch script with the #BSUB directive as in the above example. The format for runtime is HH:MM so 12:00 means your job is estimated to run for 12 hours. Setting the -W, runtime limit is also recommended. The scheduler will kill your job if it runs longer than the limit time. Provide an estimated time for how long you think your job should run and a limit the absolute longest time your job will take. Using both is recommended for effective scheduling and so your job doesn't tie up resources if it something goes wrong.
To see how long a job actually took look in your log file. You did save the log file, right? Right!? Partway down in the file, if your job completed successfully, you'll find something that looks like this:
Successfully completed.
Resource usage summary:
CPU time : 2130.35 sec.
Max Memory : 4847 MB
Average Memory : 3759.85 MB
Total Requested Memory : 5000.00 MB
Delta Memory : 153.00 MB
Max Swap : -
Max Processes : 5
Max Threads : 35
Run time : 370 sec.
Turnaround time : 364 sec.
The Run time : line will give an idea of how long your job took. Note, don't use the CPU time as that will always be much longer and doesn't represent actual elapsed time. And yes, I know, it's in seconds - just divide by 60 and you'll be golden. Use this value as a starting place to set your runtime estimate for the next time you run a similar job. For example you can see them my job took about 6 minutes so I would estimate 10 minutes for runtime (-We 00:10) and maybe an hour for the runtime limit (-W 01:00).
As indicated above there are no runtime defaults and runtime limits. This means if you don't specify runtime then your job could run forever. However, I strongly recommend setting your runtime for two reasons. 1) Jobs with resource requests (runtime, memory) get priority in scheduling over ones that don't and 2) the system will work much better for everyone if acurate runtimes are used for schedulling. The same is true for memory - see below for more on that.
Much of what is described above for runtime can be applied to memory as well. However, the way you request memory is slightly different. The estimated amount of memory (and I'm refering to RAM here, for you newbies 😉) is given through the resouce requirement string with the -R parameter. This request can actually be very complex but for our purposes it's quite simple ands looks like this -R "rusage[mem=4000]". This will request 4 GB for your job.
A good rule of thumb, if you're unsure about how much memory a job will need is to use ~4GB per requested core (process). This is because our nodes have 256 GB of memory so 256 / 56 = 4.5 and using 4 gives a bit of breathing room. However, once you know how much memory you'll need then you can set it here. If you request more than 256 GB (and most of you won't need to) then your job will be automatically scheduled to run on node036, the large memory node.
The -M parameter sets the memory limit for you job. Unlike the runtime limit, on our system your job won't be killed if goes over the memory limit - it will simply not be allowed to use more memory. However, this means your job may not complete in time or at all. Setting a memory limit is recommend as it will keep runaway jobs under control and keep them from interfering with other jobs. So use -R for what you think you'll need and -M for "my job should never use more than this".
As for how much memory your job actually used, again you can refer to your log file. This reports max, average, and amount requested and tells you if you went over or under the amount requested. Please refer to this log file often and use to more accurately set memory and runtime requests - it's to your own benefit to do so.
bjobs will show you your current pending and running jobs.
- Include recently finished jobs:
bjobs -a - Full output:
bjobs -l - Show all users:
bjobs -u all
The bhist command can be used to get a list your previous jobs. The most useful column in the default display is RUN which shows the total run time in seconds. Use bhist -l to show long format, which has loads more info and is most useful when you provide the jobid of the job you'd like to look a: bhist -l 9999.
The bacct command will give you some stats on your job accounting - how much resources you've used, etc.
https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0