UsageGuide.txt

BBMap/BBTools usage guide.
Last updated December 11, 2015

Table of Contents

System Requirements
Installation
Terminology Notes
Usage
Standard Syntax
Paired Reads
Multiple Output and % Symbol
File Formats
Piping
Memory and Java Flags
Threads
Subprocesses
Additional Help
Standard flags
Help Flags
Config Flags
Input Flags
Output Flags
Sampling Flags
Compression Flags
Quality-Related Flags
Length-Related Flags
Histogram Flags
Advanced Flags
Buffer Flags
MPI and JNI Flags


System Requirements

BBTools is written in Java and requires Java 7 or higher for full functionality.  It is tested on Oracle's JDK, not OpenJDK.  Most tools will work with Java 6 (if not, they will throw a ClassNotFound exception), and most tools will work with OpenJDK, but if you experience a problem with Java 6 or OpenJDK it is recommended that you install Oracle's latest JDK.  All operating systems that support Java are supported.  Note that many of the tools require a substantial amount of memory, so the 64-bit JDK should be installed if you are using a 64-bit operating system.  You can determine your current Java version like this:

java -Xmx90m -version

Installation (for non-NERSC users; if you don't know what NERSC is, you are not a NERSC user)

BBTools can be installed by downloading the gzipped tar file from Sourceforge (http://sourceforge.net/projects/bbmap/files/latest/download)  and decompressing it. In Linux, the command would be:

tar xvzf BBMap_35.74.tar.gz

Then, optionally, you can export the path of the shellscripts to your environment to make it easier to run.  The Java code is already compiled and does not need recompilation. Source code is also available from bitbucket (https://bitbucket.org/berkeleylab/jgi-bbtools), but this is currently a private Berkeley account.

BBTools includes bash wrapper scripts to make the command lines shorter. The package also contains C code that can accelerate certain programs, and experimental MPI code that can make it difficult to compile BBTools on systems without MPI support.  None of these is required.  But, you can accelerate BBMerge, Dedupe, and BBMap by following the instructions in /jni/README.txt if you have a C compiler.


Installation (for NERSC users)

BBTools is in Shifter, which can be used on Denovo and Cori with no installation.  If you want to run the command "reformat.sh in=x.fq out=y.fq", the syntax would be:
shifter --image=bryce911/bbtools reformat.sh in=x.fq out=y.fq


Terminology Notes

"Read" in this file is used synonymously with "sequence", whether it is contig in a fasta file or a short read produced by a sequencing platform.  "Paired reads" or "pair" refer to 2 reads that are generated by sequencing both ends of a single fragment of DNA.  These are typically delivered in two fastq files, named something like "read1.fastq.gz" and "read2.fastq.gz".  The alternative is single-ended reads, in which only one end of the molecule is sequenced.  When paired reads are available, it is important to always process them together, rather than for example mapping the read 1 file and the read 2 file in two separate processes.


Usage

Most BBTools use the same syntax and operate with a set of standard flags.  Individual tools also have specific flags - for example, kmer-based tools support the flag "k" to specify the kmer length, and non-kmer-based tools don't.  This guide describes the standard syntax and most common flags.  Custom syntax and flags for a given tool are described in that tool's shellscript.


Standard Syntax

Most BBTools (such as Reformat or BBNorm) process genomic sequences in some fashion, and are executed like this:

reformat.sh in=reads.fq out=processed.fq

The shellscript allows autodetection of memory (in some cases) and classpath.
The above command is equivalent to this:

java -ea -Xmx200m -cp /path/to/bbmap/current/ jgi.ReformatReads in=reads.fq out=processed.fq

Note that “/path/to/bbmap/current/” needs to be replaced with an actual path.  While the shellscript will only work in bash (or some other Linux/Unix/MacOS shells),
the full Java command will work on any environment with Java installed, such as Windows.

Tools that use a reference (such as BBMap, BBDuk, and Seal) will also need the additional flag "ref=":

bbmap.sh in=reads.fq out=mapped.sam ref=genome.fasta

In each of the above cases, the flags can be arranged in any order.  


Paired Reads

Most BBTools support paired reads.  These may be in two files, or interleaved in a single file, which BBTools will autodetect based on the read names.  When the reads are in two files, you can use the in2 and out2 flags, like this:

reformat.sh in1=read1.fq in2=read2.fq out1=processed1.fq out2=processed2.fq

It is also possible to specify paired files like this:

reformat.sh in=read#.fq out=processed#.fq

...which is equivalent to the above command.

It is important to process paired files together in one command so that they are kept in the proper order.  If you have dual input files and only 1 output file, the output will be written interleaved, and vice-versa.  All tools that support paired reads will keep pairs together.  For example, Reformat supports subsampling; if read 1 is discarded, read 2 will also be discarded.  This prevents a loss of synchronization that corrupts the output.


Multiple Output and % Symbol

Some tools (such as Seal, BBSplit, BBMap, Dedupe) can use the % symbol as a wildcard, to be replaced by some other word when generating many files from a single input.  It is recommended that the % symbol be avoided in filenames.  As an example, assume you run Seal to bin some reads based on matching sequences in the fasta file "ref.fa", which contains the genomes of e.coli and salmonella:

seal.sh in=reads.fq pattern=out_%.fq ref=ref.fa

This would produce the output files "out_e.coli.fq" and "out_salmonella.fq".


File Formats and Extensions

BBTools support most standard sequence formats, including fastq, fasta, fasta+qual, scarf, sam, and (if samtools is installed) bam.  They also support gzip and (if bzip2 or pbzip2 is installed) bzip2.  The tools are sensitive to file extensions.  For example:

reformat.sh in=reads.fq.gz out=processed.fa

In this case, reformat will try to read a gzip-compressed fastq file and output an uncompressed fasta file.  For BBMap, this means that it will output a sam file if you name the output ".sam", bam if you name it ".bam", fastq if you name it ".fastq", and so forth.  BBTools are usually capable of autodetecting input format (for example, if you feed it a fasta file called "stuff.txt" it will be able determine that it is in fasta format), but this is not recommended.  Also, it is possible to specify an extensionless name for an output file, in which case the default format is used; the default varies by tool.

List of supported file extensions:

Fastq: fastq, fq
Fasta: fasta, fa, fas, fna, ffn, frn, seq, fsa, faa
Bread: bread (BBMap internal format; deprecated)
Sam: sam
Bam: bam
Qual: qual (should be accompanied with fasta)
Scarf: scarf (an old Illumina format; input only)
Phylip: phylip (only supported by phylip2fasta; input only)
Text: txt (used for logs, stats, and histograms)
Header: header (use this extension to write read names only)
One Line: oneline (name <tab> sequence)
Sketch: sketch (for Sketch tools).

List of supported compression extensions:

Gzip: gzip, gz
Bzip2: bz2
Zip: zip

Piping and Screen Output

Most tools can accept input from stdin and write output to stdout, with notable exceptions being BBNorm and Tadpole in some processing modes, which require reading the input file multiple times.  Piping works like this:

cat reads.fq.gz | reformat.sh in=stdin.fq.gz out=stdout.fa int=f > x.fa

Note that the extensions are added to stdin and stdout so that Reformat knows how to interpret the data; when piping, it cannot first autodetect the file format.  Similarly, it cannot autodetect whether the reads are interleaved or not.  So, "int=f" (equivalent to "interleaved=false") was added to force it to treat the data as single-ended.

By default, all tools write status information to stderr, not stdout.  To capture a program’s screen output, do this:

reformat.sh in=a.fq out=b.fq 1>out.txt 2>err.txt

Or, to direct both to a single file:

reformat.sh in=a.fq out=b.fq 1>out.txt 2>&1

Memory and Java Flags

There are two flags that are passed by the shellscripts directly to Java rather than to BBTools, "-Xmx" and "-da".
Java does not dynamically grow virtual memory as needed like C programs.  The amount of virtual memory must be specified up front, and it will immediately be grabbed; the physical memory used will only increase as needed.  The shellscripts will try to autodetect memory and set it to an appropriate value, but sometimes this will need to be overridden (for example, if you are using a shared node and don't really need all the memory, or not enough memory was allocated and the program crashed with a memory exception).  To force a program to use 3 gigs of RAM, use the flag "-Xmx3g".  For example:

reformat.sh in=reads.fq out=processed.fq -Xmx3g

That's the equivalent of:

java -ea -Xmx3g -cp /path/to/bbmap/current/ jgi.ReformatReads in=reads.fq out=processed.fq

The "-ea" flag means "enable assertions", which will make BBTools crash if they detect a problem.  If you want to ignore the problem and force it to run anyway, you can use the "-da" flag.  The -da flag may also increase speed slightly.


Threads

Most BBTools are multithreaded, and will automatically detect and use all available threads.  This is usually desirable when you have exclusive use of a computer, but may not be on a shared node.  The number of threads can be capped at X with the flag "t=X" (threads=X).  The total CPU usage may still go higher, though, due to several factors:
1) Input and output are handled in separate threads; "t=X" only regulates the number of worker threads.
2) Java uses additional threads for garbage collection and other virtual machine tasks.
3) When subprocesses (such as pigz) are spawned, they also individually obey the thread limit, but if you set "t=4" and the process spawns 3 pigz instances, you could still potentially use over 16 threads - 4 worker threads, 4 threads for each pigz process, plus other threads for the JVM and I/O.
If you have exclusive use of a computer, you don't need to worry about spawning too many threads; this is only an issue with regards to fairness on shared nodes.


Subprocesses

If they are installed, BBTools will automatically use samtools for sam<->bam conversion, and bzip2 or pbzip2 for processing bz2 files.  It may use pigz to accelerate processing of gzipped files, depending on the number of threads available.  This is generally fine on a standalone computer, but in some circumstances, depending on the cluster configuration, the scheduler may kill a process that spawns a subprocess for violating virtual memory limits (Amazon instances may do this).  In that case, you can disable pigz support with the flags "pigz=f unpigz=f".  Alternatively, you can force pigz to be used with "pigz=t unpigz=t".  The default for those flags depends on the tool.  Pigz processes will never be spawned unless the number of threads allowed is at least 3.  It is also possible to spawn gzip instances instead of pigz instances, but this only gives a small speed increase over using Java for gzip processing.


Additional Help

There are many forum threads on SeqAnswers describing the usage of different BBTools, linked from this thread:
http://seqanswers.com/forums/showthread.php?t=41057


*Standard flags*

The flags below work with many or all BBTools that process reads, but the list is not complete because it does not include flags specific to only one or a few tools - those are listed in the shellscript.  They are listed with their default setting, but some of the defaults differ between tools; the specific default is also listed in the shellscript.  Where the description starts with something in parentheses, like "(in1)", that is an acceptable alternative version of the flag.


Flag Syntax

With the exception of certain special flags like help flags (--help, --version) and Java flags (-Xmx, -da), all flags are in the same format: “a=b” where “a” is the name of the flag, and is not case-sensitive, and “b” is the value, which is case-sensitive (for filenames).  Flags may be in any order, and never need leading hyphens, except for those special flags mentioned above.  If a flag is set twice, the later value will override the former; for example, “reformat.sh in=x.fq in=y.fq” will use y.fq as input.  The special value “null” means “blank”.  For example, “reformat.sh in=a.fq out=null” and “reformat.sh in=a.fq” are equivalent - neither will output anything.
For boolean variables, “null” is equivalent to “true”, and values may be abbreviated “t and f.  So, these are all equivalent:  

Help Flags

--help                      Print the usage information from the shellscript (when run from a shellscript).  Alternately you can just look at the shellscript with a text editor.
--version               Print the version of BBTools.

Config Files

config=<file>           A file or a comma-delimited list of files.  If this flag is present, the contents of the config file will be added to the command line.  Config files must contain one argument per line.  Config files are never required, but may be useful when a command line would be too long or when arguments contain whitespace.  See readme_config.txt for more information.

Input Flags

in=<file>               (in1) Main input.
in2=<file>              Input for 2nd read of pairs in a different file.
interleaved=auto        (int) t/f overrides interleaved autodetection.
samplerate=1            Set lower to only process a fraction of input reads.
qfin=<.qual file>       Read qualities from this qual file, for the reads coming from 'in'
qfin2=<.qual file>      Read qualities from this qual file, for the reads coming from 'in2'
extout=                 Allows overriding of input file format.  For example, "extin=.fq" would force the input to be read in fastq format regardless of the file name.
trimreaddescription=f   (trd) Trim the names of reads after the first whitespace.
touppercase=f           (tuc) Convert lowercase letters in reads to upper case.
lowercaseton=f          (lctn) Convert lowercase letters in reads to N.
utot=f                  Convert U bases to T.

Output Flags

out=<file>              (out1) Main output.
out2=<file>             Output for 2nd read of pairs in a different file.
qfout=<.qual file>      Write qualities from this qual file, for the reads going to 'out'
qfout2=<.qual file>     Write qualities from this qual file, for the reads coming from 'out2'
extout=                 Allows overriding of output file format.  For example, "extout=.fq" would force the output to be written in fastq format regardless of the file name.
fastawrap=70            Length of lines in fasta output.
overwrite=f             Allow overwriting of existing files.
append=f                Append to existing files.

Sampling Flags

reads=-1                Set to a positive number to only process this many input reads (or pairs), then quit.
samplerate=1            Randomly output only this fraction of reads; 1 means sampling is disabled.
sampleseed=-1           Set to a positive number to use that prng seed for sampling (allowing deterministic sampling).  A negative number will use a random seed.

Threading Flags

threads=auto            (t) Number of worker threads to spawn.

Compression Flags

ziplevel=2              (zl) Compression level for zip or gzip output; 1-9.
unpigz=                 Spawn a pigz process for faster decompression.  Requires pigz to be installed.  Valid values are t or f; the default varies by program.
pigz=                   Spawn a pigz process for faster compression.  Requires pigz to be installed.  Valid values are t, f, or a number; the default varies by program.  "pigz=X" will enable pigz, and also force all pigz processes to use exactly X threads.

Quality-Related Flags

qin=auto                Input quality offset: 33 (Sanger), 64, or auto.
qout=auto               ASCII offset for output quality.  May be 33 (Sanger), 64 (Illumina), or auto (same as input).
qfake=30                Quality value used for fasta to fastq reformatting.
maxcalledquality=41     Cap quality values at this upper level.
mincalledquality=0      Cap quality values at this lower level.
ignorebadquality        (ibq) Don't crash if quality values appear to be incorrect.
qtrim=f                 Enable or disable quality trimming.  May be set to r, l, or rl to trim the right, left, or both sides.
trimq=                  Trim bases below this quality value.

Length-Related Flags

fastareadlen=           Fasta sequences longer than this are broken into subsequences of at most this length, and given a suffix such as _part_1.  Only works with fasta files; generally designed for mapping very long sequences with BBMap.
fastaminlen=            Discard fasta sequences shorter than this.
maxlen=                 Has different meanings depending on the program.  For BBMap, reads longer than this will be broken to pieces this length.  For most other programs, it acts as a filter.
minlen=                 Has different meanings depending on the program.  Typically, reads shorter than this will be discarded.

Histogram Flags

bhist=<file>            Base composition histogram by position.
qhist=<file>            Quality histogram by position.
qchist=<file>           Count of bases with each quality value.
aqhist=<file>           Histogram of average read quality.
bqhist=<file>           Quality histogram designed for box plots.
lhist=<file>            Read length histogram.
gchist=<file>           Read GC content histogram.
gcbins=100              Number gchist bins.  Set to 'auto' to use read length.

*Advanced Flags*
Debugging and Benchmarking Flags


verbose=f               Print status messages for debugging.
parsecustom=f           Parse synthetic read names for custom data stored by RandomReads.

Buffering and I/O Flags

readbufferlength=200    Number of reads to store per ListNum.  A ListNum is the smallest unit of work sent to a worker thread.
readbuffers=            Number of ListNums to store in the queue waiting for worker threads.  The default is 150% of the number of threads.
bf1=                    Set to true to force ByteFile1 to be used for reading files.
bf2=                    Set to true to force ByteFile2 to be used for reading files (faster).

MPI and JNI Flags

usejni=f                Set to true to enable JNI-accelerated versions of BBMerge, BBMap, and Dedupe.  Requires the C code to be compiled.
mpi=0                   Inform the program of how many MPI processes will be used.  Most programs are not currently MPI-capable.
crismpi=f               Use an MPI version of ConcurrentReadInputStreams.
mpikeepall=t            If using MPI, send all reads to all processes.