Processing

Processing paired reads

process reads two mate files side by side, checks every pair as check --paired does, and writes the pairs it keeps to two new files. On the way it can move a UMI from the start of the reads into their names, trim adapters, overlaps, poly-G runs, and low-quality ends, and drop pairs that fail quality, N, or length filters. Each trim and each filter is off until you ask for it, so a plain process run validates and copies, and nothing else.

flowchart LR
    R[Read a pair<br/>validate both mates] --> U[Move the UMI<br/>into both names]
    U --> T[Trim<br/>quality tail, poly-G, adapters]
    T --> F[Filter<br/>quality, N, length, dimers]
    F -->|both mates pass| W[Write the pair]
    F -->|a mate fails| D[Drop the pair<br/>count the reason]

Loading

_{Figure 1. What happens to each pair. Every step is off until its options are given.}

The trimming and filtering rules are fastp 1.1.0's, followed step by step, so process keeps exactly the records fastp keeps with the same settings. I chose to match fastp rather than invent new rules: its behavior is well known, and a workflow that already trusts it can compare the two outputs record by record. z-fastq is not a fastp replacement, though; it covers only these trims and filters (References).

A first run

z-fastq process --gzip \
  --adapter1 AGATCGGAAGAGCACACGTCTGAACTCCAGTCA --adapter2 AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT \
  --min-length 15 --report sample.json \
  --out1 trimmed_R1.fastq.gz --out2 trimmed_R2.fastq.gz R1.fastq.gz R2.fastq.gz

This trims the Illumina TruSeq adapters, drops pairs with a mate shorter than 15 bases, writes gzip outputs, and records what happened in sample.json. Quality control explains the measurements in that report, and Report reference lists its fields.

Outputs

• --out1 and --out2 must be new files; an existing path, an input included, fails before anything is written.
• Mate 1 goes to --out1 and mate 2 to --out2, in input order. Headers and plus lines are copied as they are, apart from the UMI that --umi-length adds to the names; trims shorten the sequence and quality together.
• --gzip compresses both outputs with Zipir's fast preset. --gzip-preset even or dense gives smaller files for more CPU time: on a 1.5 million pair NovaSeq sample the whole run took 5.8 s with fast and 14.0 s with even, for 216 MB and 180 MB. The output carries no time stamp, so the same inputs give the same bytes.
• One input may be standard input; outputs are always files.
• --max-pairs N stops after the first N pairs, and the report records that the inputs were not read to the end. Use check --paired to validate whole files. It cannot be combined with the MultiQC-compatible outputs, which cannot say that only part of a file was read.

UMIs

Some libraries start each read with a unique molecular identifier (UMI): a few random bases that tell apart two molecules that start at the same place. Left in the read, an aligner treats them as part of the insert, and duplicate marking cannot use them. --umi-length N --umi-mates M cuts the first N bases from mate 1, mate 2, or both (1, 2, or both) and adds them to the names of both mates; --umi-skip S also removes the S spacer bases that follow the UMI. N is 1 to 100 and S is 0 to 100, as in fastp.

z-fastq process --umi-length 8 --umi-skip 4 --umi-mates both \
  --out1 R1.umi.fastq --out2 R2.umi.fastq R1.fastq.gz R2.fastq.gz

A pair whose mates begin with AACCGGTT and TTGGCCAA, each followed by a 4-base spacer, is written with these names:

@A00123:8:HFLKW:1:1101:1000:2000:AACCGGTT_TTGGCCAA 1:N:0:ACGT
@A00123:8:HFLKW:1:1101:1000:2000:AACCGGTT_TTGGCCAA 2:N:0:ACGT

• The name takes fastp's form: a : and the UMI at the end of the read's identifier, with the two mates' UMIs joined by _, mate 1 first. Both mates carry the same UMI, so check --paired still pairs them. UMI-tools reads the UMI with --umi-separator=:. samtools markdup --barcode-name reads the eighth : field, which is the UMI on Illumina names of seven fields, as above; for other names, such as SRA's SRR7646915.1, give it --barcode-rgx instead.
• The identifier ends at the first space or tab, before a trailing /1 or /2: @r7/1 x becomes @r7:AACCGGTT_TTGGCCAA/1 x. A plus line that repeats the header gets the new name as well.
• The UMI is removed first, so trimming, filters, and insert sizes see only the bases after the spacer. The raw statistics and the pair duplication rate describe the reads as read, UMI included; the retained statistics and QC tables describe the reads as written. fastp counts them the same way.
• A pair in which a UMI mate has no base left after the UMI and spacer is dropped as too short. The report counts those pairs under umi, with the pairs whose UMI holds a base other than A, C, G, or T in either case, such as N.

A UMI at the 3' end of a read or in the index read is not supported yet.

Trimming

The trims run in this order on every pair, after the UMI, each only when its options are given.

Quality tail

--trim-tail-window W --trim-tail-quality Q looks at windows of W bases, starting at the 3' end and moving toward the 5' end, and stops at the first window whose mean quality is at least Q (Figure 2). If the final window already passes, the full read is kept. Otherwise, the read is cut after the first base of the passing window. Trailing N bases are then removed. Reads no longer than the window are trimmed to zero bases.

_{Figure 2. Quality-tail trimming with a 4-base window and Q = 20. Windows are tried from the 3' end; the first whose mean reaches Q is marked, and the read keeps everything up to that window's first base.}

W is 1 to 1000 and Q is 1 to 30, given together. A read no longer than W is emptied, and a read with no passing window keeps one base, even when its first W bases are high quality, because the window at the 5' end is never tested. When either mate ends up empty, the pair is not trimmed further and is dropped.

Poly-G

--trim-poly-g N removes the run of G that two-colour instruments write when the signal ends; N is at least 1. Bases are scanned from the 3' end; the scan stops after more than 5 other bases, or, from the Nth base on, when other bases exceed one in eight. If at least N bases come before the base that stopped the scan, or the scan reaches the 5' end of a read of N bases or more, the read is cut before the leftmost G seen. A shorter run can still go when the scan carries past it: with N = 10, a read ending in nine G loses them.

Adapters

--adapter1 A1 --adapter2 A2 removes adapters in two ways (Figure 3).

_{Figure 3. The two ways adapters are found. Top: the insert is shorter than the reads, so each read runs into the adapter, and the overlap between the mates shows where the insert ends. Bottom: without an overlap, each mate is searched for its own adapter.}

1. By overlap. Mate 1 is compared with the reverse complement of mate 2 at each offset in turn. The first offset where the overlap has at most 5 mismatches in its first 50 positions, and no more than one per 5 bases, wins; more than 30 bases must overlap. When mate 2 then starts before mate 1, the insert is shorter than the reads, and both mates are cut to the overlap.
2. By sequence. Otherwise each mate is searched for its own adapter: a full copy, or at least 5 bases of its start at the read's 3' end, with one mismatch per 8 bases. For an adapter of 16 bases or more the search begins 4 bases before the read (3 for 12 to 15 bases, 2 for 8 to 11), so a read that starts with the tail of an adapter is emptied. If nothing is found, a copy with one inserted or deleted base is tried. Like fastp, this last test compares the adapter with the start of the read, so it can also cut near the 3' end where no adapter is present.

Adapters are 4 to 128 uppercase bases of A, C, G, and T. The adapter search compares exact bytes, so a lowercase base never matches an adapter. The overlap search complements mate 2 into uppercase, as fastp does, so there only mate 1's lowercase bases fail to match.

Filtering

Each read is checked after trimming, and a pair is written only when both mates pass. A read fails on the first check below that applies.

Check	A read fails when	Range
always, with any trim or filter	no bases are left
--low-quality Q --max-low-quality-percent P	more than P% of bases are below Q	Q 0 to 93, P 0 to 100
--max-n N	more than N uppercase N	0 to 50
--min-length N	fewer than N bases	0 to 4,294,967,295

• With adapters, a pair whose adapter or overlap step cut a mate and left both mates at 2 bases or fewer is an adapter dimer, and is dropped.
• A pair is dropped whole. Its reason is the worse of its mates' reasons, in the order adapter dimer, low quality, too short, too many N.
• Dropping pairs never changes the exit status. A malformed record still fails the run, because every pair is validated before it is trimmed.

fastp turns its filters on by default; z-fastq applies only the filters you give. --low-quality 15 --max-low-quality-percent 40 --max-n 5 --min-length 15 are fastp's default values.

Matching fastp settings

Each pair of lines keeps the same records with fastp 1.1.0:

z-fastq  --adapter1 A1 --adapter2 A2
fastp    --adapter_sequence A1 --adapter_sequence_r2 A2 --disable_trim_poly_g -Q -L

z-fastq  --adapter1 A --adapter2 A --trim-poly-g 10 --trim-tail-window 4 --trim-tail-quality 20
fastp    --adapter_sequence A --adapter_sequence_r2 A --trim_poly_g --poly_g_min_len 10
         --cut_tail --cut_tail_window_size 4 --cut_tail_mean_quality 20 -Q -L

z-fastq  --adapter1 A1 --adapter2 A2 --low-quality 15 --max-low-quality-percent 40 --max-n 5 --min-length 15
fastp    --adapter_sequence A1 --adapter_sequence_r2 A2 --disable_trim_poly_g

fastp 1.4.0 keeps the same records, except on mate 2 reads with the IUPAC letters S, W, or D, where its result depends on the CPU. fastp's automatic adapter detection, automatic poly-G mode, and its other filters are not part of z-fastq.

UMI options add to any line above, and fastp writes the same records with its own:

z-fastq  --umi-length 8 --umi-skip 4 --umi-mates both
fastp    --umi --umi_loc per_read --umi_len 8 --umi_skip 4

--umi-mates 1 or 2 matches --umi_loc read1 or read2. The names differ in three cases: fastp adds the UMI after a trailing /1 or /2, so its two mates no longer share a name, and aligners that cut names at /, such as STAR by default, lose the UMI; fastp looks only for a space, so in a name with a tab before any space its UMI lands after the comment; and fastp leaves a header repeated on the plus line unchanged. fastp also keeps one base of a read shorter than the UMI and spacer, where z-fastq drops the pair.

When a run fails

A failure stops the run and exits with a nonzero status: 1 for invalid input, 2 for a usage error, 3 for I/O or memory, 4 for a limit. Pairs written before the failure stay in the outputs: after an input failure both files end with the same complete pairs, but after a failed write to an output, such as a full disk, the files can end inside a record and hold different numbers of pairs. The report is still written, with "status": "error" and the reason, unless the report itself cannot be written, in which case it is removed; the MultiQC-compatible files are written only when the run succeeds.

Treat a run as complete only when it exits with 0.

Speed

_{Figure 4. What each step costs on one 1 GiB pair, one CPU each: copying the pair (reading, checking, and writing it), then the two trims. The process benchmark has every input, threads, and QC tables.}

Performance compares every command, and what other tools gain from threads.

Related pages

• Performance: speed and memory against fastp, with threads.
• Quality control: what the run measures, and the files MultiQC reads.
• Report reference: the trimming and filtering counts in the report.
• Paired reads: the name checks every pair passes first.
• Recipes: a complete run with fastp's default filters.