Skip to content


Folders and files

Last commit message
Last commit date

Latest commit



18 Commits

Repository files navigation


Single-ended FASTQ file deduplication


fqdedup depends on a number of python packages.


usage: [-h] [-n nt] [--max-mem MB] [--use-mean-qual] [--version]

author: Gabriele Girelli
version: 1.1.0
date: 180308
project: pre-processing sequencing data

  Dr. F. Agostini for the nice chats and for providing an initial prototype.

  Deduplicate FASTQ file: remove duplicate sequence by keeping the higher
  quality one. Moreover, remove reads with "N" in the initial portion of a
  read, if requested by the user.

  Initially, the records in the FASTQ are quickly counted with bash "wc -l".
  Then, the full FASTQ file is read and parsed with Bio.SeqIO. Each record is
  stored in plain text format alongside its quality in a dictionary, with its
  sequence as key. If "N" (i.e., any nucleotide) is found in the initial
  portion (user-defined) of a sequence, the sequence is discarded. Each
  sequence is compared to the encountered ones and replaces it only and only
  if its quality is higher (either sum or mean). It is also possible to
  manually set an upper limit of resident memory using the --max-mem option.

  The current implementation requires less RAM than previous ones, and shorter
  times to compute. Instead of storing each FASTQ record as parsed, it stores
  them as plain text alongside sequence and its quality (minor redundancy).
  For a 20 GB plain FASTQ, approx. 15 GB of resident memory are required.

positional arguments:
  fastq            Path to input FASTQ file. Both gzipped and plain FASTQ
                   formats are supported

optional arguments:
  -h, --help       show this help message and exit
  -n nt            Length [nt] of sequence initial portion to search for N.
                   Default: 0.
  --max-mem MB     Upper limit (in MB) of resident memory for the
                   deduplication process. Use -1 for unlimited. Not compatible
                   with MacOS. Default: -1.
  --use-mean-qual  Select sequences based on mean quality instead of quality
  --version        show program's version number and exit