Data Preparation

This page will take you through the different scripts and functions that covers initial parsing, demultiplexing and adapter removal from the basecalled Nanopore reads. IMPORTANT: Apart from the initial configuration file (runMetaPipe.sh), the user does not need to do any other changes in the package scripts. The descriptions below are to act as a breakdown of what is happening at major steps and what functions/tools/commands are being used!

Pre-Checks

flowchart LR
	b1[User options] 
	b2{Reads: \ndscDNA \nand/or \nDNA}
	b3{Transcriptome: \ncreate \nor provided}
	b8{Demultiplexer:\n Qcat or Guppy}
	b10[Choice]
	b5{Transcript \nreads and \nadapters?}
	b9[Ask user to \nprovide reads \nand adapters]
	b7([End])
	b11[Pass choice to Demultiplexing workflow]
b1 --> b2
b2 --dscDNA--> b3
b2 --DNA--> b8
b3 --create--> b5
b3 --provided--> b8
b5 --No--> b9
b5 --Yes--> b8
b9 --> b7
b8 --> b10
b10 --> b11
linkStyle 0 stroke: #008000
linkStyle 1 stroke: #008000
linkStyle 2 stroke: #008000
linkStyle 3 stroke: #008000
linkStyle 4 stroke: #008000
linkStyle 5 stroke: #ff0000
linkStyle 6 stroke: #008000
linkStyle 7 stroke: #ff0000
linkStyle 8 stroke: #008000
linkStyle 9 stroke: #008000

Loading

• The script checks the user inputs to determine which steps of the pipeline to use
• Check are done for:
  • If the user wants to create a transcriptome
    • If so, check if appropriate reads are provided
      • If not provided, stop the script and ask the user to provide reads
    • If not created, does the use provide a previously generated transcriptome
      • If so, continue the script
      • If not, stop the script and ask the use to provide the transcriptome
  • If the use provides appropriate references (DNA and/or cDNA)
    • If so, proceed
    • If not, stop the script and ask the user to provide the necessary reference genomes/transcriptomes
  • If the use chooses to make a transcriptome, does the user also provide adapters?
    • If so, proceed
    • If not, stop the script and ask the user to add the adapters
  • If the the entry file is the appropriate sequence type
  • If all checks are passed
• Additionally, checks are done to determine the demultiplexer choice given by the user
  • Where the user chooses qcat, the sequencing kit is checked. If the kit provided is NBD104, NBD103/ is also added
  • Where guppy is the chosen demultiplexer, SQK and EXP are added to the sequencing kit and expansion kit respectively
• Checks are also done to determine if the user wants to carry out filtering
  • If so, checks are done for the length and quality threshold provided
  • Checks are also done for the Bracken length thresholds which must be at least the length of the length filters provided by the user

Demultiplexing and Filtering

flowchart LR
subgraph Demultiplexing and Filtering
	d1[Demultiplex \nbasecalled reads] 
	d2[(Zipped \ndemultiplexed \nreads)]
	d3[Filter for \nlength and \nquality]
	d4[Rename files]
	d5[(Filtered \ndemultiplexed \nreads)]
	d6[AssemblyStats \nand FastQC \noutputs]
end
subgraph Functions in Preprocessing.py
	p1[demultip: do \ndemultiplexing]
	p2[dna_filter: calls filt_qc \nto filter DNA \nand format output]
	p3[cdna_filter: calls filt_qc \nto filter DNA \nand format output]
	p4[filt_qc: do filtering \nusing Nanofilt]
	p5[run_QC: QCs reads \nwith NanoStat, \nNanoQC and FastQC]
end
subgraph Functions in Tools.py
	t2[zipFiles: zips files]
end
d1 --> d2
d2 --> d3
d3 --> d4
d4 --> d5
d2 -...-> d6
d5 -...-> d6
p1 -.-> d1
p5 -.-> d2
p2 -.-> d3
p3 -.-> d3
p5 -.-> d5
p4 --> p2
p4 --> p3
t2 -.-> d2
linkStyle 0 stroke: #008000
linkStyle 1 stroke: #008000
linkStyle 2 stroke: #008000
linkStyle 3 stroke: #008000
linkStyle 4 stroke: #008000
linkStyle 5 stroke: #008000
linkStyle 6 stroke: #ff6600
linkStyle 7 stroke: magenta
linkStyle 8 stroke: #1e90ff
linkStyle 9 stroke: #996600
linkStyle 10 stroke: magenta
linkStyle 11 stroke: #9bddff
linkStyle 12 stroke: #9bddff

Loading

Demultiplexing

• Demultiplexing is carried out via a function that is housed in the Preprocessing.py script
• The function takes in the path to the newly created demultiplexed folder, the path to the basecalled reads, the choice of demultiplexer, the number of threads and the sequencing kit used
• Qcat is the default demultiplexer due to the guppy choice also requiring a GPU
• The outputs of demultiplexing are multiple files named barcodeXX.fastq with the XX representing the different barcode numbers
• IMPORTANT: The outputs folder will contain more than the barcodes which you used. This is due to the manner with which barcodes are identified by qcat and guppy. Essentially, if the algorithm is unable to confidently determine a barcode, it is placed in the none.fastq. In almost all cases of demultiplexing, output files for barcodes which were not used will be produced. This is due to a somewhat low percentage chance of the barcode you used being extended and recognised as another barcode. An example of this is the use of barcodes 03 and barcode 04. These two barcodes will be the largest files and contain the largest number of demultiplexed reads. However, there will also be a large 'none.fastq' and other barcode files e.g. barcode 01, barcode 02, barcode 05 etc., which would contain a very small number of files
• Example commands for demultiplexing are

# example demultiplexing command for qcat
cat path/to/basecalled/reads | qcat -b path/to/output --detect-middle -t threads --trim -k sequencing_kit
# example demultiplexing command for guppy
guppy_barcoder -i path/to/basecalled/reads -s path/to/output --barcode-kits sequencing_kit -r -q 0 -t threads --compress-fastq -x auto --detect_mid_strand_barcodes --trim_barcodes --trim_adapters

• For more information on what these options do for both qcat and guppy, please look in their documentation

Quality Control

• These are multiple steps that is called with the runQC function in the main NanoMetaPipe.py script. However, the runQC function itself is housed in the Preprocessing.py script.
• The runQC function carries out basic quality checks of the input data using NanoQC, NanoStat and FastQC.
• The outputs are saved in the generated Stats folder
• Example commands are:

# example command for NanoStat
NanoStat --fastq path/to/demultiplexed/barcodeXX.fastq.gz --outdir path/to/output/Stats/barcodeXX -n barcodeXX.txt
# example command for nanoQC
nanoQC -o path/to/output/Stats/barcodeXX path/to/demultiplexed/barcodeXX.fastq.gz
# example command for fastqc
fastqc -t threads -o path/to/output/Stats/barcodeXX path/to/demultiplexed/barcodeXX.fastq.gz

Filtering

• Where the user chooses to carry out filtering, depending on the sequence type, filtering is carried out using the dna_filter and/or cdna_filter functions. Both are also in the Preprocessing.py script
• Both functions take in the isolate name, the path to the demultiplexed reads, the barcodes used, the output directory, filter length threshold, filter quality, the path to the stats directory, the number of threads and the path to the scripts folder of the NanoMetaPipe package
  • Again note that this is done automatically by the script!
• If the user chooses not to carry out filtering the demultiplexed FASTQ files are moved to a script-generated folder: Filter_Demultiplexed_Reads
• Either way, the filtered reads are ready to proceed for decontamination