Skip to content

v1.0.0 Release

Choose a tag to compare

View all tags
@nrminor nrminor released this 11 Sep 21:06
· 667 commits to main since this release
v1.0.0
b2d054a

NVD1 - Retroactive release for Version 1.0 of NVD

This is a retroactive release for the commits up to and including the commit tagged with v1.0.0. Note that this version has been superseded by versions >=2.0, which can be found in more recent releases.

About NVD

NVD is an opinionated workflow for identifying and exploring metagenomes for
families of viruses that infect humans. It is opinionated in that it focuses
narrowly on these virus families to the exclusion of other potentially
interesting viral and non-viral microbial taxa. NVD works with arbitrarily large
Oxford Nanopore and Illumina datasets without downsampling. The results are
available in a LabKey Server data explorer available to O'Connor lab users and
collaborators. For those without LabKey data explorer access, results are stored
locally.

Why Another Metagenomic Workflow?

Good question. When I started working on NVD in June 2024, I hoped to find a
widely used, easy-to-use workflow optimized for viral metagenomics.
CZID is excellent, but downsamples to a subset of reads.
Sometimes this isn't a problem; sometimes it causes substantial reductions in
sensitivity. The outstanding
Sourmash tools perform informative,
efficient searches against reference databases but do not generate contigs that
can be used to characterize interesting viruses.

What Does NVD Do?

NVD begins with one or more Illumina paired-end or Oxford Nanopore datasets. It
can also evaluate datasets downloaded from NCBI SRA. In the case of Illumina
datasets, paired-end sequences are interleaved and properly paired using the
bbmap repair.sh tool before processing.

These reads are the classified with the NCBI STAT tool, specifically the
aligns_to command. A tax list containing the entire subtree of the virus
families that infect humans is used for classification. This means that NVD is
designed to find members of these virus families:

  • Adenoviridae
  • Anelloviridae
  • Arenaviridae
  • Arteriviridae
  • Astroviridae
  • Bornaviridae
  • Peribunyaviridae
  • Caliciviridae
  • Coronaviridae
  • Filoviridae
  • Flaviviridae
  • Hepadnaviridae
  • Hepeviridae
  • Orthoherpesviridae
  • Orthomyxoviridae
  • Papillomaviridae
  • Paramyxoviridae
  • Parvoviridae
  • Picobirnaviridae
  • Picornaviridae
  • Pneumoviridae
  • Polyomaviridae
  • Poxviridae
  • Sedoreoviridae
  • Retroviridae
  • Rhabdoviridae
  • Togaviridae
  • Kolmioviridae

Reads that have kmers that match one or more viruses within these families are
saved by aligns_to. These are "putative viral reads." NVD uses seqkit grep to
extract these sequences from the original dataset.

The putative viral reads are then assembled into contigs with SPAdes.
Empirically, the --sewage assembly preset provides the best results with both
ONT and Illumina data, so this mode is used by NVD.

Contigs produced by SPAdes are then quality checked. Contigs with unusually high
entropy, often consisting entirely of repetitive sequence, are filtered using
bbmask.sh. Contigs shorter than 200bp after masking are removed, since short
contigs are minimally informative. The filtered contigs are then classified with
the NCBI STAT tool against a "coarse" reference library that is broadly
representative of all the sequences in NCBI. A subtree of possible exact
classifications is derived from the coarse classifications and used for more
accurate classification. From these accurate classifications, contigs that
appear to be viral are extracted.

These viral contigs are then used as BLAST queries against the NCBI core-nt
database. Megablast classification typically identifies matches for most
contigs. Those contigs that do not have megablast matches are then used as
queries for blastn.

Additionally, the putative viral reads are mapped against the viral contigs.
This allows further exploration and confirmation of any unexpected or unusual
hits.

The blast classifications, read mapping information, and associated contig
metadata are then saved to an output folder in CSV format (and loaded into a
database in the O'Connor lab LabKey Server, if appropriate access credentials
are provided). .zst archives containing useful data files (e.g., read mapping
.bam files) are also saved in the output folder.

Getting Started

NVD is written in Snakemake and has its dependencies bundled in an Apptainer
container. NVD should work on machines with at least 32GB of available RAM. The
resources.zst file is approximately 230GB. After decompression, it is
approximately 300GB. The amount of storage needed depends on the size of the
FASTQ files; 530GB plus 3x the size of the input FASTQ files is a good guide.
For example, if the input FASTQ files are 100GB, ensure that at least 830GB of
storage are available. Most testing has been done on Ubuntu Linux, however, a
Dockerfile for replicating the Apptainer image is available for running on Macs
running Docker Desktop.

Prerequisites

  • Miniconda
  • Globus CLI
  • Apptainer (for
    Linux)
  • Docker Desktop (for MacOS)
  • Snakemake
  • The resources.zst archive containing NCBI BLAST core-nt, NCBI STAT
    databases, a taxonomic rank database, and a taxonomic list of the subtree of
    human-infecting virus families.

The resources.20240830.zst version contains core-nt downloaded from NCBI on
2024-08-03, the STAT tree index downloaded 2024-08-30, the STAT tree filter.dbss
and associated annotations downloaded 2024-08-30, gettax.sqlite downloaded on
2024-08-30, and a human_viruses_taxlist.txt created on 2024-09-05.

  • The workflow folder containing the Snakemake workflow and associated python
    scripts
  • A config folder containing a config.yaml file specifying runtime variables
    and the samples to be analyzed. You also need to get the LabKey API Key,
    LabKey username, and LabKey password information from DHO for LabKey
    integration.

Linux Installation

  1. Clone the repo to the working directory and navigate into the working
    directory 
    git clone https://github.com/dhoconno/nvd.git && cd nvd
  2. Create a conda environment with Apptainer and Snakemake 
    conda create -n nvd
conda activate nvd
conda install -y snakemake apptainer -c conda-forge
  3. Download) the
    Apptainer image file. If you have Globus access, this link is much, much
    faster than the wget access command shown below:. 
    wget https://dholk.primate.wisc.edu/_webdav/dho/projects/lungfish/InfinitePath/public/%40files/nvd.30572.sif
  4. Download
    the resources.zst file containing databases and taxonomy files (note, this
    is ~230GB and will take a while to download. I suggest going out for an iced
    tea while you wait). If you have Globus access, this link is much, much
    faster than the wget access command shown below: 
    wget https://dholk.primate.wisc.edu/_webdav/dho/projects/lungfish/InfinitePath/public/%40files/resources.20240830.zst
  5. Decompress the resources.zst file in the working directory and remove
    source after decompression 
    tar -I zstd -xvf resources.20240830.zst && rm resources.20240830.zst
  6. Copy gzipped-FASTQ files to process into data folder within the working
    directory.
  7. Modify the config.yaml file to specify the experiment number (an integer)
    and an output directory.
  8. Modify the config.yaml file to specify the samples to process and the
    path(s) to their FASTQ files. Here are example entries for the three
    supported file types: An important note on sample names -- replace all
    dashes (-) with underscores (_)     
    - name: Illumina_test
 r1_fastq: data/illumina.R1.fastq.gz
 r2_fastq: data/illumina.R2.fastq.gz
- name: AE0000100A9532
 sra: SRR24010792
- name: AE0000100A8B3C
 ont: data/AE0000100A8B3C.fastq.gz
    Many samples can be processed in the same workflow invokation. Multiple
    sample types can be processed at once. There is also a version of this
    workflow that runs one sample at a time on CHTC
    execute nodes. Ask DHO for details. (note: if you have access to upload
    results to the LabKey data explorer, you will also need to add LabKey
    credentials to the config.yaml file. Ask DHO for details.)
  9. Start an Apptainer shell in the working directory with the repo files
apptainer shell nvd.30572.sif

After installation, there should be data, config, resources, and
workflow folders in the working directory.

MacOS installation

  1. Download and install Docker Desktop. Provide at least 32GB of memory to
    Docker containers.
  2. Clone the repo to the working directory and navigate into the working
    directory 
    git clone https://github.com/dhoconno/nvd.git && cd nvd
  3. Create a conda environment with Snakemake 
    conda create -n nvd
conda activate nvd
conda install -y snakemake -c conda-forge
  4. Download the Docker
    image file. If you have Globus access, this link is much, much faster than
    the wget access command shown below: 
    wget https://dholk.primate.wisc.edu/_webdav/dho/projects/lungfish/InfinitePath/public/%40files/nvd.30572.tar
  5. Load the Docker image file 
    docker load -i docker/nvd.tar
  6. Download
    the resources.zst file containing databases and taxonomy files (note, this
    is ~230GB and will take a while to download. I suggest going out for an iced
    tea while you wait). If you have Globus access, this link is much, much
    faster than the wget access command shown below: 
    wget https://dholk.primate.wisc.edu/_webdav/dho/projects/lungfish/InfinitePath/public/%40files/resources.20240830.zst
  7. Decompress the resources.zst file in the working directory and remove
    source after decompression 
    tar -I zstd -xvf resources.20240830.zst && rm resources.20240830.zst
  8. Copy gzipped-FASTQ files to process into data folder within the working
    directory.
  9. Modify the config.yaml file to specify the samples to process and the
    path(s) to their FASTQ files. Here are example entries for the three
    supported file types: An important note on sample names -- replace all
    dashes (-) with underscores (_)     
    - name: Illumina_test
 r1_fastq: data/illumina.R1.fastq.gz
 r2_fastq: data/illumina.R2.fastq.gz
- name: AE0000100A9532
 sra: SRR24010792
- name: AE0000100A8B3C
 ont: data/AE0000100A8B3C.fastq.gz
    Many samples can be processed in the same workflow invokation. Multiple
    sample types can be processed at once. There is also a version of this
    workflow that runs one sample at a time on CHTC
    execute nodes. Ask DHO for details. (note: if you have access to upload
    results to the LabKey data explorer, you will also need to add LabKey
    credentials to the config.yaml file. Ask DHO for details.)
  10. Start a Docker shell in the working directory with the repo files
docker run -it --user $(id -u):$(id -g) -v $(pwd)/:/scratch -w /scratch nvd:30575

After installation, there should be data, config, resources, and
workflow folders in the working directory.

Usage

NVD is implemented as a snakemake workflow. It is designed to be run from within
an Apptainer/Docker container as described above. There is one mandatory config
parameter, run_id that must be specified upon invokation:

snakemake --cores XX --config run_id=`date +%s`

Note that run_id does not have to be a timestamp as shown in the example
above, but it should be something that unambiguously differentiates different
runs of the workflow.

The workflow should run to completion, with output saved to the location
specified in config.yaml['global']['out_dir']. The output includes a CSV file
describing the top five BLAST hits for each of the viral contigs. Be suspicious
of hits with relatively high BLAST
evalues or those where
only one or a few of the top hits are specific to a virus. There is also a
.zst archive containing putative viral reads mapped against the de novo
assembled contigs. The archive can be decompressed within the Apptainer/Docker
container using:

unzstd [archive_name]

By default, all intermediate files are saved in case there is a problem with the
workflow. After the workflow is run successfully, these files can be
automatically removed with:

snakemake clean --cores XX --config run_id=`date +%s`

Use this carefully, since regenerating the results directory would require
re-running the entire workflow.

(back to top)

Roadmap

  • 2024-09-10 - first public release version
  • I expect to update the databases approximately annually to incorporate newly
    discovered virus sequences

See the open issues for a full list of
proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to
learn, inspire, and create. Any contributions you make are greatly
appreciated.

If you have a suggestion that would make this better, please fork the repo and
create a pull request. You can also simply open an issue with the tag
"enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

Top contributors:

contrib.rocks image

License

See LICENSE.txt for more information.

Contact

Dave O'Connor - @dho - dhoconno@wisc.edu

Project Link: https://github.com/dhoconno/nvd

Acknowledgments

  • Marc Johnson and
    Shelby O'Connor, my partners in innovative
    pathogen monitoring from environmental samples.
  • Kenneth Katz, NCBI, for developing NCBI STAT, maintaining pre-built databases
    for STAT, and helpful discusssions
  • C. Titus Brown, for
    helpful discussions of using kmer classifiers as part of metagenomic workflows
  • Development funded by Inkfish
Assets 2
Loading