-
Notifications
You must be signed in to change notification settings - Fork 2
FAQs
No, Tree Nine expects MAPLE-formatted diff files instead. It is expected you will be running Tree Nine downstream of myco, because myco takes in FQs and outputs MAPLE-formatted diffs.
Yes, including Mycobacterium canettii. We specifically checked every unusual member of MTBC we could find on SRA to build our tree.
Please be aware that Mycobacterium canettii is an extreme phylogenetic outgroup compared to every member of MTBC. If you are using a tree that includes this species, to make visualization a little clearer, you may wish to reroot the tree to a specific sample representing a reconstructed ancestor or even M. canettii itself (Tree Nine includes support for this kind of rerooting).
As-is, no, because the decontamination step actively attempts to remove NTM-associated reads because NTM is a common contaminant of cultured MTB samples, and because H37Rv is considered the reference genome. However, with edits to the decontamination reference, you could in theory do this -- but you would still be using H37Rv as your reference for everything.
Not at the moment, because the decontamination and variant calling steps use clockwork, which strictly requires PE Illumina (as of 2024)
If all of your samples have precisely one R1 file and precisely one R2 file, you can use _1 or _R1 to mark R1 and _2 or _R2 to mark R2. For example, both of these would work:
- SAMPLE_1.fastq and SAMPLE_2.fastq
- SAMPLE_R1.fq.gz and SAMPLE_R2.fq.gz If any of your inputs are multi-lane samples that have not been concatenated, we recommend SAMPLE_LANE_1.fastq and SAMPLE_LANE_2.fastq. These are not strict requirements -- other iterations may work, but do a small-scale test first (or just rename your samples).
Yes. Multi-lane samples will be concatenated during the decontamination step.
For example, SAMN02584599 contains both SRR1166330 and SRR1169013. When pulling via TB-D_sra, you will end up with SRR1166330_1.fastq, SRR1166330_2.fastq, SRR1169013_1.fastq, and SRR1169013_2.fastq in the working directory. These get renamed to SAMN02584599_SRR1166330_1.fastq, SAMN02584599_SRR1166330_2.fastq, SAMN02584599_SRR1169013_1.fastq, and SAMN02584599_SRR1169013_2.fastq in order to keep the name of the sample in the filenames. These files are then passed to the decontamination task, and assuming they pass, your cleaned FASTQs will be called SAMN02584599_1.fastq and SAMN02584599_2.fastq and then carry on to the variant caller. The output of the variant caller will be SAMN02584599.vcf, which will appear on your tree as SAMN02584599.
TB-D will filter out samples that fail its QC standards. Additionally, when downloading from NCBI SRA, TB-D_sra will filter out:
-
Index/Barcode sequences that are sequenced seperately from R1/R2
- If a sample downloads as three FASTQ files, then that index/barcode file will be thrown out and the remaining R1/R2 pair will be kept
- If a sample downloads as one, five, seven, nine, etc FASTQ files, the whole sample will be thrown out
- Non-Illumina samples (PacBio, etc)
- SE Illumina samples
- Corrupt files, such as when the FASTQ quality score isn't the same length as the nucleotide string
Please be aware that TB-D cannot automatically detect:
- in silico synthetic data such as SAMN18146425
- samples within "sample pools" such as SAMEA968074
- samples that contain data from multiple biological samples that were incorrectly submitted as single-sample
...but you might be able to spot such oddities on a phylogenetic tree.
On default settings, TB-D will attempt to throw out bad samples silently, without causing the workflow to exit with an error. This is great for people who want to run as many samples as possible at once, because one bad sample will not cause everything else to halt. It's also useful for those who use Terra data tables, as intermediate outputs and a status code can be saved to the Terra data table, making it possible to keep some intermediate outputs and see at a glance which samples are problematic.
You can set variantcalling_crash_on_error to true (default: false) to make the pipeline crash if any error is encountered in the variant caller. "Any error" includes the variant caller running out of memory, timing out, getting an unknown error, or (most relevant to data QC) failing to actually call enough variants due to the sample being too small/corrupt/contaminated, so this isn't recommended.
Two methods:
A) Every run of Tree Nine should take in the previous run's output updated_diff_file as input existing_diffs, and the previous run's output updated_diff_contents as the input existing_samples, but have input_tree stay the same for all runs (recommended if tracking persistent cluster IDs)
B) Every run of Tree Nine should take in the previous run's output BIG_tree_usher as input input_tree, but leave existing_diffs and existing_samples undefined (untested but should work; if tracking persistent cluster IDs you must run upon the entire tree)
If I'm running Tree Nine sequentially, can I rename or replace samples after they have run through Tree Nine?
Please don't.
Leave the crappy sample as-is and upload your resequence with a name like NTT000012_resequence
Instructions assume you're running on Terra with data tables, and your sequential runs use updated_diff_file instead of inputting a new tree every time.
- Consider the alternatives -- if this is a QC failure or resequence, just upload the fixed sample with a new name and leave the bad sample as-is
- [only if workflow is automated with FISS] If it almost time (within 2 hours) for an automation run to begin, do not do anything yet; wait until it completes to avoid race conditions
- Get the ID of the sample to be removed (ex: NTT000012)
- Download the combined diff file (
updated_diff_file) from the most recent run of Tree Nine on the set data table - Delete NTT000012's information from the combined diff file and upload it somewhere in the workspace Google Bucket
- Each sample is marked with
>so delete everything from>NTT000012\nuntil the>of the very next sample
- Each sample is marked with
- In the set data table, modify the
updated_diff_filecell of the most recent Tree Nine run so it now points to your modified combined diff file instead of the old version that contained NTT000012 - Download the list of samples file (
updated_diff_contents) from the most recent run of Tree Nine on the set data table - Delete
NTT000012from the list of samples file and upload it somewhere in the workspace Google Bucket - In the set data table, modify the
updated_diff_contentscell of the most recent Tree Nine run so it now points to your modified list of samples file instead of the old version that contained NTT000012 - Check to see if NTT000012 is in
unclustered_sampleson the set table for the most recent run of Tree Nine on the set data table, if so, you can stop here
If NTT000012 did cluster, its removal may cause decimation of multiple clusters. Tree Nine assumes decimation as a result of dropped inputs is user error and will throw an exception if it detects that. Because you are intentionally removing a sample and understand this may result in decimation of clusters, you will need to disable this exception on the next run of Tree Nine.
- [only if workflow is automated with FISS] Pause automation
- Wait until you have a new batch of samples (do not re-run an old batch, that will create duplicate sample IDs and break UShER)
- Manually run the new batch of samples through myco
- Once myco completes for all samples in the new batch, manually run Tree Nine for the new batch, setting
no_dropped_sample_failsafetotrue - Once Tree Nine completes, set
no_dropped_sample_failsafeback tofalseand save (probably unnecessary due to how automation works, but do it anyway) - [only if workflow is automated with FISS] Re-enable automation
You do not need to adjust the base tree. The base tree is unchanged for every run. We track samples sequentially using the combined diff file (updated_diff_file), a list of sample IDs extract from the combined diff file (updated_diff_contents), the cluster JSON (updated_cluster_information_json), and the persistent IDs file (updated_persistent_ids). Do not remove NTT000012 from the cluster JSON nor the persistent IDs file; removing it will break how Tree Nine tracks changes in clusters over time. NTT000012 will be removed from the persistent IDs file of the run you did in step 15 anyway. NTT000012 will temporarily be referenced in the cluster JSON under the sample_ids_previously field, but that field is reset every run, so the subsequent run will remove all traces of NTT000012.
If for some reason you must scrub all traces of NTT000012 from all files immediately, such as if the name itself contains PHI, you will unfortunately need to modify the (1) persistent cluster IDs file and (2) the cluster JSON, in addition to the previously discussed edits to (3) the combined diff file and (4) the combined diff contents file. The "easiest" way to do this is probably removing NTT000012 from (3) and (4) as discussed above and removing all lines containing NTT000012 from (1) -- be aware it may be on up to three lines -- but you must also check NTT000012's removal wouldn't create any one-sample clusters. If it would create a one-sample cluster, create a duplicate of the non-NTT000012 sample and fudge it into (1), (2), (3), and (4) as if it had been in that cluster all along. Because you fudged the data, Tree Nine will not detect the loss of NTT000012, so upon the next of run of Tree Nine you must also check that the cluster-specific Microreact project(s) has been updated to exclude NTT000012 (currently we update every MR project for every cluster, so it should be updated anyway, but this behavior might change later).
My tasks are getting cancelled due to lack of resources (sigkill, return code 137, out-of-memory, etc)
If you're seeing this on Cromwell on an HPC or local machine, this is probably a task concurrency issue that can be fixed with a config change (see below). If you're seeing this on miniwdl, your system resources may not be high enough to run the pipeline as intended. Usually the "limiting reagent" seems to be RAM -- we recommend a minimum of 32 GB, although 16 GB will generally work, especially if you enable downsampling.
If you're pretty sure your hardware should be able to handle this, these documents may help you:
- [Cromwell/Dockstore CLI only] Troubleshooting the most common resource issues -- this document is in the context of the Dockstore CLI, which runs Cromwell under the hood, but it also applies to Cromwell itself
- Resource constraints on Docker Engine
- Resource constraints on Docker Desktop
- Many users only want to call variants. Having myco be its own thing allows them to skip UShER.
- When we can launch workflows on either a sample-index data table, or a set data table which selects rows of the sample-indexed data table. We want myco to write QC outputs to the sample-indexed data table, so it must run in that mode. However, we cannot run Tree Nine in that mode, or else every sample will get its own tiny tree, which isn't very helpful!
It is a workaround for several WDL limitations involving optional files. Essentially, it allows QC-failing samples drop out without throwing errors.
Nothing in here should be construed as legal advice.
To the best of my knowledge, all components of myco MIT licensed except TBProfiler (GPL 3.0) and coreutils (LGPL). My understanding is that components of TB-D that directly rely upon TBProfiler therefore must be GPL 3.0 licensed, including:
- myco_raw
- myco_sra
- TBD_raw
- TBD_sra
- TBProfiler-specific tasks and workflows
The entire Tree Nine subworkflow does not contain TBProfiler and is MIT licensed. Ranchero, which is not included in TB-D but linked in this repo, is GPL 3.0 licensed.
I want to replicate your published results as closely as possible. Which version of TB-D should I use?
Several new features, user-friendly options, and critical bugfixes have been added to the pipeline since it was used to generate data, so older versions of the pipeline aren't supported. However, you the overall logic of the pipeline has remained almost identical. You can set just_like_2024 to True to change the few "important" differences back to how they were before:
- Older version (0.11.3) of the decontamination reference
- Older version (0.11.3) of Clockwork
- TBProfiler is run on BAM files instead of FQs
This also applies to running myco_raw or myco_sra. Tree Nine has only had critical bugfixes and new features between pipeline versions, so just use the latest. Be aware that although everything else is deterministic (including any of the optional downsampling, as we use a set seed for that), the specific UShER command used to add samples to the tree is non-deterministic. This means that samples may be placed in different places on your tree.