Nextflow implementation template

[evanroyrees]

🎨 Add Autometa nextflow implementation template.
🎨 Update majority_vote.py parameters to more easily construct taxon assignment workflow.

NOTE: I use the more recent DSL2 language so we can modularize different autometa workflows. You will notice this use with the TAXON_ASSIGNMENT workflow nested within autometa.nf

[codecov]

Codecov Report

Merging #118 (0927596) into dev (5cccf75) will increase coverage by 2.10%.
The diff coverage is 51.00%.

@@            Coverage Diff             @@
##              dev     #118      +/-   ##
==========================================
+ Coverage   46.03%   48.14%   +2.10%     
==========================================
  Files          26       22       -4     
  Lines        3458     3022     -436     
==========================================
- Hits         1592     1455     -137     
+ Misses       1866     1567     -299     

Impacted Files	Coverage Δ
autometa/common/coverage.py	92.45% <ø> (ø)
autometa/common/exceptions.py	69.56% <ø> (ø)
autometa/common/external/bowtie.py	61.90% <ø> (ø)
autometa/common/external/diamond.py	17.91% <ø> (ø)
autometa/common/external/hmmer.py	16.66% <ø> (ø)
autometa/common/external/samtools.py	71.42% <ø> (ø)
autometa/config/databases.py	0.00% <0.00%> (ø)
autometa/binning/summary.py	38.93% <17.64%> (-58.33%)	⬇️
autometa/taxonomy/lca.py	10.38% <21.05%> (+0.09%)	⬆️
autometa/common/external/bedtools.py	58.18% <25.00%> (ø)
... and 16 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5cccf75...0927596. Read the comment docs.

[evanroyrees]

Example usage

cd $HOME/Autometa
# First update the inputs and parameters in `nextflow/parameters.config`
# e.g.
code nextflow/parameters.config
# or
nano nextflow/parameters.config
# or
vim nextflow/parameters.config

Now run autometa using nextflow

nextflow run main.nf \
    # Specify a working directory
    -w $HOME/work
    # run using SLURM
    -profile cluster \
    -c nextflow/parameters.config

Now if the pipeline was interrupted for some reason

nextflow run main.nf \
    -w $HOME/work \
    # run using SLURM
    -profile cluster \
    -c nextflow/parameters.config
    # Provide the `resume` flag so the pipeline will continue where it left off
    -resume

Workflow Visualized

# Draw the DAG from the written DOT file
dot -Tgif < $HOME/Autometa/pipeline_info/autometa-dot > autometa-dag.gif

[evanroyrees]

Sharing the Autometa pipeline using Nextflow

See Nextflow's documentation on pipeline sharing for more information.

Possible issue that may arise

no nextflow.config or main.nf files in the master branch

This nextflow-io/nextflow#1032 (comment) suggests that in order to share the Autometa workflow via nextflow we must have the main.nf and nextflow.config files in the master branch. So currently I believe

nextflow run WiscEvan/Autometa -r nextflow -c paramters.config

...will fail with the error:

## ERROR OUTPUT ##
WARN: Cannot read project manifest -- Cause: Remote resource not found: > > > https://api.github.com/repos/WiscEvan/Autometa/contents/nextflow.config
Remote resource not found: https://api.github.com/repos/WiscEvan/Autometa/contents/main.nf

Note: The above command would be updated once merged into dev

nextflow run KwanLab/Autometa -r dev -c paramters.config

[jason-c-kwan]

I resolved some comments that will be addressed in future pull requests - i.e. making some steps optional and parallelizing some steps. There were a few things though that weren't addressed from last time (see comments).

[jason-c-kwan]

Also copyright still says 2020

[jason-c-kwan]

Still needs to be addressed.

[jason-c-kwan]

Can you clarify this?

[jason-c-kwan]

See also comment below.

[jason-c-kwan]

When running the pipeline on the 78 Mbp dataset, I get the following error:

N E X T F L O W  ~  version 20.10.0
Launching `main.nf` [small_darwin] - revision: 3f8baf0f47
WARN: Access to undefined parameter `kmer_norm_method` -- Initialise it to a default value eg. `params.kmer_norm_method = some_value`
WARN: Access to undefined parameter `kmer_pca_dimensions` -- Initialise it to a default value eg. `params.kmer_pca_dimensions = some_value`
WARN: Access to undefined parameter `kmer_embed_method` -- Initialise it to a default value eg. `params.kmer_embed_method = some_value`
WARN: Access to undefined parameter `kmer_embed_dimensions` -- Initialise it to a default value eg. `params.kmer_embed_dimensions = some_value`
Autometa - Automated Extraction of Genomes from Shotgun Metagenomes
 =====================================================
 projectDir                         : /Users/jasonkwan/git_repos/autometa_PRs/PR_118/Autometa
 -----------------------------------------------------
 Data
 -----------------------------------------------------
 metagenome                         : /Users/jasonkwan/git_repos/autometa_PRs/PR_118/Autometa/test_data/metagenome.fna.gz
 interim                            : /Users/jasonkwan/git_repos/autometa_PRs/PR_118/Autometa/test_data/interim/
 processed                          : /Users/jasonkwan/git_repos/autometa_PRs/PR_118/Autometa/test_data/processed/
 -----------------------------------------------------
 Parameters
 -----------------------------------------------------
 cpus                               : 8
 length_cutoff                      : 3000
 kmer_size                          : 5
 kmer_norm_method                   : null
 kmer_pca_dimensions                : null
 kmer_embed_method                  : null
 kmer_embed_dimensions              : null
 clustering_method                  : dbscan
 classification_kmer_pca_dimensions : 50
 classification_method              : decision_tree
 completeness                       : 20.0
 purity                             : 95.0
 gc_stddev_limit                    : 5.0
 cov_stddev_limit                   : 25.0
 kingdom                            : bacteria
 -----------------------------------------------------
 Databases
 -----------------------------------------------------
 ncbi_database                      : /Users/jasonkwan/git_repos/autometa_PRs/PR_118/Autometa/databases/ncbi/
 -----------------------------------------------------
executor >  local (8)
executor >  local (8)
[84/3c4311] process > AUTOMETA:LENGTH_FILTER (filtering metagenome metagenome)                                [100%] 1 of 1 ✔
[1a/98641e] process > AUTOMETA:KMER_COVERAGE (Calculating k-mer coverage for metagenome)                      [100%] 1 of 1 ✔
[a9/60fb26] process > AUTOMETA:ORFS (Calling orfs for metagenome)                                             [100%] 1 of 1 ✔
[fa/0545af] process > AUTOMETA:MARKERS (Finding markers for metagenome)                                       [100%] 1 of 1 ✔
[0e/58e148] process > AUTOMETA:TAXON_ASSIGNMENT:DIAMOND (diamond blastp on metagenome)                        [100%] 1 of 1 ✔
[68/1b3a3e] process > AUTOMETA:TAXON_ASSIGNMENT:LCA (Assigning LCA to metagenome)                             [100%] 1 of 1 ✔
[86/f1e058] process > AUTOMETA:TAXON_ASSIGNMENT:MAJORITY_VOTE (Performing taxon majority vote on metagenome)  [100%] 1 of 1 ✔
[c0/8b1257] process > AUTOMETA:TAXON_ASSIGNMENT:SPLIT_KINGDOMS (Splitting votes into kingdoms for metagenome) [100%] 1 of 1, failed: 1 ✘
[-        ] process > AUTOMETA:KMERS                                                                          -
[-        ] process > AUTOMETA:BINNING                                                                        -
[-        ] process > AUTOMETA:UNCLUSTERED_RECRUITMENT                                                        -
[-        ] process > AUTOMETA:BINNING_SUMMARY                                                                -
Error executing process > 'AUTOMETA:TAXON_ASSIGNMENT:SPLIT_KINGDOMS (Splitting votes into kingdoms for metagenome)'

Caused by:
  Process `AUTOMETA:TAXON_ASSIGNMENT:SPLIT_KINGDOMS (Splitting votes into kingdoms for metagenome)` terminated with an error exit status (1)

Command executed:

  autometa-taxonomy     --votes metagenome.votes.tsv     --output .     --prefix metagenome     --split-rank-and-write superkingdom     --assembly metagenome.filtered.fna     --ncbi /ncbi
  # Handling case where no archaea were recovered...
  if [[ ! -f metagenome.archaea.fna ]]
  then touch metagenome.archaea.fna
  fi

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File "/opt/conda/bin/autometa-taxonomy", line 33, in <module>
      sys.exit(load_entry_point('Autometa==2.0a0', 'console_scripts', 'autometa-taxonomy')())
    File "/opt/conda/lib/python3.8/site-packages/Autometa-2.0a0-py3.8.egg/autometa/taxonomy/vote.py", line 368, in main
      taxa_df = pd.read_csv(args.taxonomy, sep="\t", index_col="contig")
  AttributeError: 'Namespace' object has no attribute 'taxonomy'

Work dir:
  /Users/jasonkwan/git_repos/autometa_PRs/PR_118/Autometa/work/c0/8b1257c65ed17507f13c5045fa9caa

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

Is this an error in the script itself?

[jason-c-kwan]

I have tested the latest commit, and it successfully runs.

[chasemc]

Nice work @WiscEvan