Documentation (#34)

* initial commit of documentation for readthedocs format

* first commit

* Commiting all files in Scripts folder

* Modification to COPYRIGHT and config.py

* stable build

* Removed scripts_2, can now be fetched and run by any user

* Create README.md

File to explain how the documentation can be installed and used by anyone

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* automatic argparse, autosummary, automatic copyright update, last updated, can be run by anyone

* Update README.md

* Update README.md

* modified all copyrights added usage and autodoc for all scripts

* changed autometa to run_autometa

* initial commit of documentation for readthedocs format

* first commit

* Commiting all files in Scripts folder

* Modification to COPYRIGHT and config.py

* stable build

* Removed scripts_2, can now be fetched and run by any user

* Create README.md

File to explain how the documentation can be installed and used by anyone

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* automatic argparse, autosummary, automatic copyright update, last updated, can be run by anyone

* Update README.md

* Update README.md

* modified all copyrights added usage and autodoc for all scripts

* changed autometa to run_autometa

* Applied changes to scripts and docs source files to remove warnings emitted by Sphinx.

* fixes PR Review comments Sidduppal#1

* environment.yaml and .readthedocs.yaml files for readthedocs integration

* attempt to reduce memory consumption in readthedocs.org. Removed packages already available in readthedocs docker image

* removed most dependencies from conda env and have moved to docs/requirements.txt

* changed conda file to conda environment.

* added pip in environment.yaml dependencies.

* removed numba from docs/requirements.txt

* fixes Sidduppal/documentation#1. minor changes in template.py to reflect changes to overall source code. Added work_queue.py to remove warning for readthedocs.org. Fixed suggestions from Sidduppal/documentation#1 PR review.

* addressed Jason's comments for merge to dev

* todo box added, function to automatically input modules, sidebar, and other comments by evan

* Updated markers docstring (fixed incorrect f-string) to allow parameters/attributes/methods to be imported

* addressed Jason's comments for merge to dev

* todo box added, function to automatically input modules, sidebar, and other comments by evan

* final changes, added Ian to copyright, removed hardcoded copyright

* added reference for todo.py

Co-authored-by: EvanRees <erees@wisc.edu>

.readthedocs.yaml

@@ -0,0 +1,20 @@
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Optionally build your docs in additional formats such as PDF and ePub
+formats: all
+
+conda:
+  environment: docs/environment.yaml
+
+python:
+    version: 3.7
+    install:
+        - requirements: docs/requirements.txt

Dockerfile

@@ -25,13 +25,13 @@ conda install -c bioconda -c conda-forge --yes \
     tqdm \
     numpy \
     scikit-learn \
+    scipy \
     samtools \
     bedtools \
     bowtie2 \
     hmmer \
     prodigal \
     diamond \
-    ipython \
     ndcctools \
     parallel \
     requests \

README.md

@@ -34,6 +34,7 @@ conda install -n autometa -c bioconda -c conda-forge --yes \
     tqdm \
     numpy \
     scikit-learn \
+    scipy \
     samtools \
     bedtools \
     bowtie2 \

autometa.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,6 +19,7 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
+COPYRIGHT
 
 Main script to run Autometa
 """
@@ -34,6 +36,8 @@
 logger = logging.getLogger('autometa')
 
 
+__version__ = "2.0.0"
+
 def init_logger(fpath=None, level=logging.INFO):
     """Initialize logger.
 
@@ -137,7 +141,6 @@ def main(args):
         main(args)
     except KeyboardInterrupt:
         logger.info('User cancelled run. Exiting...')
-        sys.exit(1)
     except Exception as err:
         issue_request = '''
 

autometa/binning/recursive_dbscan.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,6 +19,7 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
+COPYRIGHT
 
 Cluster contigs recursively searching for bins with highest completeness and purity.
 """
@@ -238,8 +240,7 @@ def get_clusters(master_df, markers_df, domain='bacteria', completeness=20., pur
     ----------
     master_df : pd.DataFrame
         index=contig,
-        cols=
-            embedded-kmers are cols 'x','y' and 'z'
+        cols=['x','y','coverage']
     markers_df : pd.DataFrame
         wide format, i.e. index=contig cols=[marker,marker,...]
     domain : str
@@ -304,10 +305,9 @@ def binning(master, markers, domain='bacteria', completeness=20., purity=90.,
     ----------
     master : pd.DataFrame
         index=contig,
-        cols=
-            embedded-kmers are cols 'x','y' and 'z'
-            taxa cols should be present if `taxonomy` is True.
-            i.e. [taxid,superkingdom,phylum,class,order,family,genus,species]
+        cols=['x','y']
+        taxa cols should be present if `taxonomy` is True.
+        i.e. [taxid,superkingdom,phylum,class,order,family,genus,species]
     markers : pd.DataFrame
         wide format, i.e. index=contig cols=[marker,marker,...]
     domain : str

autometa/common/coverage.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,8 +19,9 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
+COPYRIGHT
 
-Construct contig coverage table given an input assembly and reads or alignments.
+Calculates coverage of contigs
 """
 
 
@@ -86,29 +88,25 @@ def make_length_table(fasta, out):
 
 def get(fasta, out, fwd_reads=None, rev_reads=None, se_reads=None, sam=None,
     bam=None, lengths=None, bed=None, nproc=1):
-    """Get coverages for assembly `fasta` file using provided files:
-
-    Either:
-        `fwd_reads` and `rev_reads` and/or `se_reads`
-    or:
-        `sam`
-    or:
-        `bam`
-    or:
-        `bed`
-
-    Will begin coverage calculation based on files provided checking in the
-    following order:
-        1. `bed`
-        2. `bam`
-        3. `sam`
-        4. `fwd_reads` and `rev_reads` and `se_reads`
-
-    Event sequence to calculate contig coverages:
-    1. align paired-end reads to generate alignment.sam
-    2. sort samfile to generate alignment.bam
-    3. calculate assembly coverages to generate alignment.bed
-    4. calculate contig coverages to generate coverage.tsv
+    """Get coverages for assembly `fasta` file using provided files.
+    Either: `fwd_reads` and `rev_reads` and/or `se_reads` or,`sam`, or `bam`, or `bed`.
+
+    Notes
+    -----
+        Will begin coverage calculation based on files provided checking in the
+        following order:
+
+        #. `bed`
+        #. `bam`
+        #. `sam`
+        #. `fwd_reads` and `rev_reads` and `se_reads`
+
+        Event sequence to calculate contig coverages:
+
+        #. align reads to generate alignment.sam
+        #. sort samfile to generate alignment.bam
+        #. calculate assembly coverages to generate alignment.bed
+        #. calculate contig coverages to generate coverage.tsv
 
 
     Parameters

autometa/common/exceptions.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,6 +19,7 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
+COPYRIGHT
 
 File containing customized AutometaExceptions for more specific exception handling
 """

autometa/common/external/bedtools.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,7 +19,7 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
-
+COPYRIGHT
 Script containing wrapper functions for bedtools.
 """
 

autometa/common/external/bowtie.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,7 +19,7 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
-
+COPYRIGHT
 Script containing wrapper functions for bowtie2.
 """
 

autometa/common/external/diamond.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,7 +19,7 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
-
+COPYRIGHT
 Class and functions related to running diamond on metagenome sequences
 """
 
@@ -38,13 +39,44 @@
 
 
 class DiamondResult:
-    """docstring for DiamondResult.
+    """DiamondResult class
 
-    Some operator overloading here...
-    For other examples of this see:
-        https://www.geeksforgeeks.org/operator-overloading-in-python/
+    Parameters
+    ----------
+    qseqid : str
+        query sequence ID
+    sseqid : str
+        subject sequence ID
+    pident : float
+        Percentage of identical matches.
+    length : int
+        Alignment length.
+    mismatch : int
+        Number of mismatches.
+    gapopen : int
+        Number of gap openings.
+    qstart : int
+        Start of alignment in query.
+    qend : int
+        End of alignment in query.
+    sstart : int
+        Start of alignment in subject.
+    send : int
+        End of alignment in subject sequence.
+    evalue : float
+        Expect value.
+    bitscore : float
+        Bitscore.
+
+    Attributes
+    ----------
+    sseqids : dict
+        {sseqid:parameters, sseqid:parameters, ...}
+    qseqid: str
+        result query sequence ID
 
     """
+
     def __init__(self, qseqid, sseqid, pident, length, mismatch, gapopen,
         qstart, qend, sstart, send, evalue, bitscore):
         self.qseqid = qseqid
@@ -365,7 +397,9 @@ def main(args):
 
 if __name__ == '__main__':
     import argparse
-    parser = argparse.ArgumentParser('Retrieves blastp hits with provided input assembly')
+    parser = argparse.ArgumentParser(description="""
+    Retrieves blastp hits with provided input assembly
+    """)
     parser.add_argument('fasta', help='</path/to/faa/file>')
     parser.add_argument('database', help='</path/to/diamond/formatted/database>')
     parser.add_argument('acc2taxids', help='</path/to/ncbi/prot.accession2taxid.gz>')

autometa/common/external/hmmer.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,7 +19,7 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
-
+COPYRIGHT
 Functions related to running hmmer on metagenome sequences
 """
 

autometa/common/external/prodigal.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 """
+COPYRIGHT
 Copyright 2020 Ian J. Miller, Evan R. Rees, Kyle Wolf, Siddharth Uppal,
 Shaurya Chanana, Izaak Miller, Jason C. Kwan
 
@@ -18,6 +19,7 @@
 
 You should have received a copy of the GNU Affero General Public License
 along with Autometa. If not, see <http://www.gnu.org/licenses/>.
+COPYRIGHT
 
 Functions to retrieve orfs from provided assembly using prodigal
 """
@@ -176,9 +178,32 @@ def contigs_from_headers(fpath):
     First determines if all of ID=3495691_2 from description is in header.
     "3495691_2" represents the 3,495,691st gene in the 2nd sequence.
 
-    i.e. : record.description
+    Example
+    -------
+    .. code-block:: python
+
+        #: prodigal versions < 2.6 record
+        >>>record.id
+        'k119_1383959_3495691_2'
+
+        >>>record.description
         'k119_1383959_3495691_2 # 688 # 1446 # 1 # ID=3495691_2;partial=01;start_type=ATG;rbs_motif=None;rbs_spacer=None'
-                                                      ^       ^
+
+        >>>record.description.split('#')[-1].split(';')[0].strip()
+        'ID=3495691_2'
+
+        >>>orf_id = '3495691_2'
+        '3495691_2'
+
+        >>>record.id.replace(f'_{orf_id}', '')
+        'k119_1383959'
+
+        #: prodigal versions >= 2.6 record
+        >>>record.id
+        'k119_1383959_2'
+        >>>record.id.rsplit('_',1)[0]
+        'k119_1383959'
+
     Parameters
     ----------
     fpath : str
@@ -189,11 +214,6 @@ def contigs_from_headers(fpath):
     dict
         contigs translated from prodigal ORF description.  {orf_id:contig_id, ...}
 
-    Raises
-    -------
-    ExceptionName
-        Why the exception is raised.
-
     """
     version = get_versions('prodigal')
     if version.count('.') >= 2: