Docs: Update docs for latest version

segrelab · May 29, 2023 · cece96a · cece96a
1 parent ac31f51
commit cece96a
Show file tree

Hide file tree

Showing 5 changed files with 436 additions and 246 deletions.
diff --git a/README.rst b/README.rst
@@ -1,41 +1,229 @@
-=======================================
-Microbial Interaction Database Pipeline
-=======================================
+MiCoNE - Microbial Co-occurrence Network Explorer
+=================================================
 
+|Build Status| |Documentation Status| |codecov| |CodeFactor| |Updates|
+|Code style: black|
 
-.. image:: https://img.shields.io/pypi/v/micone.svg
-        :target: https://pypi.python.org/pypi/micone
+``MiCoNE`` is a Python package for the exploration of the effects of
+various possible tools used during the 16S data processing workflow on
+the inferred co-occurrence networks. It is also developed as a flexible
+and modular pipeline for 16S data analysis, offering parallelized, fast
+and reproducible runs executed for different combinations of tools for
+each step of the data processing workflow. It incorporates various
+popular, publicly available tools as well as custom Python modules and
+scripts to facilitate inference of co-occurrence networks from 16S data.
 
-.. image:: https://img.shields.io/travis/dileep-kishore/micone.svg
-        :target: https://travis-ci.org/dileep-kishore/micone
+-  Free software: MIT license
+-  Documentation: https://micone.readthedocs.io/
 
-.. image:: https://readthedocs.org/projects/micone/badge/?version=latest
-        :target: https://micone.readthedocs.io/en/latest/?badge=latest
-        :alt: Documentation Status
+Manuscript can be found on
+`bioRxiv <https://www.biorxiv.org/content/10.1101/2020.09.23.309781v2>`__
+(to be updated with link to publication).
 
+Features
+--------
 
-.. image:: https://pyup.io/repos/github/dileep-kishore/micone/shield.svg
-     :target: https://pyup.io/repos/github/dileep-kishore/micone/
-     :alt: Updates
+-  Plug and play architecture: allows easy additions and removal of new
+   tools
+-  Flexible and portable: allows running the pipeline on local machine,
+   compute cluster or the cloud with minimal configuration change
+   through the usage of `nextflow <www.nextflow.io>`__
+-  Parallelization: automatic parallelization both within and across
+   samples (needs to be enabled in the ``nextflow.config`` file)
+-  Ease of use: available as a minimal ``Python`` library (without the
+   pipeline) or as a full ``conda`` package
 
+Installation
+------------
 
+Installing the ``conda`` package:
 
-The pipeline that powers the Microbial Interaction Network Database
+.. code:: sh
 
+   mamba env create -n micone -f https://raw.githubusercontent.com/segrelab/MiCoNE/master/env.yml
 
-* Free software: MIT license
-* Documentation: https://micone.readthedocs.io.
+..
 
+   NOTE: 1. MiCoNE requires the ``mamba`` package manager, otherwise
+   ``micone init`` will not work. 2. Direct installation via anaconda
+   cloud will be available soon.
 
-Features
+Installing the minimal ``Python`` library:
+
+.. code:: sh
+
+   pip install micone
+
+..
+
+   NOTE: The ``Python`` library does not provide the functionality to
+   execute pipelines
+
+Workflow
 --------
 
-* TODO
+.. figure:: assets/pipeline.png
+   :alt: pipeline
+
+   pipeline
+
+It supports the conversion of raw 16S sequence data into co-occurrence
+networks. Each process in the pipeline supports alternate tools for
+performing the same task, users can use the configuration file to change
+these values.
+
+Usage
+-----
+
+The ``MiCoNE`` pipelines comes with an easy-to-use CLI. To get a list of
+subcommands you can type:
+
+.. code:: sh
+
+   micone --help
+
+Supported subcommands:
+
+1. ``install`` - Initializes the package and environments (creates
+   ``conda`` environments for various pipeline processes)
+2. ``init`` - Initialize the nextflow templates for the micone workflow
+3. ``clean`` - Cleans files from a pipeline run (cleans temporary data,
+   log files and other extraneous files)
+4. ``validate-results`` - Check the results of the pipeline execution
+
+Initializing the environments
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In order to run the pipeline various ``conda`` environments must first
+be installed on the system. Use the following comand to initialize all
+the environments:
+
+.. code:: sh
+
+   micone install
+
+Or to initialize a particular environment use:
+
+.. code:: sh
+
+   micone install -e "micone-qiime2"
+
+The list of supported environments are: - micone-cozine - micone-dada2 -
+micone-flashweave - micone-harmonies - micone-mldm - micone-propr -
+micone-qiime2 - micone-sparcc - micone-spieceasi - micone-spring
+
+Initializing the pipeline template
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To initialize the full pipeline (from raw 16S sequencing reads to
+co-occurrence networks):
+
+.. code:: sh
+
+   micone init -w <workflow> -o <path/to/folder>
+
+Other supported pipeline templates are (work in progress): - full - ni -
+op_ni - ta_op_ni
+
+To run the pipeline, update the relevant config files (see next
+section), activate the ``micone`` environment and run the ``run.sh``
+script that was copied to the directory:
+
+.. code:: sh
+
+   bash run.sh
+
+This runs the pipeline locally using the config options specified.
+
+To run the pipeline on an SGE enabled cluster, add the relevant
+project/resource allocation flags to the ``run.sh`` script and run as:
+
+.. code:: sh
+
+   qsub run.sh
+
+Configuration and the pipeline template
+---------------------------------------
+
+The pipeline template for the micone “workflow” (see previous section
+for list of supported options) is copied to the desired folder after
+running ``micone init -w <workflow>``. The template folder contains the
+following folders and files:
+
+-  nf_micone: Folder contatining the ``micone`` default configs, data,
+   functions, and modules
+-  templates: Folder containing the templates (scripts) that are
+   executed during the pipeline run
+-  main.nf: The pipeline “workflow” defined in the ``nextflow`` DSL 2
+   specification
+-  nextflow.config: The configuration for the pipeline. This file needs
+   to be modified in order to change any configuration options for the
+   pipeline run
+-  metadata.json: Contains the basic metadata that describes the dataset
+   that is to be processed. Should be updated accordingly before
+   pipeline execution
+-  samplesheet.csv: The file that contains the locations of the input
+   data necessary for the pipeline run. Should be updated accordingly
+   before pipeline execution
+-  run.sh: The ``bash`` script that contains commands used to execute
+   the ``nextflow`` pipeline
+
+The folder ``nf_micone/configs`` contains the default configs for all
+the ``micone`` pipeline workflows. These options can also be viewed in
+tabular format in the
+`documentation <https://micone.readthedocs.io/en/latest/usage.html#configuring-the-pipeline>`__.
+
+For example, to change the tool used for OTU assignment to ``dada2`` and
+``deblur``, you can add the following to ``nextflow.config``:
+
+.. code:: groovy
+
+   // ... config initialization
+   params {
+          // ... other config options
+          denoise_cluster {
+           otu_assignment {
+               selection = ['dada2', 'deblur']
+           }
+       }
+   }
+
+Example configuration files used for the analyses in the manuscript can
+be found
+`here <https://github.com/segrelab/MiCoNE-pipeline-paper/tree/master/scripts/runs>`__.
+
+Visualization of results (coming soon)
+--------------------------------------
+
+The results of the pipeline execution can be visualized using the
+scripts in the `manuscript
+repo <https://github.com/segrelab/MiCoNE-pipeline-paper/tree/master/scripts>`__
+
+Know issues
+-----------
+
+1. If you have a version of ``julia`` that is preinstalled, make sure
+   that it does not conflict with the version downloaded by the
+   ``micone-flashweave`` environment
+2. The data directory (``nf_micone/data``) needs to be manually
+   downloaded (link here).
 
 Credits
 -------
 
-This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
-
-.. _Cookiecutter: https://github.com/audreyr/cookiecutter
-.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
+This package was created with
+`Cookiecutter <https://github.com/audreyr/cookiecutter>`__ and the
+`audreyr/cookiecutter-pypackage <https://github.com/audreyr/cookiecutter-pypackage>`__
+project template.
+
+.. |Build Status| image:: https://github.com/segrelab/MiCoNE/workflows/build/badge.svg
+.. |Documentation Status| image:: https://readthedocs.org/projects/micone/badge/?version=latest
+   :target: https://micone.readthedocs.io/en/latest/?badge=latest
+.. |codecov| image:: https://codecov.io/gh/segrelab/MiCoNE/branch/master/graph/badge.svg?token=2tKiI0lUJb
+   :target: https://codecov.io/gh/segrelab/MiCoNE
+.. |CodeFactor| image:: https://www.codefactor.io/repository/github/segrelab/micone/badge
+   :target: https://www.codefactor.io/repository/github/segrelab/micone
+.. |Updates| image:: https://pyup.io/repos/github/segrelab/MiCoNE/shield.svg
+   :target: https://pyup.io/repos/github/segrelab/MiCoNE/
+.. |Code style: black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
+   :target: https://github.com/ambv/black
diff --git a/docs/data_import_export.rst b/docs/data_import_export.rst
@@ -1,5 +1,5 @@
 ======================
-Data Import and Export
+Data Import and Export (Python API)
 ======================
 
 1. Importing a network from a Table

diff --git a/docs/installation.rst b/docs/installation.rst
@@ -4,41 +4,53 @@
 Installation
 ============
 
+From anaconda (complete pipeline)
+---------------------------------
 
-Stable release (only Python package)
-------------------------------------
+This is the preferred way to install the complete MiCoNE pipeline.
 
-To install the MiCoNE Python package, run this command in your terminal:
+First clone the public repository:
 
-.. code-block:: console
+.. code:: sh
 
-    $ pip install MiCoNE
+   git clone git://github.com/segrelab/MiCoNE
 
-If you don't have `pip`_ installed, this `Python installation guide`_ can guide
-you through the process.
+Then create a conda environment and install the required packages:
 
-.. _pip: https://pip.pypa.io
-.. _Python installation guide: http://docs.python-guide.org/en/latest/starting/installation/
+.. code:: sh
 
-.. note:: This does not install the `nextflow`_ related pipeline elements. Those can be installed through anaconda (see below).
+   cd MiCoNE
+   mamba env create -n micone -f env.yml
 
-From anaconda (complete pipeline)
----------------------------------
+Or you can directly install from the repository:
 
-This is the preferred way to install the complete MiCoNE pipeline.
+.. code:: sh
 
-First clone the public repository:
+   mamba env create -n micone -f https://raw.githubusercontent.com/segrelab/MiCoNE/master/env.yml
 
-.. code-block:: console
+..
 
-   $ git clone git://github.com/segrelab/MiCoNE
+   NOTE: 1. MiCoNE requires the ``mamba`` package manager, otherwise
+   ``micone init`` will not work. 2. Direct installation via anaconda
+   cloud will be available soon.
 
-Then create a conda environment and install the required packages:
 
-.. code-block:: console
+From pip (only Python package)
+------------------------------------
+
+To install the MiCoNE Python package, run this command in your terminal:
+
+.. code:: sh
+
+    pip install MiCoNE
+
+If you don't have `pip`_ installed, this `Python installation guide`_ can guide
+you through the process.
+
+.. _pip: https://pip.pypa.io
+.. _Python installation guide: http://docs.python-guide.org/en/latest/starting/installation/
 
-   $ cd MiCoNE
-   $ conda env create -n micone -f env.yml
+.. note:: This does not install the `nextflow`_ related pipeline elements, and does not provide the functionality to execute pipelines. Those can be installed through anaconda (see above).
 
 
 From sources (only Python package)
@@ -48,21 +60,21 @@ The sources for Microbial Interaction Database Pipeline can be downloaded from t
 
 You can either clone the public repository:
 
-.. code-block:: console
+.. code:: sh
 
-    $ git clone git://github.com/segrelab/MiCoNE
+    git clone git://github.com/segrelab/MiCoNE
 
 Or download the `tarball`_:
 
-.. code-block:: console
+.. code:: sh
 
-    $ curl  -OL https://github.com/segrelab/MiCoNE/tarball/master
+    curl  -OL https://github.com/segrelab/MiCoNE/tarball/master
 
 Once you have a copy of the source, you can install it with:
 
-.. code-block:: console
+.. code:: sh
 
-    $ pip install .
+    pip install .
 
 
 .. _Github repo: https://github.com/segrelab/MiCoNE