MNT patch 3.0.2

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.3.0
+    rev: v4.4.0
     hooks:
       - id: check-yaml
       - id: end-of-file-fixer
@@ -10,32 +10,32 @@ repos:
       - id: check-toml
 
   - repo: https://github.com/psf/black
-    rev: 22.6.0
+    rev: 23.3.0
     hooks:
       - id: black
       - id: black-jupyter
 
   - repo: https://github.com/kynan/nbstripout
-    rev: 0.5.0
+    rev: 0.6.1
     hooks:
       - id: nbstripout
 
   - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: v2.7.1
+    rev: v3.0.0-alpha.6
     hooks:
       - id: prettier
         additional_dependencies:
           - prettier@2.3.2
           - "prettier-plugin-toml"
 
   - repo: https://github.com/python-jsonschema/check-jsonschema
-    rev: 0.17.1
+    rev: 0.22.0
     hooks:
       - id: check-github-workflows
       - id: check-readthedocs
 
   - repo: https://github.com/pycqa/flake8
-    rev: "5.0.4"
+    rev: "6.0.0"
     hooks:
       - id: flake8
         args:

CHANGELOG.md

@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ## [Unreleased] - started 2023-02
 
+## [3.0.2] 2023-05-19
+
+### Added
+
+- `riboseq_sample_name_map`, `riboseq_condition_name_map` to `rpbp_predictions_dashboard.py`
+- Exception for duplicated transcript ids with _de novo_ annotation.
+
+### Fixed
+
+- STAR output
+- Redundant transcripts with _de novo_ annotation in `summarize_rpbp_predictions.py`
+- ORF numbers in labels
+
 ## [3.0.1] 2023-02-10
 
 ### Changed

MANIFEST.in

@@ -1,3 +1,4 @@
 include README.md
 include src/rpbp/models/*/*.stan
 include src/rpbp/analysis/rpbp_predictions/dashboard/assets/*.*
+include src/rpbp/analysis/profile_construction/dashboard/assets/*.*

docs/source/api-app-profiles.rst

@@ -5,18 +5,15 @@ Launch a Dash app for quality control and visualization of ribosome profiling da
 
 .. code-block:: bash
 
-    usage: rpbp-profile-construction-dashboard [-h] [-d] [--host] [--port] config
+    usage: rpbp-profile-construction-dashboard [-h] [-c CONFIG] [-d] [--host] [--port]
 
 
-Positional Arguments
---------------------
-
--config
-    A YAML configuration file. The same used to run the pipeline.
-
 Named Arguments
 ---------------
 
+-c, --config
+    A YAML configuration file (required). The same used to run the pipeline.
+
 -d, --debug
     Enable debug mode.
 

docs/source/api-app-rpbp.rst

@@ -5,18 +5,15 @@ Launch a Dash app to visualize Ribo-seq ORF predicted with Rp-Bp.
 
 .. code-block:: bash
 
-    usage: rpbp-predictions-dashboard [-h] [-d] [--host] [--port] config
+    usage: rpbp-predictions-dashboard [-h] [-c CONFIG] [-d] [--host] [--port]
 
 
-Positional Arguments
---------------------
-
--config
-    A YAML configuration file. The same used to run the pipeline.
-
 Named Arguments
 ---------------
 
+-c, --config
+    A YAML configuration file (required). The same used to run the pipeline.
+
 -d, --debug
     Enable debug mode.
 

docs/source/apps.rst

@@ -87,7 +87,7 @@ To launch the *profile construction dashboard*
 
 .. code-block:: bash
 
-    rpbp-profile-construction-dashboard <config>
+    rpbp-profile-construction-dashboard -c CONFIG
 
 
 The application has multiple views to facilitate quality control, *e.g.*
@@ -104,7 +104,7 @@ To launch the *predictions dashboard*
 
 .. code-block:: bash
 
-    rpbp-predictions-dashboard <config>
+    rpbp-predictions-dashboard -c CONFIG
 
 
 The application has multiple views to facilitate ORF discovery, including an integrated `IGV browser <https://software.broadinstitute.org/software/igv/>`_ for the visual exploration of predicted Ribo-seq ORFs, *e.g.*

docs/source/faq.rst

@@ -5,6 +5,9 @@ Frequently asked questions
 * :ref:`q2`
 * :ref:`q3`
 * :ref:`q4`
+* :ref:`q5`
+* :ref:`q6`
+* :ref:`q7`
 
 .. _q1:
 
@@ -19,7 +22,7 @@ Example calls are also given in the user guide and tutorials.
 How do I launch the app remotely?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-By default, the application is opened in a browser page on *localhost:8050*. You don't have to actually worry about this. But you can also specify a ``--host`` and a ``--port`` when calling the app, enabling you to launch it from a remote server. In the latter case, however, you have to open a browser page at the correct address. For example, if you use ``--host 123.123.123.123``, then open a page on *http://123.123.123.123:8050/*.
+By default, the application is opened in a browser page on *localhost:8050*. You don't have to actually worry about this. But you can also specify a ``--host`` and a ``--port`` when calling the app, enabling you to launch it from a remote server. In the latter case, however, you have to open a browser page at the correct address. For example, if you use ``--host 123.123.123.123``, then open a page on *http://123.123.123.123:8050/*. To launch the app from a container, see :ref:`tutorial_containers`.
 
 .. _q3:
 
@@ -34,3 +37,26 @@ I have my own alignments, can I use **Rp-Bp**?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The short answer is yes. The pipeline is designed to handle all steps from raw FASTQ files up to the final list of translated Ribo-seq ORFs, but you can start the pipeline from any step. Check the tutorial :ref:`existing-alignment`.
+
+
+.. _q5:
+
+I get errors when calling ``summarize-rpbp-predictions``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The most common errors *e.g.* ``AttributeError: 'float' object has no attribute 'left'`` are due to the bin width for the `Circos <http://circos.ca/>`_ plot. Try reducing it using ``--circos-bin-width VALUE`` (default VALUE: 10000000). You may also have to use ``--circos-show-chroms`` if your organism has a different nomenclature. Use ``summarize-rpbp-predictions --help`` for details.
+
+.. _q6:
+
+The IGV browser does not load or the predictions app fails to start
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The most likely reason is that your reference genome sequence (given by the config key ``fasta``) is not indexed, *i.e* the file **\*.fasta.fai** is missing. You can create the missing index using `samtools faidx <http://www.htslib.org/doc/samtools-faidx.html>`_.
+
+
+.. _q7:
+
+I get ``RuntimeWarning: invalid value encountered in divide res, _ = _lowess(y, x, x, np.ones_like(x),``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This happens for 3 nt ORFs.

docs/source/index.rst

@@ -9,7 +9,7 @@ Rp-Bp: Ribosome profiling with Bayesian predictions
 Introduction
 ------------
 
-Ribosome profiling (Ribo-seq) is an RNA-sequencing-based readout of RNA translation. Isolation and deep-sequencing of ribosome-protected RNA fragments (ribosome footprints) provides a genome-wide snapshot of the translatome at sub-codon resolution. Aligned by their P-site positions, the footprints from actively translating ribosomes should exhibit a 3-nt periodicity along the ORF (the ribosome moves by steps of a codon, *i.e.* 3-nt). To select reads and predict translation, most methods, including **Rp-Bp**, take advantage of this periodic signal.
+Ribosome profiling (Ribo-seq) is an RNA-sequencing-based readout of RNA translation. Isolation and deep-sequencing of ribosome-protected RNA fragments (ribosome footprints) provides a genome-wide snapshot of the translatome at sub-codon resolution. Aligned by their P-site positions, the footprints from actively translating ribosomes should exhibit a 3-nt periodicity. To select reads and predict translation, most methods, including **Rp-Bp**, take advantage of this periodic signal.
 
 **Rp-Bp** is an unsupervised Bayesian approach to predict translated open reading frames (ORFs) from ribosome profiles, using the automatic Bayesian Periodic fragment length and ribosome P-site offset Selection (BPPS), *i.e.* read lengths and ribosome P-site offsets are inferred from the data, without supervision. **Rp-Bp** is able to handle *de novo* translatome annotation by directly assessing the periodicity of the Ribo-seq signal.
 

docs/source/installation.rst

@@ -15,7 +15,7 @@ To use a container (Docker or Singularity) with **Rp-Bp** pre-installed, simply
     # ...singularity
     singularity pull rpbp.sif docker://quay.io/biocontainers/rpbp:<tag>
 
-There is no *latest* tag, you need to specify the version tag. See `rpbp/tags <https://quay.io/repository/biocontainers/rpbp?tab=tags>`_ for valid values for <tag>.
+There is no *latest* tag, you need to specify the version tag. See `rpbp/tags <https://quay.io/repository/biocontainers/rpbp?tab=tags>`_ for valid values for <tag>. Check the `Tutorials <tutorial.html>`_ on how to use the containers.
 
 
 .. _conda_install:
@@ -36,6 +36,10 @@ or create an environment, called *rpbp*, containing the **Rp-Bp** package
 
     conda create -n rpbp rpbp
 
+.. tip::
+
+    `Mamba <https://mamba.readthedocs.io/en/latest/user_guide/mamba.html#mamba>`_ can be used as a drop-in replacement, you can swap almost all commands between conda and mamba.
+
 .. _pypi_install:
 
 Contributing to **Rp-Bp**
@@ -49,12 +53,12 @@ To install the local VCS project in development mode
     conda create -n rpbp_dev
     # ...activate it...
     conda activate rpbp_dev
-    # ... and only install dependencies
-    (rpbp_dev) conda install --only-deps rpbp
+    # ... and only install dependencies (rpbp_dev is now activated)
+    conda install --only-deps rpbp
     # clone the git repository
-    (rpbp_dev) git clone https://github.com/dieterich-lab/rp-bp.git && cd rp-bp
+    git clone https://github.com/dieterich-lab/rp-bp.git && cd rp-bp
     # install
-    (rpbp_dev) pip --verbose install --no-deps -e . 2>&1 | tee install.log
+    pip --verbose install --no-deps -e . 2>&1 | tee install.log
 
 
 PyPI installation
@@ -69,17 +73,11 @@ However, if you already have the required dependencies installed on your system,
     python3 -m venv rpbp_pypi
     # ... activate it ...
     source rpbp_pypi/bin/activate
-    # ... and install Rp-Bp
-    (rpbp_pypi) pip install rpbp
-
+    # ... and install Rp-Bp (rpbp_pypi is now activated)
+    pip install rpbp
 
-Required dependencies
-"""""""""""""""""""""
 
-* `Flexbar <https://github.com/seqan/flexbar>`_
-* `Bowtie 2 <http://bowtie-bio.sourceforge.net/bowtie2/index.shtml>`_
-* `STAR <https://github.com/alexdobin/STAR>`_
-* `Samtools <http://www.htslib.org>`_
+**Required dependencies:** `Flexbar <https://github.com/seqan/flexbar>`_, `Bowtie 2 <http://bowtie-bio.sourceforge.net/bowtie2/index.shtml>`_, `STAR <https://github.com/alexdobin/STAR>`_, `Samtools <http://www.htslib.org>`_.
 
 .. warning::
 
@@ -101,7 +99,7 @@ or remove the package installed in another environment
 .. code-block:: bash
 
     # remove the rpbp package from myenv environment...
-    (myenv) conda remove -n myenv rpbp
+    conda remove -n myenv rpbp
 
 
 To remove **Rp-Bp** if installed with pip

docs/source/tutorial-cel.rst

@@ -60,17 +60,20 @@ Launch any of the web applications with
 
 .. code-block:: bash
 
-    rpbp-profile-construction-dashboard c-elegans-test.yaml
+    rpbp-profile-construction-dashboard -c c-elegans-test.yaml
 
 or
 
 .. code-block:: bash
 
-    rpbp-predictions-dashboard c-elegans-test.yaml
+    rpbp-predictions-dashboard -c c-elegans-test.yaml
 
 
 To navigate the apps is easy, just follow the "hints". Most items are interactive. Press ``CTRL+C`` to quit.
 
+.. attention::
+
+    For the apps only, the configuration file is passed using a (required) named argument ``-c/--config CONFIG``.
 
 .. note::
 

docs/source/tutorial-containers.rst

@@ -0,0 +1,115 @@
+.. _tutorial_containers:
+
+How to use the containers
+=========================
+
+First pull a Docker or Singularity container, see `installation <installation.html>`_. For this tutorial, we use a general mechanism for persisting data, which allows to create and modify files on the host system from within the container. We use the *c-elegans-chrI-example* with the *c-elegans-test.yaml* configuration file, see also `How to run the pipeline <tutorial-cel.html>`_.
+
+.. note::
+
+    You can also launch a container with an interactive shell *e.g.* ``docker run -it quay.io/biocontainers/rpbp:3.0.1--py310h30d9df9_0 /bin/bash`` or ``singularity shell rpbp.sif``. With ``singularity shell``, ``$HOME`` is mounted by default.
+
+.. attention::
+
+    In the following, do not forget to modify the tag ``3.0.1--py310h30d9df9_0`` according to what you pulled! For Singularity, adjust the name of the Singularity image format file ``rpbp.sif`` and/or the path according to your needs.
+
+
+How to run the pipeline
+-----------------------
+
+To run the pipeline, change the paths in the configuration file to point to the location where the directory is mounted in the container. You can do this using a text editor, or simply by modifying the file in place
+
+.. code-block:: bash
+
+    sed -i 's|/path/to/your/c-elegans-example|/data|g' c-elegans-test.yaml
+
+
+.. important::
+
+    Default parameters were modified for the example and included in the configuration file. If you use this configuration file as a general template for your data, do not forget to remove everything below the line "REMOVE BELOW THIS LINE IF YOU USE THIS CONFIGURATION FILE AS TEMPLATE FOR YOUR DATA".
+
+
+You can now create the genome indices and annotations. For this small example, it is important to downscale the STAR ``--genomeSAindexNbases`` parameter as follows
+
+
+.. code-block:: bash
+
+    docker run --volume `pwd`:/data quay.io/biocontainers/rpbp:3.0.1--py310h30d9df9_0 prepare-rpbp-genome /data/c-elegans-test.yaml --star-options "--genomeSAindexNbases 10" --num-cpus 4 --logging-level INFO --log-file /data/rpbp-genome.log
+
+
+.. code-block:: bash
+
+    singularity run --bind `pwd`:/data rpbp.sif prepare-rpbp-genome /data/c-elegans-test.yaml --star-options "--genomeSAindexNbases 10" --num-cpus 4 --logging-level INFO --log-file /data/rpbp-genome.log
+
+
+The file *rpbp-genome.log* contains logging output for the reference preprocessing. You now have a new directory called *WBcel235.79.chrI* with genome indices and annotations.
+
+Finally, run the ORF discovery pipeline
+
+
+.. code-block:: bash
+
+    docker run --volume `pwd`:/data quay.io/biocontainers/rpbp:3.0.1--py310h30d9df9_0 run-all-rpbp-instances /data/c-elegans-test.yaml --merge-replicates --run-replicates --keep-intermediate-files --num-cpus 4 --logging-level INFO --log-file /data/rpbp-pipeline.log
+
+
+.. code-block:: bash
+
+    singularity run --bind `pwd`:/data rpbp.sif run-all-rpbp-instances /data/c-elegans-test.yaml --merge-replicates --run-replicates --keep-intermediate-files --num-cpus 4 --logging-level INFO --log-file /data/rpbp-pipeline.log
+
+
+The file *rpbp-pipeline.log* contains logging output for the different processing steps. You now have four new directories (*with-*, *without-*) including output from Flexbar, Bowtie2, and STAR, and directories with **Rp-Bp** output: *metagene-profiles*, *orf-profiles*, and *orf-predictions*. The *orf-predictions* include the output for each sample *c-elegans-rep-1* and *c-elegans-rep-2* as well as for the merged replicates *c-elegans-test*.
+
+How to summarize the results and launch the apps
+------------------------------------------------
+
+Prepare the summary output for the *profile construction dashboard*
+
+.. code-block:: bash
+
+    docker run --volume `pwd`:/data quay.io/biocontainers/rpbp:3.0.1--py310h30d9df9_0 summarize-rpbp-profile-construction /data/c-elegans-test.yaml --num-cpus 4 --logging-level INFO --log-file /data/rpbp-profile-summary.log
+
+
+.. code-block:: bash
+
+    singularity run --bind `pwd`:/data rpbp.sif summarize-rpbp-profile-construction /data/c-elegans-test.yaml --num-cpus 4 --logging-level INFO --log-file /data/rpbp-profile-summary.log
+
+
+and for the *predictions dashboard*
+
+.. code-block:: bash
+
+    docker run --volume `pwd`:/data quay.io/biocontainers/rpbp:3.0.1--py310h30d9df9_0 summarize-rpbp-predictions /data/c-elegans-test.yaml --no-replicates --circos-bin-width 10000 --circos-show-chroms I --logging-level INFO --log-file /data/rpbp-predictions-summary.log
+
+.. code-block:: bash
+
+    singularity run --bind `pwd`:/data rpbp.sif summarize-rpbp-predictions /data/c-elegans-test.yaml --no-replicates --circos-bin-width 10000 --circos-show-chroms I --logging-level INFO --log-file /data/rpbp-predictions-summary.log
+
+
+Due to the size of the data, we reduce the bin width for the `Circos <http://circos.ca/>`_ plot. We also need to specify which sequences or chromosomes we want to include (by default, only numbered chromosomes and X/x, Y/y are shown). You now have a new directory *analysis* with *profile_construction* and *rpbp_predictions* output.
+
+Launch any of the web applications with
+
+.. code-block:: bash
+
+    docker run -p 8050:8050 --volume `pwd`:/data quay.io/biocontainers/rpbp:3.0.1--py310h30d9df9_0 rpbp-profile-construction-dashboard -c /data/c-elegans-test.yaml --host="0.0.0.0"
+
+.. code-block:: bash
+
+    singularity run --bind `pwd`:/data rpbp.sif rpbp-profile-construction-dashboard -c /data/c-elegans-test.yaml --host="0.0.0.0"
+
+or
+
+.. code-block:: bash
+
+    docker run -p 8050:8050 --volume `pwd`:/data quay.io/biocontainers/rpbp:3.0.1--py310h30d9df9_0 rpbp-predictions-dashboard -c /data/c-elegans-test.yaml --host="0.0.0.0"
+
+.. code-block:: bash
+
+    singularity run --bind `pwd`:/data rpbp.sif rpbp-predictions-dashboard -c /data/c-elegans-test.yaml --host="0.0.0.0"
+
+
+You then have to open a browser page at the correct address, *e.g.* you see ``Running on http://127.0.0.1:8050``, click on this link, or open a browser page at this address. To navigate the apps is easy, just follow the "hints". Most items are interactive. Press ``CTRL+C`` to quit.
+
+.. attention::
+
+    For the apps only, the configuration file is passed using a (required) named argument ``-c/--config CONFIG``.