Add functionality and documentation for building Conda pacakges.

- Add documentation that links out to external sources for building Conda recipes - recommending specific docs and a specific order. - Allow Planemo to work with locally built Conda packages using the ``--conda_use_local`` command. - Add an exercise to the documentation that requires building a package and the use of ``--conda_use_local``. This uses a new project template that adds a tool targeting the package https://github.com/jmchilton/fleeqtk I forked from seqtk for this purpose. - Update galaxy-lib dependency for the required library functionality to use ``--conda_use_local``. - Add test case that follows the example in the documentation targeting the fleeqtk package. It builds a fleetqk recipe, uses conda_install --conda_use_local on a tool that requires it, and finally runs planemo test to ensure that Galaxy properly uses the resulting package.
galaxyproject · Feb 27, 2017 · 24764cd · 24764cd
1 parent e2454ef
commit 24764cd
Show file tree

Hide file tree

Showing 23 changed files with 411 additions and 15 deletions.
diff --git a/docs/_writing_dependencies_conda.rst b/docs/_writing_dependencies_conda.rst
@@ -267,12 +267,13 @@ Use the ``project_init`` command to download this exercise.
 
 ::
 
-    $ planemo project_init --template conda_exercise conda_exercise
-    $ cd conda_exercise
+    $ planemo project_init --template conda_exercises conda_exercises
+    $ cd conda_exercises/exercise1
     $ ls 
     pear.xml              test-data
 
-This will download a tool for `PEAR - Paired-End reAd mergeR
+This project template contains a few exercises. The first uses an adapted
+version of an IUC tool for `PEAR - Paired-End reAd mergeR
 <http://sco.h-its.org/exelixis/web/software/pear/>`__. This tool however has
 no ``requirement`` tags and so will not work properly.
 
@@ -301,18 +302,79 @@ general Bioconda_ is usually the correct starting point.
     been accepted into Bioconda_ - including tools for image analysis,
     natural language processing, and cheminformatics.
 
-At this time, the most relevant source for information on building Conda packages for Galaxy
-is probably the Bioconda_ documentation - in particular check out the `contributing documentation
-<https://bioconda.github.io/contributing.html>`__.
+To get quickly learn to write Conda_ recipes for typical Galaxy tools,
+please read the following pieces of external documentation.
+
+- `Contributing to Bioconda <https://bioconda.github.io/contributing.html>`__ in particular focusing on
+
+  - `One time setup <https://bioconda.github.io/contrib-setup.html>`__
+  - `Contributing a recipe <https://bioconda.github.io/contribute-a-recipe.html>`__ (through "Write a Recipe")
+- `Building conda packages <https://conda.io/docs/building/bpp.html#>`__ in particular
+
+  - `Building conda packages with conda skeleton <https://conda.io/docs/build_tutorials/pkgs.html>`__ (the best approach for common scripting languages such as R and Python)
+  - `Building conda packages from scratch <https://conda.io/docs/build_tutorials/pkgs2.html>`__
+  - `Building conda packages for general code projects <https://conda.io/docs/build_tutorials/postgis.html>`__
+  - `Using conda build <https://conda.io/docs/building/recipe.html>`__
+- Then return to the Bioconda documentation and read
+
+  - The rest of "Contributing a recipe" continuing from `Testing locally <https://bioconda.github.io/contribute-a-recipe.html#test-locally>`__
+  - And finally `Guidelines for bioconda recipes <https://bioconda.github.io/guidelines.html>`__
+
+These guidelines in particular can be skimmed depending on your recipe type, for
+instance that document provides specific advice for:
+
+- `Python <https://bioconda.github.io/guidelines.html#python>`__
+- `R (CRAN) <https://bioconda.github.io/guidelines.html#r-cran>`__
+- `R (Bioconductor) <https://bioconda.github.io/guidelines.html#r-bioconductor>`__
+- `Perl <https://bioconda.github.io/guidelines.html#perl>`__
+- `C/C++ <https://bioconda.github.io/guidelines.html#c-c>`__
+
+To go a little deeper, you may want to read: 
+
+- `Specification for meta.yaml <https://conda.io/docs/building/meta-yaml.html>`__
+- `Environment variables <https://conda.io/docs/building/environment-vars.html>`__
+- `Custom channels <https://conda.io/docs/custom-channels.html>`__
+
+And finally to debug problems the `Bioconda troubleshooting <https://bioconda.github.io/troubleshooting.html>`__
+documentation may prove useful.
 
 ----------------------------------------------------------------
 Exercise - Build a Recipe
 ----------------------------------------------------------------
 
-1. Build a recipe for Bioconda.
-2. Open a pull request to contribute it.
+If you have just completed the exercise above - this exercise can be found in parent folder. Get
+there with ``cd ../exercise2``. If not, the exercise can be downloaded with
 
+::
 
+    $ planemo project_init --template conda_exercises conda_exercises
+    $ cd conda_exercises/exercise2
+    $ ls 
+    fleeqtk_seq.xml              test-data
+
+This is the skeleton of a tool wrapping the parody bioinformatics software package fleeqtk_.
+fleeqtk is a fork of the project seqtk that many Planemo tutorials are built around and the
+example tool ``fleeqtk_seq.xml`` should be fairly familiar. fleeqtk version 1.3 can be downloaded
+from `here <https://github.com/jmchilton/fleeqtk/archive/v1.3.tar.gz>`__ and built using
+``make``. The result of ``make`` includes a single executable ``fleeqtk``.
+
+1. Clone and branch Bioconda_.
+2. Build a recipe for fleeqtk version 1.3. You may wish to use ``conda skeleton``, start from
+   scratch, or copy the recipe of seqtk and work from there - any of these strategies should work.  
+3. Use ``conda build`` or Bioconda tooling to build the recipe.
+4. Run ``planemo conda_install --conda_use_local fleeqtk_seq.xml`` to verify the resulting package
+   can be built into a Galaxy environment.
+5. Run ``planemo test fleeqtk_seq.xml`` to verify the resulting package works as expected.
+
+.. note: The planemo flag ``--conda_use_local`` causes planemo and Galaxy to use locally built
+     packages during dependency resolution and related commands.
+
+Congratulations on writing a Conda recipe and building a package.  Upon succesfully building
+and testing such a Bioconda package, you would normally push your branch to Github
+and open a pull request. This step is skipped here as to not pollute Bioconda with unneeded
+software packages.
+
+.. _fleeqtk: https://github.com/jmchilton/fleeqtk
 .. _Bioconda: https://github.com/bioconda/bioconda-recipes
 .. _Conda: https://conda.io/docs/
 .. _Anaconda: https://anaconda.org/
diff --git a/planemo/commands/cmd_conda_build.py b/planemo/commands/cmd_conda_build.py
@@ -0,0 +1,21 @@
+"""Module describing the planemo ``conda_build`` command."""
+from __future__ import print_function
+
+import click
+
+from planemo import options
+from planemo.cli import command_function
+from planemo.conda import build_conda_context
+
+
+@click.command('conda_build')
+@options.conda_target_options(include_local=False)  # No reason to expose local, we have to use it.
+@options.recipe_arg(multiple=True)
+@command_function
+def cli(ctx, paths, **kwds):
+    """Perform conda build with Planemo's conda."""
+    # Force conda_use_local for building...
+    kwds["conda_use_local"] = True
+    conda_context = build_conda_context(ctx, handle_auto_init=True, **kwds)
+    build_args = list(paths)
+    conda_context.exec_command("build", build_args)
diff --git a/planemo/commands/cmd_conda_init.py b/planemo/commands/cmd_conda_init.py
@@ -16,7 +16,7 @@
 
 
 @click.command('conda_init')
-@options.conda_target_options()
+@options.conda_target_options(include_local=False)  # Always use local during init.
 @command_function
 def cli(ctx, **kwds):
     """Download and install conda.
@@ -36,7 +36,7 @@ def cli(ctx, **kwds):
         warn(MESSAGE_ERROR_ALREADY_EXISTS % conda_context.conda_exec)
         exit = EXIT_CODE_ALREADY_EXISTS
     else:
-        exit = conda_util.install_conda(conda_context=conda_context)
+        exit = conda_util.install_conda(conda_context=conda_context, force_conda_build=True)
         if exit:
             warn(MESSAGE_ERROR_FAILED % conda_context.conda_exec)
         else:

diff --git a/planemo/commands/cmd_conda_search.py b/planemo/commands/cmd_conda_search.py
@@ -9,7 +9,7 @@
 
 
 @click.command('conda_search')
-@options.conda_target_options()
+@options.conda_target_options(include_local=False)
 @click.argument(
     "term",
     metavar="TERM",
@@ -22,5 +22,5 @@ def cli(ctx, term, **kwds):
 
     Implicitly adds channels Planemo is configured with.
     """
-    conda_context = build_conda_context(ctx, use_planemo_shell_exec=False, **kwds)
+    conda_context = build_conda_context(ctx, handle_auto_init=True, **kwds)
     conda_context.exec_command("search", [term])
diff --git a/planemo/conda.py b/planemo/conda.py
@@ -28,10 +28,12 @@ def build_conda_context(ctx, **kwds):
     use_planemo_shell = kwds.get("use_planemo_shell_exec", True)
     ensure_channels = kwds.get("conda_ensure_channels", "")
     condarc_override = kwds.get("condarc", condarc_override_default)
+    use_local = kwds.get("conda_use_local", False)
     shell_exec = shell if use_planemo_shell else None
     conda_context = conda_util.CondaContext(conda_prefix=conda_prefix,
                                             ensure_channels=ensure_channels,
                                             condarc_override=condarc_override,
+                                            use_local=use_local,
                                             shell_exec=shell_exec)
     handle_auto_init = kwds.get("handle_auto_init", False)
     if handle_auto_init and not conda_context.is_installed():
@@ -50,6 +52,8 @@ def build_conda_context(ctx, **kwds):
 
         if failed:
             raise ExitCodeException(EXIT_CODE_FAILED_DEPENDENCIES)
+    if handle_auto_init:
+        conda_context.ensure_conda_build_installed_if_needed()
     return conda_context
 
 

diff --git a/planemo/options.py b/planemo/options.py
@@ -446,6 +446,14 @@ def conda_debug_option():
     )
 
 
+def conda_use_local_option():
+    return planemo_option(
+        "--conda_use_local",
+        is_flag=True,
+        help="Use locally built packages while building Conda environments."
+    )
+
+
 def conda_ensure_channels_option():
     return planemo_option(
         "conda_ensure_channels",
@@ -874,12 +882,13 @@ def shed_target_options():
     )
 
 
-def conda_target_options():
+def conda_target_options(include_local=True):
     return _compose(
         conda_prefix_option(),
         conda_exec_option(),
         conda_debug_option(),
         conda_ensure_channels_option(),
+        conda_use_local_option(),
     )
 
 

diff --git a/...lates/conda_exercise/exercise_1/.shed.yml → ...ates/conda_exercises/exercise_1/.shed.yml b/...lates/conda_exercise/exercise_1/.shed.yml → ...ates/conda_exercises/exercise_1/.shed.yml
diff --git a/...plates/conda_exercise/exercise_1/pear.xml → ...lates/conda_exercises/exercise_1/pear.xml b/...plates/conda_exercise/exercise_1/pear.xml → ...lates/conda_exercises/exercise_1/pear.xml
diff --git a/...ercise/exercise_1/test-data/forward.fastq → ...rcises/exercise_1/test-data/forward.fastq b/...ercise/exercise_1/test-data/forward.fastq → ...rcises/exercise_1/test-data/forward.fastq
diff --git a/...1/test-data/pear_assembled_results1.fastq → ...1/test-data/pear_assembled_results1.fastq b/...1/test-data/pear_assembled_results1.fastq → ...1/test-data/pear_assembled_results1.fastq
diff --git a/...a/pear_unassembled_forward_results1.fastq → ...a/pear_unassembled_forward_results1.fastq b/...a/pear_unassembled_forward_results1.fastq → ...a/pear_unassembled_forward_results1.fastq
diff --git a/...ercise/exercise_1/test-data/reverse.fastq → ...rcises/exercise_1/test-data/reverse.fastq b/...ercise/exercise_1/test-data/reverse.fastq → ...rcises/exercise_1/test-data/reverse.fastq
diff --git a/...exercise/exercise_1/tool_dependencies.xml → ...xercises/exercise_1/tool_dependencies.xml b/...exercise/exercise_1/tool_dependencies.xml → ...xercises/exercise_1/tool_dependencies.xml
diff --git a/project_templates/conda_exercises/exercise_2/fleeqtk_seq.xml b/project_templates/conda_exercises/exercise_2/fleeqtk_seq.xml
@@ -0,0 +1,99 @@
+<tool id="fleeqtk_seq" name="Convert to FASTA (fleeqtk)" version="0.1.0">
+    <stdio>
+        <exit_code range="1:" />
+    </stdio>
+    <command><![CDATA[
+        fleeqtk seq
+              #if $settings.advanced == "advanced"
+              $settings.shift_quality
+              -q $settings.quality_min
+              -X $settings.quality_max
+              #if $settings.mask_regions
+              -M '$settings.mask_regions'
+              #end if
+              #if $settings.sample.sample
+              -f $settings.sample.fraction
+              -s $settings.sample.seed
+              #end if
+              #end if
+              -a '$input1' > '$output1'
+    ]]></command>
+    <inputs>
+        <param type="data" name="input1" format="fastq" />
+        <conditional name="settings">
+            <param name="advanced" type="select" label="Specify advanced parameters">
+                <option value="simple" selected="true">No, use program defaults.</option>
+                <option value="advanced">Yes, see full parameter list.</option>
+            </param>
+            <when value="simple">
+            </when>
+            <when value="advanced">
+                <param name="shift_quality" type="boolean" label="Shift quality" 
+                       truevalue="-V" falsevalue=""
+                       help="shift quality by '(-Q) - 33' (-V)" />
+                <param name="quality_min" type="integer" label="Mask bases with quality lower than" 
+                       value="0" min="0" max="255" help="(-q)" />
+                <param name="quality_max" type="integer" label="Mask bases with quality higher than" 
+                       value="255" min="0" max="255" help="(-X)" />
+                <param name="mask_regions" type="data" label="Mask regions in BED" 
+                       format="bed" help="(-M)" optional="true" />
+                <conditional name="sample">
+                    <param name="sample" type="boolean" label="Sample fraction of sequences" />
+                    <when value="true">
+                        <param name="fraction" label="Fraction" type="float" value="1.0"
+                               help="(-f)" />
+                        <param name="seed" label="Random seed" type="integer" value="11"
+                               help="(-s)" />
+                    </when>
+                    <when value="false">
+                    </when>
+                </conditional>
+            </when>
+        </conditional>
+    </inputs>
+    <outputs>
+        <data name="output1" format="fasta" />
+    </outputs>
+    <tests>
+        <test>
+            <param name="input1" value="2.fastq"/>
+            <output name="output1" file="2.fasta"/>
+        </test>
+    </tests>
+    <help><![CDATA[
+        
+Usage:   fleeqtk seq [options] <in.fq>|<in.fa>
+
+Options: -q INT    mask bases with quality lower than INT [0]
+         -X INT    mask bases with quality higher than INT [255]
+         -n CHAR   masked bases converted to CHAR; 0 for lowercase [0]
+         -l INT    number of residues per line; 0 for 2^32-1 [0]
+         -Q INT    quality shift: ASCII-INT gives base quality [33]
+         -s INT    random seed (effective with -f) [11]
+         -f FLOAT  sample FLOAT fraction of sequences [1]
+         -M FILE   mask regions in BED or name list FILE [null]
+         -L INT    drop sequences with length shorter than INT [0]
+         -c        mask complement region (effective with -M)
+         -r        reverse complement
+         -A        force FASTA output (discard quality)
+         -C        drop comments at the header lines
+         -N        drop sequences containing ambiguous bases
+         -1        output the 2n-1 reads only
+         -2        output the 2n reads only
+         -V        shift quality by '(-Q) - 33'
+         -U        convert all bases to uppercases
+
+
+    ]]></help>
+    <citations>
+        <citation type="bibtex">
+@misc{githubfleeqtk,
+  author = {LastTODO, FirstTODO},
+  year = {TODO},
+  title = {fleeqtk},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  url = {https://github.com/jmchilton/fleeqtk},
+}</citation>
+    </citations>
+</tool>
diff --git a/project_templates/conda_exercises/exercise_2/test-data/2.fasta b/project_templates/conda_exercises/exercise_2/test-data/2.fasta
@@ -0,0 +1,6 @@
+>EAS54_6_R1_2_1_413_324
+CCCTTCTTGTCTTCAGCGTTTCTCC
+>EAS54_6_R1_2_1_540_792
+TTGGCAGGCCAAGGCCGATGGATCA
+>EAS54_6_R1_2_1_443_348
+GTTGCTTCTGGCGTGGGTGGGGGGG
diff --git a/project_templates/conda_exercises/exercise_2/test-data/2.fastq b/project_templates/conda_exercises/exercise_2/test-data/2.fastq
@@ -0,0 +1,12 @@
+@EAS54_6_R1_2_1_413_324
+CCCTTCTTGTCTTCAGCGTTTCTCC
++
+;;3;;;;;;;;;;;;7;;;;;;;88
+@EAS54_6_R1_2_1_540_792
+TTGGCAGGCCAAGGCCGATGGATCA
++
+;;;;;;;;;;;7;;;;;-;;;3;83
+@EAS54_6_R1_2_1_443_348
+GTTGCTTCTGGCGTGGGTGGGGGGG
++EAS54_6_R1_2_1_443_348
+;;;;;;;;;;;9;7;;.7;393333
diff --git a/requirements.txt b/requirements.txt
@@ -13,6 +13,6 @@ virtualenv
 lxml
 gxformat2>=0.1.1
 ephemeris>=0.2.0
-galaxy-lib>=17.5.2
+galaxy-lib>=17.5.5
 html5lib>=0.9999999,!=0.99999999,!=0.999999999,!=1.0b10,!=1.0b09 ; python_version == '2.7'
 cwltool==1.0.20160726135535 ; python_version == '2.7'
diff --git a/tests/data/recipes/fleeqtk/build.sh b/tests/data/recipes/fleeqtk/build.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+export C_INCLUDE_PATH=${PREFIX}/include
+export LIBRARY_PATH=${PREFIX}/lib
+
+make all
+mkdir -p $PREFIX/bin
+cp -f fleeqtk $PREFIX/bin/
diff --git a/tests/data/recipes/fleeqtk/meta.yaml b/tests/data/recipes/fleeqtk/meta.yaml
@@ -0,0 +1,29 @@
+package:
+  name: fleeqtk
+  version: 1.3
+
+source:
+  fn: v1.3.tar.gz
+  url: https://github.com/jmchilton/fleeqtk/archive/v1.3.tar.gz
+  md5: 015daf2e21789519eea4e156fc155272
+
+build:
+  number: 0
+
+requirements:
+  build:
+    - gcc   # [not osx]
+    - llvm  # [osx]
+    - zlib
+  run:
+    - zlib
+
+
+about:
+  home: https://github.com/jmchilton/fleeqtk
+  license: MIT License
+  summary: fleeqtk - those sequences are on fleeqtk
+
+test:
+  commands:
+    - fleeqtk seq