Skip to content

Commit

Permalink
Merge pull request #643 from jmchilton/conda_build
Browse files Browse the repository at this point in the history
Add functionality and documentation for building Conda pacakges.
  • Loading branch information
bgruening authored Feb 27, 2017
2 parents e2454ef + 24764cd commit 96782cd
Show file tree
Hide file tree
Showing 23 changed files with 411 additions and 15 deletions.
78 changes: 70 additions & 8 deletions docs/_writing_dependencies_conda.rst
Original file line number Diff line number Diff line change
Expand Up @@ -267,12 +267,13 @@ Use the ``project_init`` command to download this exercise.

::

$ planemo project_init --template conda_exercise conda_exercise
$ cd conda_exercise
$ planemo project_init --template conda_exercises conda_exercises
$ cd conda_exercises/exercise1
$ ls
pear.xml test-data

This will download a tool for `PEAR - Paired-End reAd mergeR
This project template contains a few exercises. The first uses an adapted
version of an IUC tool for `PEAR - Paired-End reAd mergeR
<http://sco.h-its.org/exelixis/web/software/pear/>`__. This tool however has
no ``requirement`` tags and so will not work properly.

Expand Down Expand Up @@ -301,18 +302,79 @@ general Bioconda_ is usually the correct starting point.
been accepted into Bioconda_ - including tools for image analysis,
natural language processing, and cheminformatics.

At this time, the most relevant source for information on building Conda packages for Galaxy
is probably the Bioconda_ documentation - in particular check out the `contributing documentation
<https://bioconda.github.io/contributing.html>`__.
To get quickly learn to write Conda_ recipes for typical Galaxy tools,
please read the following pieces of external documentation.

- `Contributing to Bioconda <https://bioconda.github.io/contributing.html>`__ in particular focusing on

- `One time setup <https://bioconda.github.io/contrib-setup.html>`__
- `Contributing a recipe <https://bioconda.github.io/contribute-a-recipe.html>`__ (through "Write a Recipe")
- `Building conda packages <https://conda.io/docs/building/bpp.html#>`__ in particular

- `Building conda packages with conda skeleton <https://conda.io/docs/build_tutorials/pkgs.html>`__ (the best approach for common scripting languages such as R and Python)
- `Building conda packages from scratch <https://conda.io/docs/build_tutorials/pkgs2.html>`__
- `Building conda packages for general code projects <https://conda.io/docs/build_tutorials/postgis.html>`__
- `Using conda build <https://conda.io/docs/building/recipe.html>`__
- Then return to the Bioconda documentation and read

- The rest of "Contributing a recipe" continuing from `Testing locally <https://bioconda.github.io/contribute-a-recipe.html#test-locally>`__
- And finally `Guidelines for bioconda recipes <https://bioconda.github.io/guidelines.html>`__

These guidelines in particular can be skimmed depending on your recipe type, for
instance that document provides specific advice for:

- `Python <https://bioconda.github.io/guidelines.html#python>`__
- `R (CRAN) <https://bioconda.github.io/guidelines.html#r-cran>`__
- `R (Bioconductor) <https://bioconda.github.io/guidelines.html#r-bioconductor>`__
- `Perl <https://bioconda.github.io/guidelines.html#perl>`__
- `C/C++ <https://bioconda.github.io/guidelines.html#c-c>`__

To go a little deeper, you may want to read:

- `Specification for meta.yaml <https://conda.io/docs/building/meta-yaml.html>`__
- `Environment variables <https://conda.io/docs/building/environment-vars.html>`__
- `Custom channels <https://conda.io/docs/custom-channels.html>`__

And finally to debug problems the `Bioconda troubleshooting <https://bioconda.github.io/troubleshooting.html>`__
documentation may prove useful.

----------------------------------------------------------------
Exercise - Build a Recipe
----------------------------------------------------------------

1. Build a recipe for Bioconda.
2. Open a pull request to contribute it.
If you have just completed the exercise above - this exercise can be found in parent folder. Get
there with ``cd ../exercise2``. If not, the exercise can be downloaded with

::

$ planemo project_init --template conda_exercises conda_exercises
$ cd conda_exercises/exercise2
$ ls
fleeqtk_seq.xml test-data

This is the skeleton of a tool wrapping the parody bioinformatics software package fleeqtk_.
fleeqtk is a fork of the project seqtk that many Planemo tutorials are built around and the
example tool ``fleeqtk_seq.xml`` should be fairly familiar. fleeqtk version 1.3 can be downloaded
from `here <https://github.com/jmchilton/fleeqtk/archive/v1.3.tar.gz>`__ and built using
``make``. The result of ``make`` includes a single executable ``fleeqtk``.

1. Clone and branch Bioconda_.
2. Build a recipe for fleeqtk version 1.3. You may wish to use ``conda skeleton``, start from
scratch, or copy the recipe of seqtk and work from there - any of these strategies should work.
3. Use ``conda build`` or Bioconda tooling to build the recipe.
4. Run ``planemo conda_install --conda_use_local fleeqtk_seq.xml`` to verify the resulting package
can be built into a Galaxy environment.
5. Run ``planemo test fleeqtk_seq.xml`` to verify the resulting package works as expected.

.. note: The planemo flag ``--conda_use_local`` causes planemo and Galaxy to use locally built
packages during dependency resolution and related commands.
Congratulations on writing a Conda recipe and building a package. Upon succesfully building
and testing such a Bioconda package, you would normally push your branch to Github
and open a pull request. This step is skipped here as to not pollute Bioconda with unneeded
software packages.

.. _fleeqtk: https://github.com/jmchilton/fleeqtk
.. _Bioconda: https://github.com/bioconda/bioconda-recipes
.. _Conda: https://conda.io/docs/
.. _Anaconda: https://anaconda.org/
21 changes: 21 additions & 0 deletions planemo/commands/cmd_conda_build.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
"""Module describing the planemo ``conda_build`` command."""
from __future__ import print_function

import click

from planemo import options
from planemo.cli import command_function
from planemo.conda import build_conda_context


@click.command('conda_build')
@options.conda_target_options(include_local=False) # No reason to expose local, we have to use it.
@options.recipe_arg(multiple=True)
@command_function
def cli(ctx, paths, **kwds):
"""Perform conda build with Planemo's conda."""
# Force conda_use_local for building...
kwds["conda_use_local"] = True
conda_context = build_conda_context(ctx, handle_auto_init=True, **kwds)
build_args = list(paths)
conda_context.exec_command("build", build_args)
4 changes: 2 additions & 2 deletions planemo/commands/cmd_conda_init.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@


@click.command('conda_init')
@options.conda_target_options()
@options.conda_target_options(include_local=False) # Always use local during init.
@command_function
def cli(ctx, **kwds):
"""Download and install conda.
Expand All @@ -36,7 +36,7 @@ def cli(ctx, **kwds):
warn(MESSAGE_ERROR_ALREADY_EXISTS % conda_context.conda_exec)
exit = EXIT_CODE_ALREADY_EXISTS
else:
exit = conda_util.install_conda(conda_context=conda_context)
exit = conda_util.install_conda(conda_context=conda_context, force_conda_build=True)
if exit:
warn(MESSAGE_ERROR_FAILED % conda_context.conda_exec)
else:
Expand Down
4 changes: 2 additions & 2 deletions planemo/commands/cmd_conda_search.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@


@click.command('conda_search')
@options.conda_target_options()
@options.conda_target_options(include_local=False)
@click.argument(
"term",
metavar="TERM",
Expand All @@ -22,5 +22,5 @@ def cli(ctx, term, **kwds):
Implicitly adds channels Planemo is configured with.
"""
conda_context = build_conda_context(ctx, use_planemo_shell_exec=False, **kwds)
conda_context = build_conda_context(ctx, handle_auto_init=True, **kwds)
conda_context.exec_command("search", [term])
4 changes: 4 additions & 0 deletions planemo/conda.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,12 @@ def build_conda_context(ctx, **kwds):
use_planemo_shell = kwds.get("use_planemo_shell_exec", True)
ensure_channels = kwds.get("conda_ensure_channels", "")
condarc_override = kwds.get("condarc", condarc_override_default)
use_local = kwds.get("conda_use_local", False)
shell_exec = shell if use_planemo_shell else None
conda_context = conda_util.CondaContext(conda_prefix=conda_prefix,
ensure_channels=ensure_channels,
condarc_override=condarc_override,
use_local=use_local,
shell_exec=shell_exec)
handle_auto_init = kwds.get("handle_auto_init", False)
if handle_auto_init and not conda_context.is_installed():
Expand All @@ -50,6 +52,8 @@ def build_conda_context(ctx, **kwds):

if failed:
raise ExitCodeException(EXIT_CODE_FAILED_DEPENDENCIES)
if handle_auto_init:
conda_context.ensure_conda_build_installed_if_needed()
return conda_context


Expand Down
11 changes: 10 additions & 1 deletion planemo/options.py
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,14 @@ def conda_debug_option():
)


def conda_use_local_option():
return planemo_option(
"--conda_use_local",
is_flag=True,
help="Use locally built packages while building Conda environments."
)


def conda_ensure_channels_option():
return planemo_option(
"conda_ensure_channels",
Expand Down Expand Up @@ -874,12 +882,13 @@ def shed_target_options():
)


def conda_target_options():
def conda_target_options(include_local=True):
return _compose(
conda_prefix_option(),
conda_exec_option(),
conda_debug_option(),
conda_ensure_channels_option(),
conda_use_local_option(),
)


Expand Down
99 changes: 99 additions & 0 deletions project_templates/conda_exercises/exercise_2/fleeqtk_seq.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
<tool id="fleeqtk_seq" name="Convert to FASTA (fleeqtk)" version="0.1.0">
<stdio>
<exit_code range="1:" />
</stdio>
<command><![CDATA[
fleeqtk seq
#if $settings.advanced == "advanced"
$settings.shift_quality
-q $settings.quality_min
-X $settings.quality_max
#if $settings.mask_regions
-M '$settings.mask_regions'
#end if
#if $settings.sample.sample
-f $settings.sample.fraction
-s $settings.sample.seed
#end if
#end if
-a '$input1' > '$output1'
]]></command>
<inputs>
<param type="data" name="input1" format="fastq" />
<conditional name="settings">
<param name="advanced" type="select" label="Specify advanced parameters">
<option value="simple" selected="true">No, use program defaults.</option>
<option value="advanced">Yes, see full parameter list.</option>
</param>
<when value="simple">
</when>
<when value="advanced">
<param name="shift_quality" type="boolean" label="Shift quality"
truevalue="-V" falsevalue=""
help="shift quality by '(-Q) - 33' (-V)" />
<param name="quality_min" type="integer" label="Mask bases with quality lower than"
value="0" min="0" max="255" help="(-q)" />
<param name="quality_max" type="integer" label="Mask bases with quality higher than"
value="255" min="0" max="255" help="(-X)" />
<param name="mask_regions" type="data" label="Mask regions in BED"
format="bed" help="(-M)" optional="true" />
<conditional name="sample">
<param name="sample" type="boolean" label="Sample fraction of sequences" />
<when value="true">
<param name="fraction" label="Fraction" type="float" value="1.0"
help="(-f)" />
<param name="seed" label="Random seed" type="integer" value="11"
help="(-s)" />
</when>
<when value="false">
</when>
</conditional>
</when>
</conditional>
</inputs>
<outputs>
<data name="output1" format="fasta" />
</outputs>
<tests>
<test>
<param name="input1" value="2.fastq"/>
<output name="output1" file="2.fasta"/>
</test>
</tests>
<help><![CDATA[
Usage: fleeqtk seq [options] <in.fq>|<in.fa>
Options: -q INT mask bases with quality lower than INT [0]
-X INT mask bases with quality higher than INT [255]
-n CHAR masked bases converted to CHAR; 0 for lowercase [0]
-l INT number of residues per line; 0 for 2^32-1 [0]
-Q INT quality shift: ASCII-INT gives base quality [33]
-s INT random seed (effective with -f) [11]
-f FLOAT sample FLOAT fraction of sequences [1]
-M FILE mask regions in BED or name list FILE [null]
-L INT drop sequences with length shorter than INT [0]
-c mask complement region (effective with -M)
-r reverse complement
-A force FASTA output (discard quality)
-C drop comments at the header lines
-N drop sequences containing ambiguous bases
-1 output the 2n-1 reads only
-2 output the 2n reads only
-V shift quality by '(-Q) - 33'
-U convert all bases to uppercases
]]></help>
<citations>
<citation type="bibtex">
@misc{githubfleeqtk,
author = {LastTODO, FirstTODO},
year = {TODO},
title = {fleeqtk},
publisher = {GitHub},
journal = {GitHub repository},
url = {https://github.com/jmchilton/fleeqtk},
}</citation>
</citations>
</tool>
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
>EAS54_6_R1_2_1_413_324
CCCTTCTTGTCTTCAGCGTTTCTCC
>EAS54_6_R1_2_1_540_792
TTGGCAGGCCAAGGCCGATGGATCA
>EAS54_6_R1_2_1_443_348
GTTGCTTCTGGCGTGGGTGGGGGGG
12 changes: 12 additions & 0 deletions project_templates/conda_exercises/exercise_2/test-data/2.fastq
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
@EAS54_6_R1_2_1_413_324
CCCTTCTTGTCTTCAGCGTTTCTCC
+
;;3;;;;;;;;;;;;7;;;;;;;88
@EAS54_6_R1_2_1_540_792
TTGGCAGGCCAAGGCCGATGGATCA
+
;;;;;;;;;;;7;;;;;-;;;3;83
@EAS54_6_R1_2_1_443_348
GTTGCTTCTGGCGTGGGTGGGGGGG
+EAS54_6_R1_2_1_443_348
;;;;;;;;;;;9;7;;.7;393333
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,6 @@ virtualenv
lxml
gxformat2>=0.1.1
ephemeris>=0.2.0
galaxy-lib>=17.5.2
galaxy-lib>=17.5.5
html5lib>=0.9999999,!=0.99999999,!=0.999999999,!=1.0b10,!=1.0b09 ; python_version == '2.7'
cwltool==1.0.20160726135535 ; python_version == '2.7'
8 changes: 8 additions & 0 deletions tests/data/recipes/fleeqtk/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash

export C_INCLUDE_PATH=${PREFIX}/include
export LIBRARY_PATH=${PREFIX}/lib

make all
mkdir -p $PREFIX/bin
cp -f fleeqtk $PREFIX/bin/
29 changes: 29 additions & 0 deletions tests/data/recipes/fleeqtk/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package:
name: fleeqtk
version: 1.3

source:
fn: v1.3.tar.gz
url: https://github.com/jmchilton/fleeqtk/archive/v1.3.tar.gz
md5: 015daf2e21789519eea4e156fc155272

build:
number: 0

requirements:
build:
- gcc # [not osx]
- llvm # [osx]
- zlib
run:
- zlib


about:
home: https://github.com/jmchilton/fleeqtk
license: MIT License
summary: fleeqtk - those sequences are on fleeqtk

test:
commands:
- fleeqtk seq
Loading

0 comments on commit 96782cd

Please sign in to comment.