Initial pass at full documentation for Conda-based dependency development. #642

jmchilton · 2017-02-22T18:04:42Z

This pull request:

Add a new "Advanced" topic for developing tools with Conda dependencies including the following 4 main sections:
1. Using the planemo conda commands such conda_init, conda_install, conda_env, lint --conda_requirements, and test.
2. Finding existing Conda packages - using the Anaconda site, conda search, or a new planemo conda_search command for searching best practice channels.
3. A formal exercise based on the tools-iuc pear.xml tool - verify the tool is broken without a requirement tag, find the correct package name and version, lint the tool to verify the requirement is found, and run a test to verify it is fixed.
4. A short stub of a section containing resources on building and contributing Conda recipes.
Adds a project template for downloading the completed intro seqtk_seq.xml example for testing out planemo conda commands.
Adds a project template for a Conda exercise based on the pear tool from tools-iuc.
Adds a planemo conda_search command for searching best practice channels from the command line.

I think this can serve as the template and example for the first third of the "Conda and Containers - A Developer Perspective" workshop at the GCC 2017.

martenson · 2017-02-22T18:11:27Z

👏

martenson · 2017-02-22T18:12:42Z

docs/_writing_dependencies_conda.rst

+Exercise - Package a Tool
+----------------------------------------------------------------
+
+1. Package a tool for Bioconda.


Is this something you want to have captured here or just link out to bioconda tutorials?

I know I would get push back if I put it here - so this is probably not going to end up in the planemo docs. 😢 If it does - it might be a smaller section on galaxy-specific or planemo-specific tricks that already assumes the reader has gone through bioconda docs.

Do you want it here?

Perhaps - but that is besides the point. I'm sure the community would be better served by working on the bioconda documentation and tooling.

Or do you mean this particular line - not the whole idea of documentation for building recipes? Sorry I'm being confusing.

I am confused indeed.

I changed the language here - I was using tool incorrectly.

…ment. - Add a new advanced topic for developing tools with Conda dependencies with 4 main sections: 1. Using the planemo conda commands such conda_init, conda_install, conda_env, lint --conda_requirements, and test. 2. Finding existing Conda packages - using the Anaconda site, conda search, or a new "planemo conda_search" command for searching best practice channels. 3. A formal exercise based on the tools-iuc pear.xml tool. 4. A short stub of a section containing resources on building and contributing Conda recipes. - Add a project template for downloading the completed intro seqtk_seq.xml example for testing out planemo conda commands. - Add a project template for a Conda exercise based on the pear tool from tools-iuc. - Add a ``planemo conda_search`` command for searching best practice channels from the command line. I think this can serve as the template and example for the first third of the "Conda and Containers - A Developer Perspective" workshop at the GCC 2017.

mvdbeek

That reads really well. Thanks a lot @jmchilton!

mvdbeek · 2017-02-23T14:59:31Z

docs/_writing_dependencies_conda.rst

+    Linting tool /Users/john/workspace/planemo/docs/writing/seqtk_seq_v6.xml
+      ...
+    Applying linter requirements_in_conda... CHECK
+    .. INFO: Requirement [seqtk@1.2] matches target in best practice Conda channel [bioconda].


Could we drop the "best-practice" ? I guess this would be helpful also if you have uploaded things to your own channel.

lint --conda_requirements checks the "best practice" channels - i.e. will these resolve with the default configuration of Galaxy. It does not check against channels you configure to target with say conda_install or test.

I guess we should make a separate issue if you want to discuss this more - I feel like a conda_check command or something would be reasonable - but it isn't what I was trying to capture with lint. Does that make sense?

Yeah, for the lint command it does make sense to limit the channels.
In any case you'd notice something is wrong once you do planemo conda_install, so I would say that a planemo conda_check is not of high priority.

mvdbeek · 2017-02-23T15:00:49Z

docs/_writing_dependencies_conda.rst

+
+::
+
+    $ . <(planemo conda_env seqtk_seq.xml)


mvdbeek · 2017-02-23T15:02:40Z

docs/_writing_dependencies_conda.rst

+    (seqtk_seq) $ conda_env_deactivate
+    $
+
+As shown above the ``conda_env_deactivate`` will be created in this environment and can


Why not use deactivate that should be present in that conda environment? Less of a change for people that know conda outside of the galaxy/planemo universe.

Conda's requires sourcing - unlike virtualenv's right? (source deactivate vs deactivate). As a personal taste I don't love that extra step.

Also I usually use planemo from within a virtualenv - what would happen if I replaced deactivate - I guess I could create an old_deactivate alias and restore it? I don't know how to do that in a cross shell compatible way. If someone wanted to work on that though - I'd be happy to merge the PR that did that.

mvdbeek · 2017-02-23T15:10:17Z

docs/_writing_dependencies_conda.rst

+   ``conda search`` command but configured to use the same channels and other options as
+   Planemo and Galaxy. The following Conda command would also work to search::
+
+       $ $HOME/miniconda3/bin/conda -c bioconda -c conda-forge -c iuc seqt


Highest priority is iuc, then bioconda, then conda-forge. See https://github.com/galaxyproject/galaxy/blob/dev/config/galaxy.ini.sample#L212.
So that would be -c iuc -c bioconda -c conda-forge (I'm excluding defaults here, assuming this is always active). Can you confirm this @bgruening ?

mvdbeek · 2017-02-23T15:11:19Z

docs/_writing_dependencies_conda.rst

+When using the website to search though, you need to aware of what channel you are using. By
+default, Planemo and Galaxy will search a few different Conda channels. While it is possible
+to configure a local Planemo or Galaxy to target different channels - the current best practice
+it to add tools to the existing channels.


mvdbeek · 2017-02-23T15:13:40Z

docs/_writing_dependencies_conda.rst

+
+This will download a tool for `PEAR - Paired-End reAd mergeR
+<http://sco.h-its.org/exelixis/web/software/pear/>`__. This tool however has
+``requirement`` tags and so will not work properly.


This tool has no requirements tag ... right?

Yeah - that is what I meant.

mvdbeek · 2017-02-23T15:18:54Z

project_templates/conda_exercise/exercise_1/pear.xml

+<tool id="iuc_pear" name="Pear" version="0.9.6.1">
+    <description>Paired-End read merger</description>
+    <stdio>
+        <exit_code range="1:" />


I think that in the IUC we would probably advocate the use of <command detect_errors="exit_code"> or "aggressive"

It is an IUC tool 😜 - I'll collect your advice and open a PR against tools-iuc.

mvdbeek · 2017-02-23T15:20:34Z

project_templates/conda_exercise/exercise_1/pear.xml

+<![CDATA[
+    pear
+        #if str( $library.type ) == "paired":
+            -f "$library.forward"


And we're advocating single quotes for paths. But I guess it's not so straightforward to implement this in planemo, right?

Also that has nothing to do with this PR, just mentioning it.

jmchilton mentioned this pull request Feb 22, 2017

Dev Corner - Conda for Tool Dependencies galaxyproject/training-material#231

Closed

martenson reviewed Feb 22, 2017

View reviewed changes

jmchilton force-pushed the conda_docs_diverge branch from 055202c to cf795a9 Compare February 22, 2017 18:21

mvdbeek approved these changes Feb 23, 2017

View reviewed changes

jmchilton merged commit cf795a9 into galaxyproject:master Feb 23, 2017

jmchilton added this to the GCC 2017 milestone Feb 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial pass at full documentation for Conda-based dependency development. #642

Initial pass at full documentation for Conda-based dependency development. #642

jmchilton commented Feb 22, 2017

martenson commented Feb 22, 2017

martenson Feb 22, 2017

jmchilton Feb 22, 2017

martenson Feb 22, 2017

jmchilton Feb 22, 2017 •

edited

jmchilton Feb 22, 2017

martenson Feb 22, 2017 •

edited

jmchilton Feb 22, 2017

mvdbeek left a comment

mvdbeek Feb 23, 2017

jmchilton Feb 23, 2017

jmchilton Feb 23, 2017 •

edited by nsoranzo

mvdbeek Feb 23, 2017

mvdbeek Feb 23, 2017

mvdbeek Feb 23, 2017

jmchilton Feb 23, 2017

mvdbeek Feb 23, 2017

mvdbeek Feb 23, 2017

mvdbeek Feb 23, 2017 •

edited

jmchilton Feb 23, 2017

mvdbeek Feb 23, 2017

jmchilton Feb 23, 2017

mvdbeek Feb 23, 2017

mvdbeek Feb 23, 2017

Initial pass at full documentation for Conda-based dependency development. #642

Initial pass at full documentation for Conda-based dependency development. #642

Conversation

jmchilton commented Feb 22, 2017

martenson commented Feb 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmchilton Feb 22, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martenson Feb 22, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvdbeek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmchilton Feb 23, 2017 • edited by nsoranzo

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvdbeek Feb 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmchilton Feb 22, 2017 •

edited

martenson Feb 22, 2017 •

edited

jmchilton Feb 23, 2017 •

edited by nsoranzo

mvdbeek Feb 23, 2017 •

edited