Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependency resolver documentation revisions. #3483

Merged
merged 5 commits into from Jan 25, 2017

Conversation

Projects
None yet
4 participants
@jmchilton
Copy link
Member

commented Jan 25, 2017

  • Add documentation for cached dependency management as solution to LOCKERRORS.
  • Add documentation for overcoming problems with hard links. (Different approaches)
  • Add documentation on upgrading Conda as needed.
  • Correct and fill out documentation of defaults.
  • Fix some bugs in the Galaxy packages documentation.
  • Expand Galaxy package documentation to include a simpler example.
  • Update Conda configuration option documentation to reflect new defaults.
  • Update Conda configuration option documentation to discuss copy_dependencies.
  • Cross link Conda and dependency resolver documentation as it makes sense.
  • Add a link to the Bioconda channel order documentation for future reference.
  • Make things a bit more concise in places - start with the implicit tl;dr.

Fixes #3248

Dependency resolver documentation revisions.
- Add documentation for cached dependency management as solution to LOCKERRORS.
- Add documentation for overcoming problems with hard links. (Different approaches)
- Add documentation on upgrading Conda as needed.
- Correct and fill out documentation of defaults.
- Fix some bugs in the Galaxy packages documentation.
- Expand Galaxy package documentation to include a simpler example.
- Update Conda configuration option documentation to reflect new defaults.
- Update Conda configuration option documentation to discuss copy_dependencies.
- Cross link Conda and dependency resolver documentation as it makes sense.
- Add a link to the Bioconda channel order documentation for future reference.
- Make things a bit more concise in places - start with the implicit tl;dr.
@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Jan 25, 2017

This does not yet cover mapping per se - but it is hopefully a very solid step forward.

martenson added some commits Jan 25, 2017

@@ -25,8 +21,8 @@ community contributions (such as implementing a Guix package manager or
enhancing the existing brew support to bring it on par with Conda).

As a community, we have decided that Conda is the one that best fulfills
community's needs. The following are some of the crucial Conda features that led
to this decision:
tje community's current needs. The following are some of the crucial Conda

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Jan 25, 2017

Member

s/tje/the

@@ -214,6 +203,11 @@ The order in which resolvers are tried is listed in the
The first system that satisfies a requirement will be used. See
`resolver docs`_ for detailed documentation.

This however is not recommended, ideally tools will target and test
against Conda for all dependencies. Also resolving all requirements
with Conda gives Conda a chance to select compatibile versions of

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Jan 25, 2017

Member

s/compatibile/compatible

per job Conda environment with just the dependencies needed for that job installed.
This will be placed on the filesystem containg the job working directory. This
is an expensive operation and Conda doesn't always link environments correctly
accross filesystems. Enabling this job caching will create a cache for each required

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Jan 25, 2017

Member

s/accross/across/


If Galaxy jobs run on filesystems that cannot hardlink Conda packages managed
by Galaxy, linking errors may occur when building environment to execute jobs.
There are a few ways to potentially work around this discussed below.

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Jan 25, 2017

Member

s/ discussed below/

resolvers succeeds a dependency resolution object is returned and no more resolvers are called. This dependency
resolution object provides shell commands to prepend to the shell script that runs the tool.

This order can be thought of as a descending order of deliberation. Tool Shed dependencies must be declared next to the
tool by the tool author and must be selected for installation at tool installation time - this requires specific actions
by both the tool author and the deployer who isntalled the tools. The dependency is therefore expected to highly craft

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Jan 25, 2017

Member

s/isntalled/installed

martenson added some commits Jan 25, 2017

@martenson martenson merged commit 3eaad9a into galaxyproject:dev Jan 25, 2017

0 of 4 checks passed

api test Test scheduled.
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
framework test Test scheduled.
Details
toolshed test Test scheduled.
Details
@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Jan 25, 2017

@martenson and @mvdbeek - thanks for cleaning up my mess and thanks for the quick merge!

@martenson

This comment has been minimized.

Copy link
Member

commented Jan 25, 2017

@jmchilton thanks for the docs dude!

@@ -48,14 +44,19 @@ Below we answer some common questions (collected by Lance Parsons):
1. How do I enable Conda dependency resolution for Galaxy jobs?

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Jan 25, 2017

Member

s/jobs/tools/

The short answer is that as of 17.01, Galaxy should install Conda the first
time it starts up and be configured to use it by default.

The long answer is that Galaxy's dependency job resolution is managed via

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Jan 25, 2017

Member

s/dependency job/tool dependency/

dependencies using the Conda system when it attempts to run tools.
Conda was already installed on the system. As of 17.01, Galaxy will also install
Conda as needed when starting up. Having Conda enabled in ``dependency_resolvers_conf.xml``
means that Galaxy can look for job dependencies using the Conda system when it

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Jan 25, 2017

Member

s/job/tool/

Conda was already installed on the system. As of 17.01, Galaxy will also install
Conda as needed when starting up. Having Conda enabled in ``dependency_resolvers_conf.xml``
means that Galaxy can look for job dependencies using the Conda system when it
attempts to run tools.

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Jan 25, 2017

Member

s/tools/a job/

per job Conda environment with just the dependencies needed for that job installed.
This will be placed on the filesystem containg the job working directory. This
is an expensive operation and Conda doesn't always link environments correctly
accross filesystems. Enabling this job caching will create a cache for each required

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Jan 25, 2017

Member

s/accross/across/

You can also use: $ conda clean --lock
First, you may wish to enable cached dependencies. This can be done by setting
``use_cached_dependency_manager`` in ``galaxy.ini``. Many jobs will create a

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Jan 25, 2017

Member

Add: Without this option, before Many jobs...

newer version of Conda than shipped with Galaxy as of 17.01. See the question below
on upgrading Conda if you must use this trick.

Alternatively, copying can be used when creating environments instead links (either

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Jan 25, 2017

Member

s/instead links/instead of links/


ensure_channels
conda channels to enable by default. See
http://conda.pydata.org/docs/custom-channels.html for more
information about channels. (default: iuc,bioconda,r,defaults,conda-forge).
information about channels. This defaults to ``iuc,bioconda,r,defaults,conda-forge``.
This order should be consistent with `Bioconda perscribed order <https://github.com/bioconda/bioconda-recipes/blob/master/config.yml#L8>`__

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Jan 25, 2017

Member

s/perscribed/prescribed/

@nsoranzo

This comment has been minimized.

Copy link
Member

commented Jan 25, 2017

This PR targeted the wrong branch!

@martenson martenson changed the title [17.01] Dependency resolver documentation revisions. Dependency resolver documentation revisions. Jan 25, 2017

@martenson martenson deleted the jmchilton:doc_revision_0 branch Jan 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.