Skip to content

Commit

Permalink
WIP framework dependencies doc update
Browse files Browse the repository at this point in the history
  • Loading branch information
@natefoo
natefoo committed Jan 25, 2018
1 parent 3b69c64 commit 6bdd755
Showing 1 changed file with 77 additions and 113 deletions.
190 changes: 77 additions & 113 deletions doc/source/admin/framework_dependencies.rst
View file
Expand Up @@ -3,85 +3,72 @@
Framework Dependencies
======================

Galaxy is a large Python application with a long list of `Python module
dependencies`_. As a result, the Galaxy developers have made significant effort
to provide these dependencies in as simple a method as possible. Prior to the
16.01 release, this was done by distributing dependencies in Python's old
standard packaging format (Egg) in a non-standard way. As of the 16.01 release,
Galaxy now distributes dependencies in Python's new standard packaging format
(Wheel), albeit still in a non-standard way. Thankfully, the new distribution
method is far more compatible with the standard package distribution tooling
(i.e. `pip`_) than the old method.
Galaxy is a large Python application with a long list of `Python module dependencies`_. As a result, the Galaxy
developers have made significant effort to provide these dependencies in as simple a method as possible while remaining
compatible with the `Python packaging best practices`_. Thus, Galaxy's runtime setup procedure makes use of virtualenv_
for package environment isolation, pip_ for installation, and wheel_ to provide pre-built versions of dependencies.

In addition to framework dependencies, as of Galaxy 18.01, the client (UI) is no longer provided in its built format
**in the development (``dev``) branch of the source repository**. The built client is still provided in
``release_YY.MM`` branches and the ``master`` branch.

.. _Python module dependencies: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/dependencies/requirements.txt
.. _pip: https://pip.pypa.io/
.. _wheel: https://wheel.readthedocs.org/
.. _Python packaging best practices: https://packaging.python.org
.. _virtualenv: https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments
.. _pip: https://packaging.python.org/tutorials/installing-packages/#use-pip-for-installing
.. _wheel: https://packaging.python.org/tutorials/installing-packages/#source-distributions-vs-wheels

How it works
------------

Upon startup (with ``run.sh``), the startup scripts will:

1. Create a Python `virtualenv`_ in the directory ``.venv``.

2. Unset the ``$PYTHONPATH`` environment variable (if set) as this can
interfere with installing `Galaxy pip`_ and dependencies.
1. Create a Python virtualenv_ in the directory ``.venv``.

3. Replace that virtualenv's pip with `Galaxy pip`_.
2. Unset the ``$PYTHONPATH`` environment variable (if set) as this can interfere with installing dependencies.

4. If applicable, create a ``binary-compatibility.cfg`` (see the `Galaxy pip
and wheel`_ section for an explanation of this file).
3. Download and install packages from the Galaxy project wheel server, wheels.galaxyproject.org_, as well as the `Python
Package Index`_ (aka PyPI) , using pip_.

5. Download and install wheels from the Galaxy project wheel server,
`wheels.galaxyproject.org`_, using pip.
4. Start Galaxy using ``.venv/bin/python``.

6. Start Galaxy using ``.venv/bin/python``.

.. _virtualenv: https://virtualenv.readthedocs.org/
.. _wheels.galaxyproject.org: https://wheels.galaxyproject.org/
.. _Python Package Index: https://pypi.org

Options
-------

A variety of options to ``run.sh`` are available to control the above behavior:

- ``--skip-venv``: Do not create or use a virtualenv, and do not replace pip
with Galaxy pip.
- ``--skip-venv``: Do not create or use a virtualenv.
- ``--skip-wheels``: Do not install wheels.
- ``--no-create-venv``: Do not create a virtualenv, but use one if it exists at
``.venv`` or if ``$VIRTUAL_ENV`` is set (this variable is set by virtualenv's
``activate``)
- ``--replace-pip/--no-replace-pip``: Do/do not replace pip with Galaxy pip.
- ``--no-create-venv``: Do not create a virtualenv, but use one if it exists at ``.venv`` or if ``$VIRTUAL_ENV`` is set
(this variable is set by virtualenv's ``activate``).
- ``--replace-pip/--no-replace-pip``: Do/do not upgrade pip if necessary.

Managing dependencies manually
------------------------------

Create a virtualenv
^^^^^^^^^^^^^^^^^^^

Using a `virtualenv`_ in ``.venv`` under the Galaxy source tree is not
required. More complicated Galaxy setups may choose to use a virtualenv
external to the Galaxy source tree, which can be done either by not using
``run.sh`` directly (an example of this can be found under the `Supervisor`_
section) or using the ``--no-create-venv`` option, explained in the `Options`_
section. It is also possible to force Galaxy to start without a virtualenv at
all, but you should not do this unless you know what you're doing.
Using a `virtualenv`_ in ``.venv`` under the Galaxy source tree is not required. More complicated Galaxy setups may
choose to use a virtualenv external to the Galaxy source tree, which can be done either by not using ``run.sh`` directly
(an example of this can be found under the `Scaling and Load Balancing` documentation) or using the ``--no-create-venv``
option, explained in the `Options` section. It is also possible to force Galaxy to start without a virtualenv at all,
but you should not do this unless you know what you're doing.

To manually create a virtualenv, you will first need to obtain virtualenv.
There are a variety of ways to do this:
To manually create a virtualenv, you will first need to obtain virtualenv. There are a variety of ways to do this:

- ``pip install virtualenv``
- ``brew install virtualenv``
- Install your Linux distribution's virtualenv package from the system package
manager (e.g. ``apt-get install python-virtualenv``).
- Download the `virtualenv source from PyPI
<https://pypi.python.org/pypi/virtualenv>`_, untar, and run the
``virtualenv.py`` script contained within as ``python virtualenv.py
/path/to/galaxy/virtualenv``
- Install your Linux distribution's virtualenv package from the system package manager (e.g. ``apt-get install
python-virtualenv``).
- Download the `virtualenv source from PyPI <https://pypi.python.org/pypi/virtualenv>`_, untar, and run the
``virtualenv.py`` script contained within as ``python virtualenv.py /path/to/galaxy/virtualenv``

Once this is done, create a virtualenv. In our example, the virtualenv will
live in ``/srv/galaxy/venv`` and the Galaxy source code has been cloned to
``/srv/galaxy/server``.
Once this is done, create a virtualenv. In our example, the virtualenv will live in ``/srv/galaxy/venv`` and the Galaxy
source code has been cloned to ``/srv/galaxy/server``.

.. code-block:: console
Expand All @@ -94,94 +81,71 @@ live in ``/srv/galaxy/venv`` and the Galaxy source code has been cloned to
Install dependencies
^^^^^^^^^^^^^^^^^^^^

Normally, ``run.sh`` calls `common_startup.sh`_, which creates the virtualenv,
installs Galaxy pip, and installs dependencies. You can call this script
yourself to set up Galaxy pip and the dependencies without creating a
virtualenv using the ``--no-create-venv`` option:
Normally, ``run.sh`` calls `common_startup.sh`_, which creates the virtualenv and installs dependencies. You can call
this script yourself to set up Galaxy pip and the dependencies without creating a virtualenv using the
``--no-create-venv`` option:

.. code-block:: console
(venv)$ PYTHONPATH= sh /srv/galaxy/server/scripts/common_startup.sh --no-create-venv
Ignoring indexes: https://pypi.python.org/simple
Collecting pip
Downloading https://wheels.galaxyproject.org/packages/pip-8.0.0+gx1-py2.py3-none-any.whl (1.2MB)
100% |████████████████████████████████| 1.2MB 37.8MB/s
Installing collected packages: pip
Found existing installation: pip 7.1.2
Uninstalling pip-7.1.2:
Successfully uninstalled pip-7.1.2
Successfully installed pip-8.0.0+gx1
Collecting bx-python==0.7.3 (from -r requirements.txt (line 2))
Downloading https://wheels.galaxyproject.org/packages/bx_python-0.7.3-cp27-cp27mu-linux_x86_64.whl (1.7MB)
100% |████████████████████████████████| 1.7MB 25.4MB/s
...
Collecting pysam==0.8.3+gx1 (from -r requirements.txt (line 69))
Downloading https://wheels.galaxyproject.org/packages/pysam-0.8.3+gx1-cp27-cp27mu-linux_x86_64.whl (7.4MB)
100% |████████████████████████████████| 7.4MB 15.1MB/s
Installing collected packages: bx-python, MarkupSafe, PyYAML, SQLAlchemy,
mercurial, numpy, pycrypto, six, Paste, PasteDeploy, docutils, wchartype,
repoze.lru, Routes, WebOb, WebHelpers, Mako, pytz, Babel, Beaker,
Markdown, Cheetah, requests, requests-toolbelt, boto, bioblend, amqp,
anyjson, kombu, pbr, sqlparse, decorator, Tempita, sqlalchemy-migrate,
Parsley, nose, svgwrite, ecdsa, paramiko, Fabric, Whoosh, pysam
Successfully installed Babel-2.0 Beaker-1.7.0 Cheetah-2.4.4 Fabric-1.10.2
Mako-1.0.2 Markdown-2.6.3 MarkupSafe-0.23 Parsley-1.3 Paste-2.0.2
PasteDeploy-1.5.2 PyYAML-3.11 Routes-2.2 SQLAlchemy-1.0.8 svgwrite-1.1.6
Tempita-0.5.3.dev0 WebHelpers-1.3 WebOb-1.4.1 Whoosh-2.4.1+gx1 amqp-1.4.8
anyjson-0.3.3 bioblend-0.6.1 boto-2.38.0 bx-python-0.7.3 decorator-4.0.2
docutils-0.12 ecdsa-0.13 kombu-3.0.30 mercurial-3.4.2 nose-1.3.7
numpy-1.9.2 paramiko-1.15.2 pbr-1.8.0 pycrypto-2.6.1 pysam-0.8.3+gx1
pytz-2015.4 repoze.lru-0.6 requests-2.8.1 requests-toolbelt-0.4.0
six-1.9.0 sqlalchemy-migrate-0.10.0 sqlparse-0.1.16 wchartype-0.1
**Warning:** If your ``$PYTHONPATH`` is set, it may interfere with the
dependency installation process (this will almost certainly be the case if you
use `virtualenv-burrito`_). Without ``--no-create-venv`` the ``$PYTHONPATH``
variable will be automatically unset, but we assume you know what you're doing
and may want it left intact if you are using ``--no-create-venv``. If you
encounter problems, try unsetting ``$PYTHONPATH`` as shown in the example
above.
Requirement already satisfied: pip>=8.1 in /home/nate/.virtualenvs/test/lib/python2.7/site-packages
Collecting numpy==1.9.2 (from -r requirements.txt (line 4))
Downloading https://wheels.galaxyproject.org/packages/numpy-1.9.2-cp27-cp27mu-manylinux1_x86_64.whl (10.2MB)
100% |████████████████████████████████| 10.2MB 21.7MB/s
Collecting bx-python==0.7.3 (from -r requirements.txt (line 5))
Downloading https://wheels.galaxyproject.org/packages/bx_python-0.7.3-cp27-cp27mu-manylinux1_x86_64.whl (2.1MB)
100% |████████████████████████████████| 2.2MB 97.2MB/s
...
Installing collected packages: numpy, bx-python, ...
Successfully installed numpy-1.9.2 bx-python-0.7.3 ...
.. warning::

If your ``$PYTHONPATH`` is set, it may interfere with the dependency installation process. Without
``--no-create-venv`` the ``$PYTHONPATH`` variable will be automatically unset, but we assume you know what you're
doing and may want it left intact if you are using ``--no-create-venv``. If you encounter problems, try unsetting
``$PYTHONPATH`` as shown in the example above.

.. _common_startup.sh: https://github.com/galaxyproject/galaxy/blob/dev/scripts/common_startup.sh
.. _virtualenv-burrito: https://github.com/brainsik/virtualenv-burrito

Installing unpinned dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Galaxy's dependencies can be installed either "pinned" (they will be installed
at exact versions specified for your Galaxy release) or "unpinned" (the latest
versions of all dependencies will be installed unless there are known
incompatibilities with new versions). By default, the release branch(es) of
Galaxy use pinned versions for three reasons:
Galaxy's dependencies can be installed either "pinned" (they will be installed at exact versions specified for your
Galaxy release) or "unpinned" (the latest versions of all dependencies will be installed unless there are known
incompatibilities with new versions). By default, the release branches of Galaxy use pinned versions for three reasons:

1. Using pinned versions insures that the prebuilt wheels on `wheels.galaxyproject.org`_ will be installed, and no
compilation will be necesseary.

1. Using pinned versions insures that the prebuilt wheels on
`wheels.galaxyproject.org`_ will be installed, and no compilation will be
necesseary.
2. Galaxy releases are tested with the pinned versions and this allows us to give as much assurance as possible that the
pinned versions will work with the given Galaxy release (especially as time progresses and newer dependency versions
are released while the Galaxy release receives fewer updates.

2. Galaxy releases are tested with the pinned versions and this allows us to
give as much assurance as possible that the pinned versions will work with
the given Galaxy release (especially as time progresses and newer dependency
versions are released while the Galaxy release receives fewer updates.
3. Pinning furthers Galaxy's goal of reproducibility as differing dependency versions could result in non-reproducible
behavior.

3. Pinning furthers Galaxy's goal of reproducibility as differing dependency
versions could result in non-reproducible behavior.
Pinning management is being migrated to `pipenv`_. See `Pull Request #4891`_ for details.

Install dependencies using the `unpinned requirements file`_, and then instruct
Galaxy to start without attempting to fetch wheels:
If you would like to install unpinned versions of Galaxy's dependencies, Install dependencies using the `unpinned
requirements file`_, and then instruct Galaxy to start without attempting to fetch wheels:

.. code-block:: console
(venv)$ pip install --index-url=https://wheels.galaxyproject.org/simple/ -r lib/galaxy/dependencies/requirements.txt
(venv)$ pip install -r lib/galaxy/dependencies/requirements.txt
(venv)$ deactivate
$ sh run.sh --no-create-venv --skip-wheels
Including ``--index-url=https://wheels.galaxyproject.org/simple/`` is important
as two dependencies (pysam, Whoosh) include modifications specific to Galaxy
which are only available on `wheels.galaxyproject.org`_.
You may be able to save yourself some compiling by adding the argument ``--index-url
https://wheels.galaxyproject.org/simple/`` to ``pip install``, but it is possible to install all fo Galaxy's
dependencies directly from PyPI_.

.. _pipenv: http://pipenv.readthedocs.io/
.. _Pull Request #4891: https://github.com/galaxyproject/galaxy/pull/4891
.. _unpinned requirements file: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/dependencies/requirements.txt
.. _PyPI: https://pypi.org

Wheel interaction with other software
-------------------------------------
Expand Down

0 comments on commit 6bdd755

Please sign in to comment.