Permalink
Browse files

Add --skip_venv flag for conda, other cases.

Fixes #377 ... or makes it possible to fix it anyway.

To preserve conda's environment for test execution run planemo, first install Galaxy's depndencies into the conda environment using:

    % cd $GALAXY_ROOT
    % pip install requirements.txt
    % pip install -r lib/galaxy/dependencies/dev-requirements.txt

Then run planemo test, serve, etc... with the --skip_venv flag.

    % planemo test --skip_venv .
  • Loading branch information...
jmchilton committed Jan 13, 2016
1 parent cd7cabc commit 9f3957d3552153cd76ec34697e9857be13a24e73
Showing with 15 additions and 5 deletions.
  1. +4 −1 planemo/galaxy_run.py
  2. +0 −4 planemo/galaxy_serve.py
  3. +11 −0 planemo/options.py
@@ -33,6 +33,9 @@


def setup_venv(ctx, kwds):
if kwds.get("skip_venv", False):
return ""

create_template_params = {
'create_virtualenv': create_command("$GALAXY_VIRTUAL_ENV")
}
@@ -67,7 +70,7 @@ def shell_if_wheels(command):

def setup_common_startup_args():
return set_variable_if_wheels(
"COMMON_STARTUP_ARGS", "--skip-venv --dev-wheels"
"COMMON_STARTUP_ARGS", "--dev-wheels"
)


@@ -17,9 +17,6 @@ def serve(ctx, paths, **kwds):

with galaxy_config.galaxy_config(ctx, paths, **kwds) as config:
# TODO: Allow running dockerized Galaxy here instead.
setup_common_startup_args = galaxy_run.set_variable_if_wheels(
"COMMON_STARTUP_ARGS", "--skip-venv"
)
setup_venv_command = galaxy_run.setup_venv(ctx, kwds)
run_script = os.path.join(config.galaxy_root, "run.sh")
run_script += " $COMMON_STARTUP_ARGS"
@@ -33,7 +30,6 @@ def serve(ctx, paths, **kwds):
cd_to_galaxy_command = "cd %s" % config.galaxy_root
cmd = io.shell_join(
cd_to_galaxy_command,
setup_common_startup_args,
setup_venv_command,
run_script,
)
@@ -18,6 +18,16 @@ def force_option(what="files"):
)


def skip_venv_option():
return click.option(
"--skip_venv",
is_flag=True,
help=("Do not create or source a virtualenv environment for Galaxy, "
"this should be used or instance to preserve an externally "
"configured virtual environment or conda environment.")
)


def test_data_option():
return click.option(
"--test_data",
@@ -643,6 +653,7 @@ def galaxy_target_options():
install_galaxy_option(),
galaxy_branch_option(),
galaxy_source_option(),
skip_venv_option(),
no_cache_galaxy_option(),
no_cleanup_option(),
job_config_option(),

7 comments on commit 9f3957d

@gregvonkuster

This comment has been minimized.

Copy link
Contributor

gregvonkuster replied Jan 14, 2016

Thanks for this @jmchilton

@bgruening

This comment has been minimized.

Copy link
Member

bgruening replied Jan 14, 2016

The first command should probably be:

pip install -r requirements.txt

But even than it's not working, because some requirements are only available on Galaxy depot. In this case the following command would be useful.

pip install -r requirements.txt --extra-index-url https://wheels.galaxyproject.org/simple

But even that is not working for me, because a few packages could not be found. @gregvonkuster is it working for you?

@jmchilton

This comment has been minimized.

Copy link
Member Author

jmchilton replied Jan 14, 2016

Conda FAQ:

  1. You can absolutely not install planemo via conda and run Galaxy inside the conda environment currently.

    galaxy-lib is a dependency of planemo and having this on the Python path prevents correct operation of Galaxy.

  2. How do I enable conda dependency resolution within Galaxy tools (install conda dependencies corresponding to requirement tags)?

    Just install planemo normally via pip into a virtual environment or via brew and use the appropriate commands and options:

    $ planemo conda_init
    $ planemo conda_install . 
    $ planemo test --conda_dependency_resolution .
    $ planemo serve --conda_dependency_resolution .
    

    The test and serve commands above require the target Galaxy to be 16.01 or higher, the default target is still 15.10 so the above command is assuming galaxy_root has been overridden to ~/.planemo.yml.

  3. How do I manually install some conda dependencies and test my tools using them in Galaxy.

    This is an advanced technique and I'd frankly discourage it currently, but should be possible.

    planemo must be installed outside the conda environment - via pip into a virtual environment or via homebrew. In the former case, don't place the virtualenv's bin directory on your PATH - it will conflict with conda. Just reference planemo directly /path/to/planemo_venv/bin/planemo test.

    To run Galaxy from within the environment you will need to install Galaxy dependencies into the conda environment. I'd absolutely recommend targeting the development branch of Galaxy for this since it has "unpinned" dependencies that are easier to fullfill for conda.

    (conda-env-test) $ conda install numpy bx-python pysam # install the hard ones using conda
    (conda-env-test) $ cd $GALAXY_ROOT
    (conda-env-test) $ ./scripts/common_startup.sh --skip-venv --dev-wheels # install remaining ones using pip via Galaxy
    (conda-env-test) $ cd /path/to/my/tools
    (conda-env-test) $ /path/to/planemo_venv/bin/planemo test --skip_venv  .
    (conda-env-test) $ /path/to/planemo_venv/bin/planemo serve --skip_venv  .
    

    A small test script in the planemo source tree demonstrates this https://github.com/galaxyproject/planemo/blob/master/tests/scripts/conda_test.sh.

@jmchilton

This comment has been minimized.

Copy link
Member Author

jmchilton replied Jan 14, 2016

@bgruening Absolutely - I missed up those pip calls are not a good way to get Galaxy dependencies into your virtualenv. I created a short FAQ above, which should land up in planemo's docs at some point once we all like it based on the problems we discussed out of band and here. In the FAQ I recommend installing numpy and a couple other big dependencies using conda directly and then install the rest using common_startup.sh so the correct index is available.

@bgruening

This comment has been minimized.

Copy link
Member

bgruening replied Jan 14, 2016

Ok, will test again. The --skip_venv was also misleading in my case.

@jmchilton

This comment has been minimized.

Copy link
Member Author

jmchilton replied Jan 14, 2016

I have added a test script that demonstrates (and tests) my answer to question 2 above - lets call this running planemo around conda.

https://github.com/galaxyproject/planemo/blob/master/tests/scripts/conda_test.sh

If anyone really cares about the use case, feel free to extend the example in either of these directions:

  • Create a new tool example to with a test case that actually installs something useful in the environment
    requires that something for the tool test to path.
  • Add the script as a tox test.
  • Add the script as a travis test.
@jmchilton

This comment has been minimized.

Copy link
Member Author

jmchilton replied Jan 14, 2016

Moved above comment (with cleanup language) to the README.

Please sign in to comment.