Skip to content

Commit

Permalink
Add --skip_venv flag for conda, other cases.
Browse files Browse the repository at this point in the history 
Fixes #377 ... or makes it possible to fix it anyway.

To preserve conda's environment for test execution run planemo, first install Galaxy's depndencies into the conda environment using:

    % cd $GALAXY_ROOT
    % pip install requirements.txt
    % pip install -r lib/galaxy/dependencies/dev-requirements.txt

Then run planemo test, serve, etc... with the --skip_venv flag.

    % planemo test --skip_venv .
  • Loading branch information
@jmchilton
jmchilton committed Jan 13, 2016
1 parent cd7cabc commit 9f3957d
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 5 deletions.
5 changes: 4 additions & 1 deletion planemo/galaxy_run.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@


def setup_venv(ctx, kwds):
if kwds.get("skip_venv", False):
return ""

create_template_params = {
'create_virtualenv': create_command("$GALAXY_VIRTUAL_ENV")
}
Expand Down Expand Up @@ -67,7 +70,7 @@ def shell_if_wheels(command):

def setup_common_startup_args():
return set_variable_if_wheels(
"COMMON_STARTUP_ARGS", "--skip-venv --dev-wheels"
"COMMON_STARTUP_ARGS", "--dev-wheels"
)


Expand Down
4 changes: 0 additions & 4 deletions planemo/galaxy_serve.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,6 @@ def serve(ctx, paths, **kwds):

with galaxy_config.galaxy_config(ctx, paths, **kwds) as config:
# TODO: Allow running dockerized Galaxy here instead.
setup_common_startup_args = galaxy_run.set_variable_if_wheels(
"COMMON_STARTUP_ARGS", "--skip-venv"
)
setup_venv_command = galaxy_run.setup_venv(ctx, kwds)
run_script = os.path.join(config.galaxy_root, "run.sh")
run_script += " $COMMON_STARTUP_ARGS"
Expand All @@ -33,7 +30,6 @@ def serve(ctx, paths, **kwds):
cd_to_galaxy_command = "cd %s" % config.galaxy_root
cmd = io.shell_join(
cd_to_galaxy_command,
setup_common_startup_args,
setup_venv_command,
run_script,
)
Expand Down
11 changes: 11 additions & 0 deletions planemo/options.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,16 @@ def force_option(what="files"):
)


def skip_venv_option():
return click.option(
"--skip_venv",
is_flag=True,
help=("Do not create or source a virtualenv environment for Galaxy, "
"this should be used or instance to preserve an externally "
"configured virtual environment or conda environment.")
)


def test_data_option():
return click.option(
"--test_data",
Expand Down Expand Up @@ -643,6 +653,7 @@ def galaxy_target_options():
install_galaxy_option(),
galaxy_branch_option(),
galaxy_source_option(),
skip_venv_option(),
no_cache_galaxy_option(),
no_cleanup_option(),
job_config_option(),
Expand Down

7 comments on commit 9f3957d

@gregvonkuster
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @jmchilton

@bgruening
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first command should probably be:

pip install -r requirements.txt

But even than it's not working, because some requirements are only available on Galaxy depot. In this case the following command would be useful.

pip install -r requirements.txt --extra-index-url https://wheels.galaxyproject.org/simple

But even that is not working for me, because a few packages could not be found. @gregvonkuster is it working for you?

@jmchilton
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conda FAQ:

  1. You can absolutely not install planemo via conda and run Galaxy inside the conda environment currently.

    galaxy-lib is a dependency of planemo and having this on the Python path prevents correct operation of Galaxy.

  2. How do I enable conda dependency resolution within Galaxy tools (install conda dependencies corresponding to requirement tags)?

    Just install planemo normally via pip into a virtual environment or via brew and use the appropriate commands and options:

    $ planemo conda_init
$ planemo conda_install . 
$ planemo test --conda_dependency_resolution .
$ planemo serve --conda_dependency_resolution .

    The test and serve commands above require the target Galaxy to be 16.01 or higher, the default target is still 15.10 so the above command is assuming galaxy_root has been overridden to ~/.planemo.yml.

  3. How do I manually install some conda dependencies and test my tools using them in Galaxy.

    This is an advanced technique and I'd frankly discourage it currently, but should be possible.

    planemo must be installed outside the conda environment - via pip into a virtual environment or via homebrew. In the former case, don't place the virtualenv's bin directory on your PATH - it will conflict with conda. Just reference planemo directly /path/to/planemo_venv/bin/planemo test.

    To run Galaxy from within the environment you will need to install Galaxy dependencies into the conda environment. I'd absolutely recommend targeting the development branch of Galaxy for this since it has "unpinned" dependencies that are easier to fullfill for conda.

    (conda-env-test) $ conda install numpy bx-python pysam # install the hard ones using conda
(conda-env-test) $ cd $GALAXY_ROOT
(conda-env-test) $ ./scripts/common_startup.sh --skip-venv --dev-wheels # install remaining ones using pip via Galaxy
(conda-env-test) $ cd /path/to/my/tools
(conda-env-test) $ /path/to/planemo_venv/bin/planemo test --skip_venv  .
(conda-env-test) $ /path/to/planemo_venv/bin/planemo serve --skip_venv  .

    A small test script in the planemo source tree demonstrates this https://github.com/galaxyproject/planemo/blob/master/tests/scripts/conda_test.sh.

@jmchilton
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bgruening Absolutely - I missed up those pip calls are not a good way to get Galaxy dependencies into your virtualenv. I created a short FAQ above, which should land up in planemo's docs at some point once we all like it based on the problems we discussed out of band and here. In the FAQ I recommend installing numpy and a couple other big dependencies using conda directly and then install the rest using common_startup.sh so the correct index is available.

@bgruening
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will test again. The --skip_venv was also misleading in my case.

@jmchilton
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a test script that demonstrates (and tests) my answer to question 2 above - lets call this running planemo around conda.

https://github.com/galaxyproject/planemo/blob/master/tests/scripts/conda_test.sh

If anyone really cares about the use case, feel free to extend the example in either of these directions:

  • Create a new tool example to with a test case that actually installs something useful in the environment
    requires that something for the tool test to path.
  • Add the script as a tox test.
  • Add the script as a travis test.

@jmchilton
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved above comment (with cleanup language) to the README.

Please sign in to comment.