Conda dependency resolver can break Galaxy's set metadata process

[blankenberg]

If a tool creates and uses a conda environment that contains python, and a default samtools installation is not specified, the set metadata process will fail causing a tool failure.

What appears to be happening is that the set metadata process is run under an environment containing both the tool's conda deps and the samtools conda dependency. Errors that I've seen include:

(probably using e.g. python2.7)

Traceback (most recent call last):
  File "/mnt/galaxy/tmp/job_working_directory/001/1259/set_metadata_RM1qnb.py", line 1, in <module>
    from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/mnt/galaxy/galaxy-app/lib/galaxy_ext/metadata/set_metadata.py", line 23, in <module>
    from sqlalchemy.orm import clear_mappers
ImportError: No module named sqlalchemy.orm

These are due to conda env containing python 3:

Fatal error: 
Traceback (most recent call last):
  File "/mnt/galaxy/tmp/job_working_directory/001/1268/set_metadata_X7301T.py", line 1, in <module>
    from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/mnt/galaxy/galaxy-app/lib/galaxy_ext/metadata/set_metadata.py", line 140
    except Exception, e:
                    ^
SyntaxError: invalid syntax

and (after fixing the exception syntax not valid in py3):

Fatal error: 
Traceback (most recent call last):
  File "/mnt/galaxy/tmp/job_working_directory/001/1270/set_metadata_TWdgBO.py", line 1, in <module>
    from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/mnt/galaxy/galaxy-app/lib/galaxy_ext/metadata/set_metadata.py", line 15, in <module>
    import cPickle
ImportError: No module named 'cPickle'

etc.

The work around is to define a default samtools dependency, e.g. on a cloudman instance:

ln -s /mnt/galaxy/tools/samtools/0.1.19/devteam/package_samtools_0_1_19/95d2c4aefb5f//mnt/galaxy/tools/samtools/default

this prevents the conda environment from being sourced prior to the set metadata tool process being called.

[bgruening]

I discussed yesterday with @jmchilton a little bit if it would be feasible to have the set_metadata function as separate tool, with it's own dependencies. What do you think is this likely? TS datatypes might be a problem?

[blankenberg]

The set metadata function is actually already its own tool (with a non-versioned samtools requirement: lib/galaxy/datatypes/set_metadata_tool.xml), but the conda environment is shared between the tool and the set metadata process. First step is probably to break these into two separate conda envs that are activated and deactivated for their respective portions. We could then allow datatypes to individually declare their dependencies.

A significant issue is that we don't always know an output's datatype until after the tool finishes and the files have been created, e.g. created due to pattern matching.

[jmchilton]

I can't replicate this in dev - is it possible that tool separation between metadata and tool execution wasn't enabled when this was seen?

Does someone have logging around and the scripts generated during this error? On my local machine if I let samtools get resolved with conda - the job script has a line that starts like this:

mkdir -p working; cd working; /home/john/workspace/galaxy/database/jobs_directory/000/135/\
tool_script.sh; return_code=$?; cd '/home/john/workspace/galaxy/database/jobs_directory/00\
0/135'; [ "$CONDA_DEFAULT_ENV" = "/home/john/workspace/galaxy/database/jobs_directory/000/\
135/conda-metadata-env" ] || . /home/john/workspace/galaxy/database/tool_dependencies/_con\
da/bin/activate '/home/john/workspace/galaxy/database/jobs_directory/000/135/conda-metadat\
a-env' 2>&1 ; python "/home/john/workspace/galaxy/database/jobs_directory/000/135/set_meta\
data_CpVfa_.py"

and then the Python 3 environment inside conda looks like this:

[ "$CONDA_DEFAULT_ENV" = "/home/john/workspace/galaxy/database/jobs_directory/000/135/cond\
a-env" ] || . /home/john/workspace/galaxy/database/tool_dependencies/_conda/bin/activate '\
/home/john/workspace/galaxy/database/jobs_directory/000/135/conda-env' 2>&1 ; cp /home/joh\
n/workspace/galaxy/database/files/000/dataset_211.dat /home/john/workspace/galaxy/database\
/files/000/dataset_225.dat; echo "chrM" > "/home/john/workspace/galaxy/database/files/000/\
dataset_224.dat"

I don't understand how this could cause this problem. That is being sourced inside of tool_script.sh and conda is being resourced outside of tool_script.sh with the correct environment. When this problem occurs - is tool_script.sh not being generated?

Is the problem maybe that the conda samtools environment is getting Python 3 instead of the tool's?

% . /home/john/workspace/galaxy/database/tool_dependencies/_conda/bin/activate '/home/john/workspace/galaxy/database/jobs_directory/000/135/conda-metadata-env'
% python --version
Python 2.7.6

On a machine that has this problem, can someone source the metadata environment and check the resulting Python version?

[mvdbeek]

@blankenberg that was exactly my motivation for #2253.
Do you have those commits?
Also, is enable_beta_tool_command_isolation active?

[jmchilton]

This was a 16.04 cloudman image originally - I bet it didn't have #2253. I forgot this was introduced so late 😢, I bet this problem is already resolved. Thanks @mvdbeek!

[nsoranzo]

Can we backport #2253 to 16.04? I have a similar problem:

Traceback (most recent call last):
  File "/tmp/tmpvGDLc3/job_working_directory/000/3/set_metadata_66ueQp.py", line 1, in <module>
    from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/usr/users/ga002/soranzon/software/galaxyproject_galaxy_release_16.04/lib/galaxy_ext/metadata/set_metadata.py", line 23, in <module>
    from sqlalchemy.orm import clear_mappers
ImportError: No module named sqlalchemy.orm

when testing meme with Planemo on Galaxy 16.04, but not on dev branch.

[jmchilton]

@nsoranzo - I think that is a very good idea - see #2662.

[blankenberg]

Thanks for the work on this all.