Job metadata scripts should not run in conda environment

[rekado]

The scripts to handle job metadata should not run in a conda environment. Currently __handle_metadata builds a command that runs inside a conda environment by passing resolve_metadata_dependencies as True (as that's what job_wrapper.commands_in_new_shell evaluates to).

This is a problem because the job might need to run in an environment in which Python is available. Running the metadata handler inside of this environment would make the scripts use the Python variant from the job's conda environment, including its PYTHONPATH. This causes the metadata scripts to fail unless Galaxy's Python dependencies are installed in the job's conda environment (but they should not).

Setting resolve_metadata_dependencies = False in __handle_metadata of lib/galaxy/jobs/command_factory.py fixes the behaviour in this case as the job itself continues to run in the conda environment while the metadata scripts run in the server's environment (where Galaxy dependencies are available).

[dyusuf]

@bgruening this is the issue about the environment conflict involving conda which I reported to you earlier.

[mvdbeek]

Setting resolve_metadata_dependencies = False in __handle_metadata of lib/galaxy/jobs/command_factory.py fixes the behaviour in this case as the job itself continues to run in the conda environment while the metadata scripts run in the server's environment (where Galaxy dependencies are available).

What version of galaxy are you using?
Your solution is fine, but I would prefer to be able to resolve tool requirements for metadata tools as well if possible (that's samtools and bcftools for the moment).
Does placing

 <env file="<path_to_galaxys_venv>/bin/activate"/>

in the job_conf.xml's destination section resolve the issue? In that case we could rework the default job_script to do this.
If it doesn't, we could also manipulate conda's PYTHONPATH so that galaxy's PYTHONPATH setting takes precedence. This would at least allow resolution of metadata dependencies that are not python dependencies.