Allow management of activate.d environment variables in environment.yml files

[pgunn]

This is a new feature proposal, to extend the format for the yml files usable to create conda environments, to allow expression of environment variables (with values) to be statically set. I propose:

Another toplevel in that file titled environment, with k=v elements under it being parsed as environment variables to set in the environment. Example:

my_tensorflow_environment.yml
channels:

• defaults
  dependencies:
• keras
• tensorflow
• python=3.6
  environment:
• KERAS_BACKEND=tensorflow
• AWS_KEY=a32131231231
• AWS_SECRET=a123123123123

When parsing this via "conda env create -f my_tensorflow_environment.yml -n tensor", conda would populate $INSTALL/envs/tensor/etc/conda/activate.d/env_vars.sh and deactivate.d/env_vars.sh accordingly.

The purposes of this proposal are:

1. For people making software that's meant to run in a conda environment, to let the environment.yml fully express what's needed for the environment to be set up correctly so we can keep these instructions out of README files, and
2. To bridge a last bit of gap between containers and conda

If people like this proposal I'd be happy to work on it.

[pgunn]

#6681 is a proof-of-concept (that would need some cleanup before people consider landing it). @kalefranz is this a reasonable approach?

[bpben]

I would love to see this implemented. Looks like it's failing some of the CI checks? Is there some way I can help with this?

[pgunn]

I was hoping to hear back from kalefranz before I spent any more time on it, to get a feel for whether they'd accept it if it passed all the tests and stuff, or if they'd prefer a different approach. If you want to dig into the CI stuff that might be mildly helpful, but I don't know if this proposal is a dead end or not.

[StanczakDominik]

I would love to get this implemented - I once asked a question on Stack about the very problem this issue discusses and I keep getting reminders that this is a thing.

I think the CI stuff may not have been due to the code itself, as it looks like a hg clone error:

.
----------------------------- Captured stdout call -----------------------------
Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was: 
Command '['hg', 'clone', 'https://bitbucket.org/asmeurer/conda-build-hg-test', '/tmp/cb/popen-gw1/test_recipe_builds_source_hg_j0/hg_cache/__bitbucket.org_asmeurer_conda-build-hg-test']' returned non-zero exit status 255.

I'm not 100% sure though.

[MarcDuQuesne]

Any update on this?

[wosiu]

Any updates? At lest it would be cool to have LD_LIBRARY_PATH set automatically like:
LD_LIBRARY_PATH="$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"
People do it like this, but extending capabilities of .yml file would be the best imo.

[mingwandroid]

Any updates? At lest it would be cool to have LD_LIBRARY_PATH set automatically like:
LD_LIBRARY_PATH="$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"

@woisu What would setting LD_LIBRARY_PATH achieve?

[pgunn]

@mingwandroid any package you install that includes a .so would benefit from it.

[mingwandroid]

@pgunn, can you explain what you mean? I've seen mention to something being 'cool' and also to some 'benefit'. Can those be defined?

[pgunn]

@mingwandroid Sure. Let's say you're writing a mostly-python package that provides libfoobar.so and some dependent packages include binaries that would like to import libfoobar. LD_LIBRARY_PATH seems a reasonable way to do that.

Common use-case:
https://stackoverflow.com/questions/1099981/why-cant-python-find-shared-objects-that-are-in-directories-in-sys-path

There may be other ways to do that (e.g. rpath), but this is a reasonably common mechanism that a lot of people are familiar with, and it's less complicated and more flexible.

[pgunn]

(I note that if there is no reasonable cross-platform equivalent of LD_LIBRARY_PATH that may be an argument against going down this route, or at least making the packager handle that complexity themselves)

[mingwandroid]

@pgunn, thanks. If you want to use Anaconda DSOs from non-Anaconda executables then LD_LIBRARY_PATH may help with that, but we do not support what I expect is the more common usage (Anaconda exe being run against non-Anaconda DSOs).

In fact, we deliberately knock out the effect of LD_LIBRARY_PATH in all of the Anaconda Distribution. We do this because we used to get bug reports all the time that our software crashed and after sometimes prolonged investigation it wasn't our software, or rather it was our software mixed with someone else's DSOs because LD_LIBRARY_PATH was set (often to something generally pointless anyway such as /usr/lib - I kid you not).

We decided that we didn't want to partake in this, with the unfortunate loss of the occasionally useful developer feature of wanting to "test against these shiny new DSOs with my super bug fix in."

I believe that LD_LIBRARY_PATH is a misfeature of glibc, and DT_RUNPATH an even worse one. The hallowed goal of mixing and matching exes and DSOs is beyond unrealistic unless those binaries have all been built in the same environment.

To whit, all of our DSOs are loaded via RPATH ORIGIN / relative paths and conda-build does not even allow an exectuable or a DSO to be built with anyDT_RUNPATH entries.

[pgunn]

@mingwandroid Fair enough; I don't share your evaluation of LD_LIBRARY_PATH, but it's still a reasonable stance, particularly given the support burden you described.

(my original feature request is much broader than this though, and for some packages I maintain I have parts of our install instructions that I'd love to ditch by having the environment file manage, such as telling Keras to always use Tensorflow and never Theano, or tweaking MKL or OpenBLAS settings)

[mingwandroid]

I'm not opposed to conda managing env vars in a more formalized way at all, I just wanted to make sure people mentioning LD_LIBRARY_PATH were aware of our position on it.

Thanks!

[github-actions]

Hi there, thank you for your contribution to Conda!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue in the latest version of Conda
2. Comment that the issue is still reproducible and include:

• What version of Conda you reproduced the issue on
• What OS and version you reproduced the issue on
• What steps you followed to reproduce the issue

3. It would also be helpful to have the output of the following commands available:

• conda info
• conda config --show-sources
• conda list --show-channel-urls

NOTE: If this issue was closed prematurely, please leave a comment and we will gladly reopen the issue.

In case this issue was originally about a project that is covered by the Anaconda issue tracker (e.g. Anaconda, Miniconda, packages built by Anaconda, Inc. like Anaconda Navigator etc), please reopen the issue there again.

Thanks!

[kostrykin]

Have there been any updates regarding environment variables in environment files?

[theavey]

@kostrykin it works: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#setting-environment-variables

This issue should be closed

[pgunn]

Awesome! Not sure when this was implemented, but this looks great!