Method for outputting environment changes conda is making for use with global installs

[stevekm]

As per the discussion here: #9152

It is getting difficult to use conda in a non-interactive method, such as in a Docker container that needs to have conda-installed tools available.

As per the docs, it is required to run conda activate in order to activate a conda environment

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment

Activating environments is essential to making the software in the environments work well. Activation entails two primary functions: adding entries to PATH for the environment and running any activation scripts that the environment may contain. These activation scripts are how packages can set arbitrary environment variables that may be necessary for their operation. You can also use the config API to set environment variables.

Previously when using conda, you did not actually have to run conda activate, you could simply make all the conda-installed software available by updating the PATH yourself:

Dockerfile:

FROM ubuntu:16.04

RUN apt-get update && \
apt-get install -y wget \
bzip2

RUN wget https://repo.continuum.io/miniconda/Miniconda3-4.3.31-Linux-x86_64.sh && \
bash Miniconda3-4.3.31-Linux-x86_64.sh -b -p /conda && \
rm -f Miniconda3-4.3.31-Linux-x86_64.sh
ENV PATH="/conda/bin:${PATH}"

RUN conda install -y \
-c r \
-c conda-forge \
r-base=3.5.1 \
r=3.5.1 \
r-binom=1.1_1

ENV LC_ALL="C.UTF-8"
ENV LANG="C.UTF-8"

When using a container like this, you do not need to ever invoke the conda commands to use the installed R and R libraries, they are available from the PATH update inside the container. This kind of deployment of conda-installed libraries has been an essential functionality for a long time.

Similarly, you could do this with a Makefile for really great results;

SHELL:=/bin/bash
UNAME:=$(shell uname)
PATH:=$(CURDIR)/conda/bin:$(PATH)
unexport PYTHONPATH
unexport PYTHONHOME

# install versions of conda for Mac or Linux
ifeq ($(UNAME), Darwin)
CONDASH:=Miniconda3-4.5.4-MacOSX-x86_64.sh
endif

ifeq ($(UNAME), Linux)
CONDASH:=Miniconda3-4.5.4-Linux-x86_64.sh
endif

CONDAURL:=https://repo.continuum.io/miniconda/$(CONDASH)
conda:
	@echo ">>> Setting up conda..."
	@wget "$(CONDAURL)" && \
	bash "$(CONDASH)" -b -p conda && \
	rm -f "$(CONDASH)"

install: conda
	conda install -y -c anaconda \
	django=2.1.5 \
	pandas=0.23.4 \
	'xlrd>=0.9.0' \
	gunicorn=19.9.0
	pip install django-ipware==2.1.0

run:
	my_app.py

Here, using make install followed by make run will install and run your program without ever having to interact directly with conda. This works really, really well and makes life so much easier for having a simple deployment & delivery system for your projects.

The discussion here touches on how the current changes to conda's activation methods (requiring the usage of conda activate) are making this kind of usage of conda difficult:

#9152

The issue is also referenced here:

#8586 (comment)

The ability to use conda-installed software and libraries without ever actually having to invoke conda is extremely important. It was suggested to use conda run, but this is not a good solution because you should not have to embed references to conda in your command invocation. What if you need to be able to access the same software regardless of if its installed inside conda, Docker, or on bare metal? Using conda run <some_binary> needlessly couples your software's invocation to conda.

Here is a concrete example: I have a very large workflow that needs a lot of software. A lot of that software is installed with conda, in containers. But it could also be installed globally on the server. Or it could be inside conda, without containers. Or it could be in a container without conda. It doesnt matter, because my command invocation will always be:

<some_binary> <my_args>

Suddenly forcing that to be

conda run <some_binary> <my_args>

is going to pointlessly limit the places and methods by which I can use my programs that run with software that may or may not be installed with conda. And it will force the execution of my workflow to be tied to conda when it was never meant to be. In most cases, conda is never even in my PATH at the time of execution. I was using conda because it was more convenient, but now with this kind of change it will actually be less convenient to use and support conda.

I understand that part of the complexity with this comes from the need for conda to run 'activate' scripts to completely initialize the environment. Some of them appear to be described here:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#macos-and-linux

My suggestion/request is that there be some method to output to the end user all of the changes that conda is making when you run conda activate, so that these changes can be saved statically in order to replicate those changes without actually having to run conda activate. For example, if conda activate is modulating a number of environment variables, then it would be great for me to be able to see what things it is changing and what their final values should be. As suggested in the other Issue comments, this can be done manually by parsing the output of env; it would be really great if we could just have conda do this itself and spit out the list of updated env variables, so that I could just copy/paste that into my Dockerfile or Makefile.

One way I could have mitigated this issue is to just only use older versions of conda that do not have the restriction of requiring interactive activation every time. However, it seems that this is not actually a good solution either. So now, users like me who have made conda a cornerstone of their software-deployment methodology, are now stuck in the situation of having to choose between using old, unsupported, conda versions that have the feature we need but are not future proof, trying to hack together a manual solution ourselves, or just dropping conda support all together.

Having conda spit out a list of env changes that need to be made in lieu of conda activate would be an easy solution for users, copy/pasting your env variables into your static Dockerfile/Makefile is not terrible. The end-result goal being that you dont have to ever actually call conda or interact with it in order to use its installed software.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!