Discussion: (micro)mamba should always be in PATH

[jonashaag]

(This is another example of why I suggest to get rid of shell activation/integration entirely)

If you install Micromamba through http://micro.mamba.pm/install.sh it will be installed to ~/.local/bin/micromamba. Not everyone has ~/.local/bin on their PATH and Micromamba is fine with this since micromamba foo in the shell will not call the micromamba binary but the micromamba shell function instead.

However if you use any other tool that wants to call Micromamba as a subprocess, eg. pre-commit, this will fail since Micromamba is not on the PATH. To users this is really confusing because if you don't know about the shell function you'd think that the micromamba binary works fine, while it doesn't! (Only the shell function works, which has its own way to find the binary.)

This also applies to Mamba.

I think we should make sure that (micro)mamba is always on the PATH. Maybe we can emit a warning in the shell function if it isn't.

[jonashaag]

fyi @pavelzw

[pavelzw]

Does the shell function or the binary on PATH take priority if the shell function exists and the binary is on PATH?

[xhochy]

The shell function takes priority.

[pavelzw]

Also, I think the current shell initialization could be simplified further. We could just put eval "$(/path/to/micromamba shell hook)" in the bashrc instead of the whole shell initialization block thing. Then we could also skip the creation of the $MAMBA_ROOT_PREFIX/etc/profile.d files since they are not needed.

At least for bash, zsh, ...
cmd.exe is probably another beast

[AntoinePrv]

In fact I don't think there is a user directory which is always gonna be on the Path on Unix, so we should add it to the Path in the initialization script.

Not sure how I feel about automatically adding ~/.local/bin. That would be my setup, but it seem to be too general of a location for Mamba to add it to the Path itself. What do you think?

[allefeld]

Is there any way to get rid of this intricate shell magic entirely? I find it highly opaque and thereby irritating and I am not aware of any other software which does such things. Adding ~/.local/bin to PATH should be all the configuration necessary to use mamba, or at least the minimalist micromamba.

If I understand correctly, the reason for this shell magic is that subprocesses cannot modify the parent shell's environment. Couldn't this be solved by making the primary executable a shell script, which takes care of activation / deactivation itself, but delegates everything else to the binary? Sorry if I'm completely clueless...

[jonashaag]

@allefeld that's exactly how it's implemented, except that the shell script isn't an executable but a shell function

Try https://github.com/prefix-dev/pixi it doesn't have these problems. But it is not fully compatible.

[AntoinePrv]

@jonashaag I think the installation location of micromamba and PATH modification should be handled in https://github.com/mamba-org/micromamba-releases, do you agree?

@allefeld As Jonas pointed out, you are pointing to the right limitation and way it is handled.
You can also use micromamba without any activation shenanigans, and only rely on commands such as micromamba run -n myenv some-command or micormamba shell -n myenv (starts a new shell, mostly similar to activating, except for command history...).

[allefeld]

I realize now that, contrary to my assumption, a shell script cannot change it's parent's environment either – so I'm guessing there's no way around the shell function in the general case.

Good to know about run and shell, thanks! Maybe I'll use those them.

Do you happen to know how VS Code handles conda / mamba / micromamba environments? Would it e.g. be a good idea to use micromamba run -n myenv code?

[allefeld]

PS: shell does not seem to work properly. If I use micromamba shell -n std, where std is an existing micromamba environment containing Python, I get a prompt indicating I'm in the base environment, and which python3 returns /usr/bin/python3.

If I activate the environment, or use micromamba run -n std bash, the prompt indicates std and which python3 returns /home/ca/micromamba/envs/std/bin/python3.

[jonashaag]

I think there is a bug in 1.4.9 with shell, fixed in master.

I think it should be fine to run VS Code like this. We run it from a Conda env in a complicated setup and it works fine

[jonashaag]

@jonashaag I think the installation location of micromamba and PATH modification should be handled in https://github.com/mamba-org/micromamba-releases, do you agree?

Agreed, although then we should change the activation as follows:

• Remove setting MAMBA_EXE from

  # (Note this example is for fish shell)
  # >>> mamba initialize >>>
  # !! Contents within this block are managed by 'mamba init' !!
  set -gx MAMBA_EXE "/opt/homebrew/bin/micromamba"
  set -gx MAMBA_ROOT_PREFIX "/Users/j/micromamba"
  $MAMBA_EXE shell hook --shell fish --root-prefix $MAMBA_ROOT_PREFIX | source
  # <<< mamba initialize <<<
  

  and just use command micromamba shell hook, or at least make it set -gx MAMBA_EXE (which micromamba) if necessary
• Remove the hardcoded path to micromamba from shell hook, replace with command micromamba as well

[AntoinePrv]

• I think we should keep (reading if defined) the MAMBA_EXE. It's useful for debugging and in case some user messes up their install.
• Well then MAMBA_EXE would serve no purpose.

But my main concerns are ⚠️ this breaks all micromamba installation ⚠️ that do no have micromamba in their PATH (a lot), and I am not sure what problem this solves.

[jonashaag]

I'll try to come up with a well thought through "RFC" that takes into account backwards compatibility.

[MartinJepsen]

The shell hook stuff also makes it super annoying to implement conda, mamba and micromamba in Docker containers.

We have a CI runner, where, to set up micromamba, we manually add a .bashrc file with the shell hook defined, after we've installed micromamba. Simply having micromamba shell init .... in the Dockerfile doesn't work, because each new layer in the image has its own shell (at least that's how I understand it)

[AntoinePrv]

@jonashaag main is now breaking with 1.x, and the next release from main would be 2.0 releases, so now can be a good time to give it a shot.

[iamthebot]

Some thoughts:

We've been building a high level CLI that calls mamba and conda under the hood and dealt with this shell magic a bit too much :) We ended up copying the existing mamba/conda approach with the init block that sources a script that provides logic to update env variables.

It gets especially tricky if running in an environment that's not actually running a shell (eg; as a docker container) or if the shell keeps changing (eg; when building a docker image).

To that end, it would be worth having an alternative activation/deactivation paradigm that relies on a fixed location being added to PATH and LD_LIBRARY_PATH but symlinks in that path are updated with activation/deactivation. This then doesn't rely on shell hackery at all (so even if .zshrc/.bashrc aren't sourced the env is still activated). The downside being that multiple envs cannot be activated at once. The situation gets more complicated for non-python environments like R however (since some things do have to be env variables).

I've looked into the approach pixi uses and while we do something similar for invoking conda/mamba right now (running in a PTY and forking+mirroring) there are gotchas to think about when running in a pty if the intent is to have activate effectively replace the current shell with one running in a pty. But I suspect these are gotchas that are solveable.

[aaronshenhao]

The shell hook stuff also makes it super annoying to implement conda, mamba and micromamba in Docker containers.

We have a CI runner, where, to set up micromamba, we manually add a .bashrc file with the shell hook defined, after we've installed micromamba. Simply having micromamba shell init .... in the Dockerfile doesn't work, because each new layer in the image has its own shell (at least that's how I understand it)

I am encountering the exact same problem trying to run mamba in a Dockerfile. It's frustrating how opaque this mechanism is. Mamba recommends installing via Miniforge in the docs, and Miniforge's README states that mamba should be available in the system path. Yet this didn't happen, and there isn't any documentation describing the install process. I couldn't even tell where it was installed (apparently /root).

Luckily, I found a StackOverflow answer here that provides an elegant solution.

I believe this is a problem that could be easily rectified if it was documented properly in the docs. Perhaps the Miniforge install script could add a Docker option which adds a symlink to /usr/local/bin.

I also tried running bash /path/to/mamba/bin/mamba init after installing in RUN but obtained the following error:

146.6 /opt/miniforge3/bin/mamba: line 4: import: command not found
146.6 /opt/miniforge3/bin/mamba: line 5: import: command not found
146.6 /opt/miniforge3/bin/mamba: line 7: from: command not found
146.6 /opt/miniforge3/bin/mamba: line 10: syntax error near unexpected token (' 146.6 /opt/miniforge3/bin/mamba: line 10: sys.argv[0] = re.sub(r'(-script.pyw?|.exe)?$', '', sys.argv[0])'

Dockerfile:

FROM ubuntu:jammy
WORKDIR /workspace
RUN apt-get update && \
    curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" && \
    bash Miniforge3-$(uname)-$(uname -m).sh -bfp /opt/miniforge3 && \
    rm Miniforge3-$(uname)-$(uname -m).sh &&\
    bash /opt/miniforge3/bin/mamba init

[maphew]

I'm here after several hours spread over a few days trying to troubleshoot problems installing and configuring micromamba. Much of that time was spent spinning because I did not understand that typing micromamba in a terminal will call and use different things based on os and shell environment conditions. I did read docs and discussion on initializing and hooking shells, but it took some time and repeated experiments for me to internalize what that means. I've not worked with any other programs that do this.

This leads me wonder: why does micromamba refer to both the executable and the shell function? is it essential they be the same word? I think would have understood the situation much sooner if, for example, micromamba called the exe and mba called the function.

[AntoinePrv]

The reason why micromamba needs a wrapper function is for modifying the current shell as explained in this issue.

Your idea is worth considering, but note that giving it a different name would have the opposite of what OP advocates for: seamless fallback from shell function to executable.