Conda update failing on git bash (libmamba-solver issue)

[LtDan33]

What happened?

GitHub actions is experiencing an error which is shown in the following:

> conda update -n base -q conda

# restart shell (start next task in CI)

> conda install python=3.8 pip --file requirements.txt --file requirements-extra.txt
PluginError: Error while loading conda plugins from entrypoints: DLL load failed while importing bindings: The specified module could not be found. 

You can see this error message here.

This seems to be due to GitHub actions using git bash as the issue has been reproduced on a Windows vm running git bash. There are current efforts to debug and probable causes with this but will keep that contained to messages in this issue.

Conda Info

conda 22.11 + conda-libmamba-solver 22.12 + Windows + Git Bash

[kenodegard]

This appears to be a Windows + Git Bash specific issue. Two possible workarounds have been identified:

1. Don't use Git Bash with conda. For GitHub Action Windows jobs don't use bash as the shell, instead, use the default powershell or cmd.
2. Ensure the base environment is activated on startup. E.g., conda config --set auto_activate_base true.

[mbargull]

Note that in https://github.com/conda/conda/actions/runs/3643394459/jobs/6154412333 from #12156 tests/test_activate.py::ShellWrapperIntegrationTests::test_cmd_exe_basic_integration also failed with the same error for CMD. So either those are two separate issues triggering the same bug or it is not Bash related.

[wolfv]

It sounds a bit like %PATH% does not contain the folders where libmamba.dll is located?

I think things were changed a bit with the activation logic, but not sure if that really is the problem.

[mbargull]

@wolfv, thanks for pointing out the activation changes!
I wasn't aware of gh-11666 -- that is very likely the cause of this. I haven't reviewed that PR completely yet, but overall its premise is not correct and insufficient tests and docs seem to have let to that decision. I'm a bit surprised the introduced regression didn't cause more havoc yet (probably because our CPython patches inject library search paths).
I'll open a separate issue to track addition of regression tests and reversal/amendment of gh-11666. (We should also prepare a timely patch release for that.)

[kenodegard]

We are aware of the cause of this issue, yes, this appears to be a side effect of removing the prefix injection codes from the shell interfaces. We also do not believe that re-adding the prefix injection is the correct move as the prefix injection caused other more confusing issues to occur elsewhere. Furthermore, relying on the prefix injection (instead of other fixes) means that conda/Python/libmamba cannot be run unless the base environment is activated (i.e., auto_activate_base: false). We ultimately decided to remove the prefix injection codes since including it obfuscated other issues.

The Python package is already patched to correctly inject the necessary DLL paths, ATM we are trying to understand why this patch doesn't already find the libmamba DLL.

[kenodegard]

Python DLL patch: https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0013-Add-CondaEcosystemModifyDllSearchPath.patch

[mbargull]

The Python package is already patched to correctly inject the necessary DLL paths, ATM we are trying to understand why this patch doesn't already find the libmamba DLL.

Generally, we should not rely on the assumption that conda is run with a custom build of an interpreter. There is not need for such assumptions and they only invite possible regressions.

Furthermore, relying on the prefix injection (instead of other fixes) means that conda/Python/libmamba cannot be run unless the base environment is activated

This pretty much goes against the base use and recommendation on how to use Conda environments.
The expectation is to activate an environment prior to running programs inside of it.
Which means running programs without an activated environment does not and is not expected to work in the general case.
The CPython patches are mostly to be lenient in letting certain things still work even if the environment is not activated. But this should not be seen as an endorsement of misuse of Conda environments. As such, conda itself, of all things, should not run in such a way. By not activating an environment you "obfuscate other issues", not in the opposite case.
Whenever we run conda from a shell (via the conda function in POSIX shells, etc. or condabin/conda.bat) whilst a non-base environment is active, we have to ensure conda is run in its own environment fully functional.

I did not fully read the mentioned PR or the issues/discussions attached to it.
What I grasped is that there was some issue with conda run leaving base on PATH or something? This is, of course, not intended and a bug.
However, removing the activation of base for conda's own operations because of that is a "throw the baby out with the bath water" case.
How conda run is supposed to work (if called with base not being the activated environment), is that it should put base on PATH first to let conda function correctly (no matter with what patched/non-patched interpreter), and remove it from PATH again when the target command is then run in its own (then activated) environment .

---

What I am saying is not that the previous prefix injection code was flawless (obviously it caused issues).
But it served an important role to ensure conda's own correct operation. Without it you'll run into major regressions again again, I am certain.
We should not and cannot rely on library path injections in some patched interpreter, not only because it is some assumption on something outside of conda's own code base, but also because adjustments to the library path of this one interpreter are only part of the things needed to ensure programs (including conda itself) run correctly.
To stress: Proper environment activation, especially for the base one, is a conceptually important for Conda overall.

[kenodegard]

Furthermore, relying on the prefix injection (instead of other fixes) means that conda/Python/libmamba cannot be run unless the base environment is activated

This pretty much goes against the base use and recommendation on how to use Conda environments. The expectation is to activate an environment prior to running programs inside of it.

I would argue that when auto_activate_base: false is set, then the base environment is not activated on shell login (only $CONDA_PREFIX/condabin appears on $PATH) and yet we still need and expect conda to run correctly.

---

Having slept on this, I agree that we shouldn't care how Python is patched or packaged. You're right that this is just asking for trouble, especially if we ever want to rerelease via pip.

IMO the issue with the previous prefix injection was how it relied on the shell interface. I'm wondering if adding os.add_dll_directory(...) in conda/__init__.py is better?

What's confusing is that this is effectively what the Python patch already does so why can't we find libmambapy.bindings?

[dholth]

I would also be generally in favor of reverting the prefix code given the impact.

[Honghe]

This appears to be a Windows + Git Bash specific issue. Two possible workarounds have been identified:

1. Don't use Git Bash with conda. For GitHub Action Windows jobs don't use bash as the shell, instead, use the default powershell or cmd.
2. Ensure the base environment is activated on startup. E.g., conda config --set auto_activate_base true.

In Windows 10.
After upgraded to conda 22.11, If keep the setting conda config --set auto_activate_base false, I have to add C:\Users\honhe\miniconda3\Library\bin(which include the libmamba.dll) to the env, otherwise will get the following error too:

PS C:\Users\h> conda env list
PluginError: Error while loading conda plugins from entrypoints: DLL load failed while importing bindings: The specified module could not be found. 

[kenodegard]

I am unable to replicate this issue anymore on Windows, likely because of the Python package fix that was introduced and described here

[travishathaway]

We believe this issue to be resolved at this time 👍 .

[hannsta]

Just ran into this issue installing into Windows. Was able to fix by adding the ..\miniforge3\Library\bin to my environment PATH.