numpy ImportError: DLL load failed: The specified module could not be found.

[nitroamos]

Is it possible to install numpy using pip/setup.py from within a conda environment?

Actual Behavior

C:\Users\AAnderson\cmder
λ conda create -n test8 python=3.6
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.3

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: c:\tools\Anaconda3\envs\test8

  added / updated specs:
    - python=3.6


The following NEW packages will be INSTALLED:

    certifi:        2018.11.29-py36_0
    pip:            19.0.1-py36_0
    python:         3.6.8-h9f7ef89_1
    setuptools:     40.7.3-py36_0
    sqlite:         3.26.0-he774522_0
    vc:             14.1-h0510ff6_4
    vs2015_runtime: 14.15.26706-h3a45250_0
    wheel:          0.32.3-py36_0
    wincertstore:   0.2-py36h7fe50ca_0

Proceed ([y]/n)?

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test8
#
# To deactivate an active environment, use
#
#     $ conda deactivate


C:\Users\AAnderson\cmder
λ activate test8

C:\Users\AAnderson\cmder
(test8) λ pip install numpy
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/b5/11/82916e23836a37c0d76babf74a7ca6f7b4fedd0814eaa166aacc2318b87c/numpy-1.16.1-cp36-cp36m-win_amd64.whl
Installing collected packages: numpy
Successfully installed numpy-1.16.1

C:\Users\AAnderson\cmder
(test8) λ python -c "import numpy"
Traceback (most recent call last):
  File "c:\tools\Anaconda3\envs\test8\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
    from . import multiarray
  File "c:\tools\Anaconda3\envs\test8\lib\site-packages\numpy\core\multiarray.py", line 12, in <module>
    from . import overrides
  File "c:\tools\Anaconda3\envs\test8\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\tools\Anaconda3\envs\test8\lib\site-packages\numpy\__init__.py", line 142, in <module>
    from . import core
  File "c:\tools\Anaconda3\envs\test8\lib\site-packages\numpy\core\__init__.py", line 47, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using c:\tools\Anaconda3\envs\test8\python.exe),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: DLL load failed: The specified module could not be found.


C:\Users\AAnderson\cmder
(test8) λ echo %PATH%
c:\tools\Anaconda3\envs\test8;c:\tools\Anaconda3\envs\test8\Library\mingw-w64\bin;c:\tools\Anaconda3\envs\test8\Library\usr\bin;c:\tools\Anaconda3\envs\test8\Library\bin;c:\tools\Anaconda3\envs\test8\Scripts;c:\tools\Anaconda3\envs\test8\bin;C:\Users\AAnderson\cmder\bin;C:\Users\AAnderson\cmder\vendor\bin;C:\Users\AAnderson\cmder\vendor\conemu-maximus5\ConEmu\Scripts;C:\Users\AAnderson\cmder\vendor\conemu-maximus5;C:\Users\AAnderson\cmder\vendor\conemu-maximus5\ConEmu;c:\tools\Anaconda3\Scripts;c:\tools\Anaconda3\;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Docker\Docker\Resources\bin;C:\Python37\Scripts\;C:\Python37\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Microsoft VS Code\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\nodejs\;C:\Program Files\dotnet\;C:\Program Files\OpenSSL\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\AAnderson\AppData\Local\Microsoft\WindowsApps;C:\Users\AAnderson\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\AAnderson\AppData\Roaming\npm;C:\ProgramData\chocolatey\lib\mingw\tools\install\mingw64\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\AAnderson\cmder

Expected Behavior

Trying to install numpy into an anaconda environment (or use a package's python setup.py which has a numpy requirement)

Steps to Reproduce

This used to work on my machine (and I can still load its environment to test it and it's using numpy 1.16.1), but something seems to have changed and now I can't reproduce that (working) environment. I don't remember how I created that environment, but I don't think I did anything beyond python setup.py develop or similar.

I'm not very experienced with development on Windows, so I don't know much about how to debug dll issues.

I uninstalled anaconda this morning, and then reinstalled using chocolatey, but that didn't resolve the issue.

I get the same problem if I do this (i.e., without the python=3.6):

(test8) λ conda create -n test10
(test8) λ activate test10
(test10) λ python -c "import numpy"
...
ImportError: DLL load failed: The specified module could not be found.

and it also fails if I use python=3.7.
I also tried uninstalling numpy from the base environment but that didn't work.

although this does work:

C:\Users\AAnderson\cmder
(test10) λ activate base

C:\Users\AAnderson\cmder
(base) λ python -c "import numpy"

C:\Users\AAnderson\cmder

I also tried this:

(test12) λ conda create -n test13 python=3.6 numpy
(test12) λ activate test13
(test13) λ python -c "import numpy"
(test13) λ

and that worked too (although it gave me numpy 1.15.4 instead of 1.16.1, not that I care about which version).

Anaconda or Miniconda version:

4.5.12

This is what's provided by default from chocolatey right now. Last night I tried upgrading to the most recent anaconda and it had the same issue.

Operating System:

see below

conda info

C:\Users\AAnderson\cmder
(test8) λ conda info

     active environment : test8
    active env location : c:\tools\Anaconda3\envs\test8
            shell level : 1
       user config file : C:\Users\AAnderson\.condarc
 populated config files : C:\Users\AAnderson\.condarc
          conda version : 4.5.12
    conda-build version : 3.17.6
         python version : 3.7.1.final.0
       base environment : c:\tools\Anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : c:\tools\Anaconda3\pkgs
                          C:\Users\AAnderson\AppData\Local\conda\conda\pkgs
       envs directories : c:\tools\Anaconda3\envs
                          C:\Users\AAnderson\AppData\Local\conda\conda\envs
                          C:\Users\AAnderson\.conda\envs
               platform : win-64
             user-agent : conda/4.5.12 requests/2.21.0 CPython/3.7.1 Windows/10 Windows/10.0.17134
          administrator : False
             netrc file : None
           offline mode : False

conda list --show-channel-urls

C:\Users\AAnderson\cmder
(test8) λ conda list --show-channel-urls
# packages in environment at c:\tools\Anaconda3\envs\test8:
#
# Name                    Version                   Build  Channel
certifi                   2018.11.29               py36_0    defaults
numpy                     1.16.1                    <pip>
pip                       19.0.1                   py36_0    defaults
python                    3.6.8                h9f7ef89_1    defaults
setuptools                40.7.3                   py36_0    defaults
sqlite                    3.26.0               he774522_0    defaults
vc                        14.1                 h0510ff6_4    defaults
vs2015_runtime            14.15.26706          h3a45250_0    defaults
wheel                     0.32.3                   py36_0    defaults
wincertstore              0.2              py36h7fe50ca_0    defaults

[msarahan]

we believe you're seeing this because of a patch to python that we added to make library finding work better. While it helps use of our conda packages, it appears that it may be breaking things installed via pip. You can downgrade to an earlier build of python to avoid the issue:

conda install python=3.6.8=h9f7ef89_0

or stick to conda-installed numpy instead of pip-installed numpy.

[nitroamos]

That addressed the issue:

(test8) λ python -c "import numpy"
...
ImportError: DLL load failed: The specified module could not be found.

(test8) λ conda install python=3.6.8=h9f7ef89_0
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.3

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: c:\tools\Anaconda3\envs\test8

  added / updated specs:
    - python==3.6.8=h9f7ef89_0


The following packages will be DOWNGRADED:

    python: 3.6.8-h9f7ef89_1 --> 3.6.8-h9f7ef89_0

Proceed ([y]/n)?

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

(test8) λ python -c "import numpy"
(test8) λ

thanks!

[mattip]

@msarahan could you provide more information? We are getting quite a few reports on the numpy issue tracker about this

[msarahan]

The latest numpy builds include this patch, which is related to stuff that you might recognize as something that the numpy team tried at some point: https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/0020-Add-CondaEcosystemModifyDllSearchPath.patch

Are these test environments being activated before running python in them? If people are running python using a fixed path without activating, then the PATH stuff won't work right.

[mattip]

we removed this after discovering it broke other peoples stuff :(

[mingwandroid]

I saw your patches after I'd written mine. For sure these are not really functions that an individual package should call but are functions a distro or a language implementation may find it best or necessary to call.

Your patch didn't respect PATH though which is maybe why it broke so much other stuff. I take care to present the entries in PATH to AddDllDirectory() in the right order.

I believe this patch (once reviewed and improved further) would ideally be accepted upstream by cpython (or that MS would just add RPATH and ORIGIN to their loader already).

I need to look into a few reported issues but my hope is that this patch will work out in the end.

We nearly pulled the packages tonight but held our nerve and found a bug in the CI setup of the other project that this change merely unmasked.

Pulling them could still happen but so far I think these patches have worked better than I expected if I'm being honest.

DLL-hell is IMHO getting worse. When DLLs with generic names from modern projects like LibreSSL turned up in C:\Windows\System32 recently from an attempt to configure a new installation for development it was the last straw. We need to do something about DLL-hell.

[mingwandroid]

Playing around I came up with this. Edit __config__.py and changed:

if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):
    os.environ.setdefault('PATH', '')
    os.environ['PATH'] += os.pathsep + extra_dll_dir

to:

if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):
    os.environ.setdefault('PATH', '')
    os.environ['PATH'] += os.pathsep + extra_dll_dir

    from ctypes import windll, c_wchar_p
    from ctypes.wintypes import DWORD, HMODULE
    AddDllDirectory = windll.kernel32.AddDllDirectory
    AddDllDirectory.restype = DWORD
    AddDllDirectory.argtypes = [c_wchar_p]
    AddDllDirectory(extra_dll_dir)

so we are calling AddDllDirectory() once more (which puts this directory to the front of the list). Then this should work again. The issue is that numpy relies on PATH being in dynamic control still, which isn't the case after my patch. I'm not sure where abouts in the numpy codebase to propose a patch. If we must only do this when running Anaconda Python then I'm ok with that. Would it be possible/beneficial to come up with an env var to control whether to use AddDllDirectory or not?

[mattip]

The code lives in 'numpy/distutils/misc_utils.py' for obscure historical reasons. Could you submit a PR to numpy with your proposed changes?

[mattip]

.. and I think having code to detect whether this is needed or not would be most beneficial, since the last time we tried this it broke other packages so we removed it. Maybe you have insights that will make the idea more robust

[mingwandroid]

Could you submit a PR to numpy with your proposed changes?

Sure, we should be using GetProcAddress to check that SetDefaultDllDirectory() and AddDllDirectory() are in the running system's kernel32.dll

last time we tried this it broke other packages so we removed it.

The difference between your old approach and what we're looking at is that you change Windows search order from:

1. dirname(dll)
2. dirname(exe)
3. C:\Windows\System32
4. %PATH%

to:

1. dirname(dll)
2. dirname(exe)
3. numpy\.libs

.. so it's not really a surprise that this breaks other things.

With my patch, our Python now uses the following order:

1. dirname(dll)
2. dirname(exe)
3. %PATH%, but copied at python startup-time into the internals of Windows via `AddDllDirectory()`

.. which is why you can not rely on PATH itself anymore. I'll work on a PR as soon as I can.

[mattip]

If you stick with using AddDllDirectory(), won't all packages that used to modify os.environ['PATH'] need to change their code? If so I would suggest that Anaconda intercept the setter for os.environ or the ImportError and warn otherwise random things will break.

[mattip]

Maybe it would be better to use a resource file as suggested near the end of https://bugs.python.org/issue35688

[mingwandroid]

If you stick with using AddDllDirectory(), won't all packages that used to modify os.environ['PATH'] need to change their code? If so I would suggest that Anaconda intercept the setter for os.environ or the ImportError and warn otherwise random things will break.

I agree we probably should intercept this. At what level would you recommend we do that? A high level monkey-patch to the python code or something lower level? In our latest iteration we resync from PATH to AddDllDirectory() just before each call to LoadLibraryEx().

Maybe it would be better to use a resource file as suggested near the end of https://bugs.python.org/issue35688

It might well be better @mattip. A manifest with relative paths would solve a lot of this and would be more clean than what I've done so far, so long as it avoids searching early in C:\Windows\System32.

I seem to have things 'working' now for the failures I've encountered so far, albeit in a messy patch with a lot of tweakables. I'll reply when I've uploaded some packages.

Thanks for the discussion so far. Bringing this to the attention of @katietz and @msarahan.

[sigbjorn]

Just to confirm the discussion and issue here, yes, it did break existing code on win platform.
Setting PATH prior to starting python resolved the issue (not an option, but just nice to know).
Setting os.environ PATH, does not help, even if it is correct, the 3rd party dependencies that are triggered load-dll after .pyd is loaded failes utterly.

[mingwandroid]

My latest version resyncs AddDllDirectory with PATH just before Python calls LoadLibraryEx. The channels these are on are c3i_test and conda-canary. New builds of Python 2.7 have just been uploaded, Python 3.6 and 3.7 will follow later.

[mattip]

This all seems a bit magical. I assume this the final version?

• If something goes wrong, how will users be able to debug it? Can they access the current search path after a failed LoadLibraryEx?
• Documentation is critical. Until now, many users have found dpends.exe to be the only way to replay DLL loading, but now the path is internal to python, so there should be a "best-practices" document stating what is being done and how to debug it.
• How will this play out for third-party dll loaders like CFFI or Cythonized code? If the import order is changed, will LoadLibraryEx now behave differently for them, with no way of knowing what the actual search path is?

[mattip]

DLL-hell is IMHO getting worse. When DLLs with generic names from modern projects like LibreSSL turned up in C:\Windows\System32 recently from an attempt to configure a new installation for development it was the last straw. We need to do something about DLL-hell.

I hope this far-reaching change is being discussed in a wider forum than just the commenters on this issue.

[mingwandroid]

This all seems a bit magical. I assume this the final version?

No it isn't the final version (code's never finished). The latest can be seen at https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/0020-Add-CondaEcosystemModifyDllSearchPath.patch

We might end up using a manifest with relative paths in it for some of this patch,

We attempt here to mirror as much as we can the normal DLL lookup procedure, just without C:\Windows\System32 appearing before PATH (classic DLL-hell).

• If something goes wrong, how will users be able to debug it? Can they access the current search path after a failed LoadLibraryEx?

You need to set an env var at present.

• Documentation is critical. Until now, many users have found dpends.exe to be the only way to replay DLL loading, but now the path is internal to python, so there should be a "best-practices" document stating what is being done and how to debug it.

Agreed we'll document it.

• How will this play out for third-party dll loaders like CFFI or Cythonized code? If the import order is changed, will LoadLibraryEx now behave differently for them, with no way of knowing what the actual search path is?

We have to make a judgement call about which is worse, DLL-hell or trouble for 3rd party loaders (which we'll attempt to erradicate when found).

[mattip]

Thanks. I am curious, is there currently a "python interpreter differences between cpython and anaconda python" documentation page? I didn't know Anaconda patches stock python, and a quick google search was not productive.

[mingwandroid]

For sure we patch stock python (as software distros must do for many packages), you are welcome to look at the patches. I linked you to one, the others are in the same place.

[rgommers]

conda install python=3.6.8=h9f7ef89_0

@msarahan what is the equivalent py37 version to use to avoid this problem?

or stick to conda-installed numpy instead of pip-installed numpy.

That's always a healthy idea. A big part of the problem here will be caused by NumPy in defaults being out of date, which causes pip install -U somepackage-that-depends-on-numpy to replace the conda-installed numpy 1.15.4 with pip-install 1.16.1.

It's been 5 weeks since 1.16.0 came out, and 3 weeks since 1.16.1. What is the problem exactly (did we make your life more difficult in 1.16.x somehow?), and do you have an ETA?

[zooba]

FYI, I just filed https://bugs.python.org/issue36085 to potentially make the painful breaking change in CPython 3.8 so that we can manage the porting process at the right version boundary. Any and all input welcome!

[zooba]

We're probably going to disable this by default though and advise people who run into DLL-hell to set an env. var. We might bring it back again in future though, hopefully through discussion on bugs.python.org. We did run the patch by the Windows Python maintainer who wasn't completely against us using it in Anaconda Distribution.

Aw man, you mean I predicted the problems and the solution and you didn't want to mention me to let me know ;)

Assuming you're getting away with the SetDefaultDllDirectories call (which is one of the problems numpy hit), then presumably you can update the conda builds to put all the DLLs in one directory and just AddDllDirectory that one? My concern was that you were adding a whole lot of paths from PATH that weren't relevant, so provided you can make it focused on the paths that matter, you should be okay.

[mingwandroid]

The patch is now working well but we've chickened out (for now). We had to taker care around cwd and some other details. To enable the new dll search you set CONDA_DLL_SEARCH_MODIFICATION_ENABLE env var.

The only problem I've seen with the latest version is that broken dlls that were not found before now do get found. PyTorch suffered this but have removed the broken dlls.

We cannot move dlls..

1. Either technically: We cannot bundle mingw-w64 dlls (the R ecosystem, things that need c99 or c++11 on py2) in the same folder as msvc dlls they often share file names but are not compatible. Same applies for msys2. We use PATH like everyone else because we have to.

2. Ecosystematically. If we move dlls from where they're expected to be found them the expectees break (if they attempt to full path load them). Fixing requires rebuilding the world then.

edit: NOTIFICATION=>MODIFICATION

[mingwandroid]

Until MS implements an alternative dll loading scheme that fixes dll hell once and for all (and I believe it's not that difficult there are two working models from which to copy!) we must reserve the option to enable this by default.

Though we will probably try something with relative paths in manifests too.

[mattip]

Just to be clear: What would Anaconda suggest numpy tell users who pip install tensorflow, which updates numpy to 1.16.1 via pip, then get an error message and open an issue on numpy's issue tracker (something that happens about twice a day now)?

[mingwandroid]

I'd say conda update python

[mingwandroid]

I'd also say they can set CONDA_DLL_SEARCH_MODIFICATION_ENABLE (which enables this new stuff) and that should also work fine since we fixed the bug you refer to already.

[CAM-Gerlach]

@mingwandroid FYI, the environment variable is misspelled with NOTIFICATION instead of MODIFICATION in your comment above, FYI. Is there any authoritative documentation on this we can point people to other than a few scattered comments on this thread? Thanks.

[msarahan]

https://docs.conda.io/projects/conda/en/latest/user-guide/troubleshooting.html#numpy-mkl-library-load-failed

Also see the SSL issue immediately before that topic.

[mingwandroid]

Only Python builds beyond these builds will react to these environment variables:

    Python 2.7.15 build 14
    Python 3.6.8 build 7
    Python 3.7.2 build 8

We should add something like "please note these fixes are not available in python 3.8 yet."

[CAM-Gerlach]

Thanks @msarahan and @mingwandroid . I couldn't find it on Google with various queries; I had to manually go to one of the original issues we were involved in. I'd suggest broadening the title/description in the docs to include DLL load failed errors more generally, as can be seen from the linked issues where it has resolved a number of such problems that were a result of DLL hell conflicts. I'd also add GoPro Studio as that was the culprit on my system. I can submit a PR if desired.

[mingwandroid]

PR would be great!

[CAM-Gerlach]

Okay, sure thing. I'll put it on my todo list. Did you mean to reopen this issue in light of that or was that an accident? Thanks.

[mingwandroid]

I was 50:50 on that. I should maybe change the title to say "Misspelling in docs .."?

[CAM-Gerlach]

The misspelling isn't in the docs (which I originally was unable to find when looking up the name of the env var for the user I was helping on spyder-ide/spyder-kernels#89 , thus my question) but rather in this comment above, which in turn was linked elsewhere. I just wanted to make sure others referencing it don't make the same mistake. Then @msarahan linked me to the actual docs and I had some suggestions I offered to help with there for making them more broadly applicable to the class of problems this can fix.

[mingwandroid]

Corrected the comment, please go ahead if you have time.

[owenlamont]

Thanks @mingwandroid - CONDA_DLL_SEARCH_MODIFICATION_ENABLE did the trick - I'd given up on using Anaconda Python outside of Jupyter and PyCharm on Windows because of this Numpy issue. I did notice despite enabling that environment variable if I use VSCode's relatively new "Run Python" button - for launching when you haven't created a launch.json I still get this problem despite enabling the environment variable - If I click Debug Python - or use the context menu to "Run Python file in Terminal I don't have any issues importing Numpy. Likewise if I create a launch.json file with default settings that runs fine too. Does anyone else get that? Should I raise a ticket with the VS Code team?