DLL load error while importing scipy.sparse.linalg module on Windows

[terem42]

My bug report is about DLL load error on running scipy-1.3.2 library for Windows 10 64bit

Reproducing code example:

import scipy.sparse.linalg
print("loaded")

After small research I've identified a culput, there is no PATH env setup for windows platform inside __config__.py of scipy package, or it's not get executed correctly.

if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):
    os.environ.setdefault('PATH', '')
    os.environ['PATH'] += os.pathsep + extra_dll_dir

Henceforth, the path to .libs folder is not added to Windows path and libraries never ever gets found.

Error message:

Traceback (most recent call last):
  File "test_lib.py", line 1, in <module>
    import scipy.sparse.linalg
  File "C:\apps\Python37\lib\site-packages\scipy\sparse\linalg\__init__.py", line 113, in <module>
    from .isolve import *
  File "C:\apps\Python37\lib\site-packages\scipy\sparse\linalg\isolve\__init__.py", line 6, in <module>
    from .iterative import *
  File "C:\apps\Python37\lib\site-packages\scipy\sparse\linalg\isolve\iterative.py", line 10, in <module>
    from . import _iterative
ImportError: DLL load failed

Scipy/Numpy/Python version information:

Python 3.7.5 win64
scipy-1.3.2
numpy-1.17.4

Temporary workaround I'm using for the moment being, is a small update within __init__.py repeating the missing search path addition.

import os
import sys

extra_dll_dir = os.path.join(os.path.dirname(__file__), '.libs')
if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):  
  os.environ['PATH'] = extra_dll_dir + os.pathsep + os.environ['PATH']

[ilayn]

That's because you haven't clicked add python to path box when you are installing python.

Instead add libs to path

[terem42]

Normally this should happen either at config or init lib stage, because I can't know in advance whether library uses DLLs or not. Hence, I propose to add aforementioned path settings to __init__.py

[rgommers]

I think this has never come up, because there are many reasons you'll want to set PATH on Windows (e.g. https://docs.python.org/2/using/windows.html#finding-the-python-executable). Anaconda et al. do this automatically. How did you install Python?

[ilayn]

We can't dictate where or how the python access should reside. I don't think we can do anything about it. It's beyond scipy's jurisdiction.

[clemextravisci]

We have a similar issue happening in scipy.interpolate. Not sure if related, but the content of the .libs folder in 1.3.2 is missing the file msvcp140.dll compared to the extra-dll folder in 1.3.1. Copying the file in the folder fixes the issue.

[ilayn]

That should be fixed in #10628

[clemextravisci]

Just to make sure I understand correctly, a new PR needs to be created to modify the destination folder of the msvcp140.dll file?

Similar to what was done here, but .libs instead of extra-dll:
https://github.com/MacPython/scipy-wheels/pull/50/files#diff-180360612c6b8c4ed830919bbb4dd459R169-R172

So, it can work seamlessly with numpy configuration tool:
https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py#L2329

[ilayn]

After switching to Python3.8 I can confirm its reproducibility. So I was wrong above. This is also affecting the numpy builds. My apologies for the quick and false response. I'll take this to NumPy maintainers.

[pv]

The problem I think is this: https://github.com/MacPython/scipy-wheels/blob/master/appveyor.yml#L167

We used a pinned version of numpy.distutils with the correct Windows patches, but for Python 3.8 this was changed to use Numpy upstream, which has a different name for the DLL directory.

EDIT: somehow also the scipy-1.3.2-cp37-cp37m-win_amd64.whl has libraries in the .libs directory, and not in extra-dll.

[ilayn]

This is also happening on my local NumPy and SciPy builds.

[pv]

So there seem to be now several different issues:

(i) MacPython/scipy-wheels#55 changed the pinned Numpy distutils version, which changed the name of the DLL directory. However, msvcp140.dll copying was not updated to match, so the file now goes to the wrong place. I opened a separate ticket for this: MacPython/scipy-wheels#57

(ii) There is something strange with the behavior of PATH and DLL loading. Those who can test it, could you try give information on whether the

extra_dll_dir = os.path.join(os.path.dirname(__file__), '.libs')

if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):
    os.environ.setdefault('PATH', '')
    os.environ['PATH'] += os.pathsep + extra_dll_dir

fail to change PATH at all, or, whether the issue is some DLL loading problem that for some reason appending the load directory does not work any more, but prepending works.

[ilayn]

isdir is the False branch somehow. My guess is that this is run on the src.win... folder and not in the lib.win... folder during the install. But I am really speculating now.

[pv]

The __config__.py code is run just when importing the installed package (see scipy/__init__.py), not at build or install time, so the contents of the build directory (src.* lib.*) should play no role and only what gets installed (or put into the .whl) should matter.

[ilayn]

That would be my guess too. I'll modify misc_utils manually and try to see the arguments. Maybe isdir() behavior changed.

[pv]

I think you can modify the scipy/__config__.py directly in the installed package (possible to test also by installing the wheel from Pypi), e.g. insert import pdb; pdb.set_trace() and see what it is doing during import scipy.

The original bug report is also for Python 3.7, so I'm not sure behavior changes in 3.8 explain...

[ilayn]

Looks like __bootstrap__ from _iterative is not run. Because that's where the error is generated. But I don't know why yet.

[terem42]

That's because you haven't clicked add python to path box when you are installing python.

Instead add libs to path

Scipy library is installed with pip, hence python installer per se can't add any scipy DLL paths, it happens much later. Also, adding DLL permanently to the Path env variable in M$ Windoze might interfere with other scipy versions, being executed in virtual python environment. So, in my opinion, it would be way better to add .libs folder to the os.environ['PATH'] variable for Windows platform in __init__.py instead of adding them permanently.

[ilayn]

It looks like we have two and a half problems intertwined as @pv mentioned. One for the wheels the naming issue and the other is explained by @mattip in the NumPy side that Py3.8 on Windows changed the way it searches for DLLs.

Hence those will have an additional 3.8 if branch in the same place and the business will be as usual. For the wheels and NumPy side story is a bit more complicated.

[clemextravisci]

If it can help, we were experiencing the issue with Python 3.6.8 on Win64.

[tylerjereddy]

The 1.3.2 wheels should be "ok" for Python 3.8 DLL path handling changes, but the msvcp140.dll being in the wrong place is something I'm working on in: MacPython/scipy-wheels#58 . That problem would indeed still exist for 1.3.2 wheels.

I'll try to backport that fix to 1.3.3 wheels/release once it is in the master wheels repo, but I'm also working on the 1.4.x release process at the same time, so apologies in advance if it takes a little longer!

[rgommers]

NumPy fix in numpy/numpy#14923, should be taken over in the wheels repo I think.

[tylerjereddy]

I don't get it---there's a comment from @ilayn in that PR indicating that this is a blocker for SciPy Python 3.8 wheels release but you let me proceed with the 1.3.3 release? Can we get a test in the wheels repo to produce this failure that persists even with hard-loading of DLLs from _distributor_init.py?

I thought I had this covered with usage of _distributor_init.py loading from .libs directly based on discussion with @mattip & the final cleanup was the stray DLL @pv identified for placement in .libs?

I suppose it might be cleaner to prefer a solution from upstream distutils, but is anything actually broken with the current approach of using _distributor_init.py to just hard-load all the DLLs? And is it really worth more turnover in the DLL handling machinery until after 1.4.0 is out if this is just a refinement in upstream distutils?

[ilayn]

I meant this is a blocker for 1.3.2 already on top of the remaining issues. I encountered it and also as you see above took me a while to verify it after we branched off. Because it specifically hits python 3.8/windows. I think only 3.8 wheels need to be updated and nothing else.

[tylerjereddy]

What's the reproducer for SciPy 1.3.3?

import scipy.sparse.linalg is still failing there? How are the tests passing on wheels CI then? I want to be able to catch the problem if there is one with 1.3.3.

[ilayn]

That's a very good question that I don't have a good answer to. Lately CIs are testing my faith in them.

[tylerjereddy]

Alright, I'll try to test on Windows tonight. If I understand correctly you're saying this would remove the _distributor_init.py requirement, which may be nice, but I don't think a critical issue if the imports/tests still work.

[ilayn]

The actual question I have is that how come this doesn't conflict with the _distributor_init.py mechanism. Seems a bit too good to be true

[pv]

_distributor_init.py and os.add_dll_directory are doing the same thing in two different ways (Things fail on Python 3.8 in the obvious way if you don't do either one. Doing both should also be ok.) One might argue that the add_dll_directory is the better way to do it, as it helps people do their own builds. Not really release blocker IMO, but can be considered for 1.4.0 if you are planning rc2.

As far as this issue is concerned, the remaining question is the original Python 3.7 report for which the reporter did not yet provide requested additional information. It's probably some non-general issue (e.g. badly quoted system PATH value would cause this, etc.) so suggest closing if there's no further clarification.

[mdhaber]

I think I just ran into this when creating a new dev environment with Python 3.8 in Windows. Build goes fine, but when I try to run tests or import a subpackage at the Python prompt, I get the DLL load error. No problem with Python 3.7.

[tylerjereddy]

@mdhaber latest master or something else? I think the latest 1.5.2 wheels should be "ok" at least because of the _distributor_init.py shims.

[mdhaber]

Latest master.

[mdhaber]

Now that Python 3.7 support has been dropped (gh-14655), this is a bigger issue for me, as I can no longer work with SciPy from the master branch.

I upgraded to Python 3.8, git clean -dfx, and rebuilt as usual python setup.py build_ext --inplace. In Python,

>>> import scipy.optimize
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\matth\scipy\scipy\optimize\__init__.py", line 400, in <module>
    from .optimize import *
  File "C:\Users\matth\scipy\scipy\optimize\optimize.py", line 33, in <module>
    from .linesearch import (line_search_wolfe1, line_search_wolfe2,
  File "C:\Users\matth\scipy\scipy\optimize\linesearch.py", line 16, in <module>
    from scipy.optimize import minpack2
ImportError: DLL load failed while importing minpack2: The specified module could not be found.
>>> import scipy.stats
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\matth\scipy\scipy\stats\__init__.py", line 444, in <module>
    from .stats import *
  File "C:\Users\matth\scipy\scipy\stats\stats.py", line 37, in <module>
    from scipy.spatial.distance import cdist
  File "C:\Users\matth\scipy\scipy\spatial\__init__.py", line 98, in <module>
    from .qhull import *
ImportError: DLL load failed while importing qhull: The specified module could not be found.
>>> import scipy.linalg
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\matth\scipy\scipy\linalg\__init__.py", line 195, in <module>
    from .misc import *
  File "C:\Users\matth\scipy\scipy\linalg\misc.py", line 3, in <module>
    from .blas import get_blas_funcs
  File "C:\Users\matth\scipy\scipy\linalg\blas.py", line 213, in <module>
    from scipy.linalg import _fblas
ImportError: DLL load failed while importing _fblas: The specified module could not be found.

Suggestions?

[ilayn]

Did you also install after build? Seems like you are in the github folder and trying to import scipy. It should search in the site-packages or whatever it was called and not in the source folder.

Did you also add the python to the path by the way?

[mdhaber]

I'm using conda, and I use conda develop . from the scipy folder to "install" after the in-place build. I think that should take care of these things. Installing to site-packages is not part of my normal workflow. When I make changes to a SciPy function in the local repo, the updated function is immediately available for testing in my IDE - often even without re-importing the function, much less re-installing SciPy to site-packages.

[ilayn]

Then it should look for it in the conda folders. Here the errors are coming from the source folders. Your workflow might be broken instead of the scipy since 3.8 changed the lookup for DLL mechanism. Can't say anything much about conda though never used it

Obvious alternative is just skip 3.8 and jump to the 3.9 (or even 3.10?) wagon.

[mdhaber]

3.8 changed the lookup for DLL mechanism

I see. Wouldn't the workflow also be broken for everyone who follows the quickerstart guide, the mac quickstart guide, and the ubuntu quickstart guide (or adapts the Windows build instructions accordingly)? All of those use python setup.py develop or python setup.py build_ext --inplace with conda develop .. My understanding is that both of those setup.py commands build in the local repo (not site-packages) and conda develop . does something equivalent to adding the local repo to the python path.

I'll try 3.9/3.10.

[ilayn]

Yeah I can't comment on the conda part but from the silence of this issue for quite some time, something is apparently specific to your use case. But unfortunately have no idea. The main issue when 3.8 came out was the DLL change but then things settled down.

[mdhaber]

Thanks @ilyan. Hopefully someone else can help. I don't think I'm doing anything too unusual. I have conda installed, then my compilers are set up and OpenBLAS built according to the Windows build instructions. Then the following works to set up a new development environment. From the local repo:

conda create -n scipydev37 python=3.7 numpy pytest cython pybind11
<copy openblas.a into ...\envs\scipydev3\Lib>
python setup.py develop (or python setup.py build_ext --inplace)
conda develop .

SciPy builds successfully, and SciPy imports work.

If I change the first line to:

conda create -n scipydev38 python=3.8 numpy pytest cython pybind11

(or a later version of Python), SciPy builds successfully, but I get the DLL load failures when I try to import a subpackage.

[ilayn]

This looks like an ana/mini/conda bug that is still open. I've got many google results about the same problem. Vanilla python is fixed but conda has still issues about this apparently.

[mdhaber]

Huh. Is it worth trying to install python from a different conda channel?

[ilayn]

I don't think so there are many reports of the same issue such as this ContinuumIO/anaconda-issues#12475

[mdhaber]

Thanks for finding that.

[mdhaber]

I thought I might as well try using conda-forge python, because that could be different from conda's default channel python and thus avoid the issue. But that doesn't work either.

[rgommers]

I thought this issue was long since solved. Note that there are two separate ones:

1. the one about extra_dll_dir which we solve by explicitly adding a DLL search path (os.add_dll_directory)
2. an in-place build where those DLLs are spread throughout the source tree, and importing those doesn't work (e.g. linalg.fblas.py does not find linalg/_fblas.so

I believe (1) is solved, but (2) still seems to be a problem (because we didn't do anything to fix it, the changes we did make are specific to (1).

@mdhaber to confirm: you did try CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1, and that made no difference?

[mdhaber]

I didn't know about CONDA_DLL_SEARCH_MODIFICATION_ENABLE. That seems to have fixed the problem.

[rgommers]

@mdhaber can you check that that fixes it for both defaults and conda-forge? I'm not sure the conda-forge Python uses CONDA_DLL_SEARCH_MODIFICATION_ENABLE.

[mdhaber]

I set up two new environments. I installed python=3.9 numpy cython pytest pybind11 pythran from the default channel in one environment and those same packages from conda-forge in the other environment. I did

git clean -dxf
python setup.py develop
python
>>> import scipy.optimize
>>> scipy.optimize.linprog(1)

in each environment, and everything seemed fine. In both environments, I get the same error without CONDA_DLL_SEARCH_MODIFICATION_ENABLE.

Strangely (to me) it doesn't matter whether CONDA_DLL_SEARCH_MODIFICATION_ENABLE=0 or CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1; just defining the environment variable is what makes the difference.

[isuruf]

@mdhaber, can you try installing python39 using -c isuruf/label/python39 -c conda-forge?"

Also is there a way for me to get RDP access to a windows machine with this issue reproduced?

[andyfaff]

I believe this issue can be closed now. We've transitioned to a new build system and we're not getting further comments on this issue. If you think that this issue should be reopened to add further details then please do so.