DLL load failed while importing _speedup with conda+python3.8+windows

[xjh19971]

Hi, I test bezier package with latest version (2020.5.19) under python3.8+anaconda+windows. It seems a compatible error.

Traceback (most recent call last):
  ...
  File "E:\pytorch-PPUU\map_i80.py", line 8, in <module>
    from traffic_gym import Simulator, Car, colours
  File "E:\pytorch-PPUU\traffic_gym.py", line 20, in <module>
    import bezier
  File "E:\PPUU\lib\site-packages\bezier\__init__.py", line 41, in <module>
    __config__.handle_import_error(exc, "_speedup")
  File "E:\PPUU\lib\site-packages\bezier\__config__.py", line 136, in handle_import_error
    raise caught_exc
  File "E:\PPUU\lib\site-packages\bezier\__init__.py", line 37, in <module>
    import bezier._speedup  # noqa: F401
ImportError: DLL load failed while importing _speedup: 找不到指定的模块。

Process finished with exit code 1

[dhermes]

Thanks for reporting! Windows binary extensions are absolutely a weak point of my expertise and I don't have a great way to test these things. Can you run conda list or conda list -n {current_env} to give an idea of what is installed? (I'd like to reproduce the error and dive in a bit to understand what you're seeing.) I'm not sure if you're running 32-bit or 64-bit but I'll assume 64-bit since it's much more common. The packaged DLL is bezier-7be4e22c.dll in 32-bit and bezier-b9fda8dc.dll in 64-bit.

Also, in that same environment could you run a few test commands? The __config__.py file is what is handling the DLL loading, so we'll borrow from there:

>>> import ctypes
>>> dll_name = "bezier-b9fda8dc.dll"  # or "bezier-7be4e22c.dll" in 32-bit
>>> ctypes.cdll.LoadLibrary(dll_name)  # Should fail before adding `extra-dll` to search path
>>> import os
>>> os.add_dll_directory(r"E:\PPUU\lib\site-packages\bezier\extra-dll")
>>> ctypes.cdll.LoadLibrary(dll_name)  # Should succeed

---

PS: 找不到指定的模块 == Cannot find the specified module (for related searches)

[xjh19971]

Thank you for your reply! If you want to reproduce this env, this environment.yaml will help. You can run conda env create -n your_name -f ./environment.yaml to create an env. I am running in a 64-bit environment.

I ran your code, the result is as follows:

>>> import ctypes
>>> dll_name = "bezier-b9fda8dc.dll"
>>> ctypes.cdll.LoadLibrary(dll_name)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "E:\PPUU-test\lib\ctypes\__init__.py", line 451, in LoadLibrary
    return self._dlltype(name)
  File "E:\PPUU-test\lib\ctypes\__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'bezier-b9fda8dc.dll' (or one of its dependencies). Try using the full path with constructor syntax.
>>> import os
>>> os.add_dll_directory(r"E:\PPUU-test\lib\site-packages\bezier\extra-dll")
<AddedDllDirectory('E:\\PPUU-test\\lib\\site-packages\\bezier\\extra-dll')>
>>> ctypes.cdll.LoadLibrary(dll_name)
<CDLL 'bezier-b9fda8dc.dll', handle 6a2c0000 at 0x2e4fa43f040>

[dhermes]

That's good to hear, so it seems the Python binary extension is what's causing issues. Can you try to import bezier._speedup directly (without having to run through bezier/__init__.py):

>>> import sys
>>> sys.path.append(r"E:\PPUU-test\lib\site-packages\bezier")
>>> import _speedup  # Should fail because `bezier._speedup` depends on `bezier-b9fda8dc.dll`
>>> import os
>>> os.add_dll_directory(r"E:\PPUU-test\lib\site-packages\bezier\extra-dll")  # Put `bezier-b9fda8dc.dll` on DLL load path
>>> import _speedup

[xjh19971]

Seems that _speedup is still not loaded after os.add_dll_directory.

>>> import sys
>>> sys.path.append(r"E:\PPUU-test\lib\site-packages\bezier")
>>> import _speedup
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing _speedup: 找不到指定的模块。
>>> import os
>>> os.add_dll_directory(r"E:\PPUU-test\lib\site-packages\bezier\extra-dll")
<AddedDllDirectory('E:\\PPUU-test\\lib\\site-packages\\bezier\\extra-dll')>
>>> import _speedup
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing _speedup: 找不到指定的模块。

[dhermes]

OK, that confirms that _speedup is the problem. I'll have to try to spin up a Windows machine and see what I can find out. Sorry for this issue. In the near term, if you'd like to use bezier without the binary extension (--no-binary=bezier), you can ask pip to install from source (instead of a wheel) and then instruct the bezier install process to skip the extension (via BEZIER_NO_EXTENSION=true):

BEZIER_NO_EXTENSION=true \
  python  -m pip install --upgrade bezier --no-binary=bezier

See: https://bezier.readthedocs.io/en/2020.5.19/#installing

[xjh19971]

Thanks! That really helps. Hope you can get the root of this problem.

[1dataenthusiast]

Can confirm having the same problem with:

• bezier 2020.5.19
• python 3.8.3
• conda 4.8.2
• pip 20.1.1

on a Windows 64-bit machine.
bezier was installed from wheel.

[dhermes]

Another report here as well: #208 (comment). Sorry I haven't had an opportunity to debug this yet, I don't have much Windows ability and my Windows machine is otherwise occupied.

---

In addition to my comment above #237 (comment) about installing without binary extension speedups, here is another:

BEZIER_INSTALL_PREFIX=.../path/to/usr/ \
  python  -m pip install --upgrade bezier --no-binary=bezier

where .../path/to/usr/ is a locally built libbezier (a shared library used by the binary extension speedup). You can install libbezier from Conda forge: https://anaconda.org/conda-forge/libbezier. (I'm not sure what the path it will be installed into is on Windows but can check at a later time.)

[travisbriles]

(I'm new to python so I apologize if this is a silly problem.) This seems to still be throwing an error. I tried using ipython in Anaconda Prompt to paste the multiline code above and it threw a syntax error.

[Frozoto1]

it works for me, just run below code. you should modify your actual path of bezier-b9fda8dc.dll

import ctypes
ctypes.cdll.LoadLibrary(r"C:\Users\oocarain\miniconda3\Lib\site-packages\bezier\extra-dll\bezier-b9fda8dc.dll")

[Peterliwenxuan]

When I install bezier without extensions by using
python -m pip install --upgrade bezier --no-binary=bezier

the terminal sends the error message that:

How to fix this problem
(if I install the full version with extensions, similar problems of ImportError: DLL load failed while importing _speedup: will appear

[dhermes]

@Peterliwenxuan you need to set the BEZIER_NO_EXTENSION=true environment variable as in #237 (comment)

[unlinking]

it works for me, just run below code. you should modify your actual path of bezier-b9fda8dc.dll

import ctypes
ctypes.cdll.LoadLibrary(r"C:\Users\oocarain\miniconda3\Lib\site-packages\bezier\extra-dll\bezier-b9fda8dc.dll")

Same issue, this works for me in Windows.

[unlinking]

def handle_import_error(caught_exc, name):
    expected_msg = TEMPLATE.format(name)
    if caught_exc.args == (expected_msg,) or caught_exc.msg.startswith('DLL load failed'):
        return

    raise caught_exc

Add caught_exc.msg.startswith('DLL load failed') to catch dll load failed will work without speedup.

But manually LoadLibrary bezier.dll, it works well with speedup.

My Python is 3.9 version.

@dhermes

---

reference

[dhermes]

@lulucas I may consider doing that, but I'd prefer to just not distribute a library with a broken DLL (even if it's only broken on some architectures / OS versions). Unfortunately I don't have enough Windows knowledge or access to systems where this is known to fail to really make progress.

[unlinking]

@lulucas I may consider doing that, but I'd prefer to just not distribute a library with a broken DLL (even if it's only broken on some architectures / OS versions). Unfortunately I don't have enough Windows knowledge or access to systems where this is known to fail to really make progress.

This is my current workaround, is to search extra_dll_dir and load dll in there.

import patch_bezier
import bezier

# do something

# patch_bezier.py
import ctypes
import os


def _is_extra_dll(path):
    """Determine if a package path is the extra DLL on Windows.

    Args:
        path (importlib.metadata.PackagePath): A package path.

    Returns:
        bool: Indicating if this is the extra DLL on Windows.
    """
    return "extra-dll" in path.parts and path.name.endswith(".dll")


def _get_extra_dll_dir(bezier_files):
    """Determine if a package path is the extra DLL on Windows.

    Args:
        bezier_files (List[importlib.metadata.PackagePath]): List of package
            paths.

    Returns:
        str: The path of the matching ``extra-dll`` directory.

    Raises:
        ImportError: If no match can be found.
    """
    for path in bezier_files:
        if not _is_extra_dll(path):
            continue

        absolute_path = path.locate()
        return str(absolute_path.parent)

    raise ImportError("No DLL directory found", bezier_files)


def patch_dll():
    """Add the DLL directory to the module search path.

    This will only modify path if
    * on Windows
    * the ``extra-dll`` directory is in package file metadata
    """
    if os.name != "nt":
        return

    # pylint: disable=import-outside-toplevel
    try:
        import importlib.metadata as importlib_metadata
    except ImportError:  # pragma: NO COVER
        import importlib_metadata
    # pylint: enable=import-outside-toplevel

    try:
        bezier_files = importlib_metadata.files("bezier")
    except importlib_metadata.PackageNotFoundError:
        return

    extra_dll_dir = _get_extra_dll_dir(bezier_files)
    # load dll
    for file in os.listdir(extra_dll_dir):
        if file.endswith(".dll"):
            ctypes.cdll.LoadLibrary(os.path.join(extra_dll_dir, file))


patch_dll()

[dhermes]

@lulucas This is an interesting wrinkle, thanks for sharing!

Looks like the diff is here:

bezier/src/python/bezier/__config__.py

Line 97 in e6925aa

add_dll_directory(extra_dll_dir)

So add_dll_directory(extra_dll_dir) would become some kind of specialized pre-load:

def add_dll_directory(extra_dll_dir):
    if not os.path.isdir(extra_dll_dir):
        return

    os.add_dll_directory(extra_dll_dir)
    for path in pathlib.Path(extra_dll_dir).glob("*.dll"):
        ctypes.cdll.LoadLibrary(path.resolve())

[dhermes]

@lulucas Now that you have shined the light on this issue, I am starting to see some related reports:

• os.add_dll_directory does not work as expected conda/conda#10897
• os.add_dll_directory does not work as expected ContinuumIO/anaconda-issues#12475
• DLL load error while importing scipy.sparse.linalg module on Windows scipy/scipy#11062

[dhermes]

@lulucas Can you print out your full Python version? e.g. for me

$ python3.9
Python 3.9.5 (default, Jun 24 2021, 16:23:43) 
[GCC 9.3.0] on linux
...

[unlinking]

@dhermes

I test it in two windows PCs, the result is intersting.

this, works
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] on win32

this is can not load dll.
Python 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32 conda

I will do more testing with it.

[dhermes]

The Anaconda distribution is the difference, right?

[dhermes]

In your patch, try to instead just do this

os.environ["PATH"] += ";" + extra_dll_dir
os.add_dll_directory(extra_dll_dir)

I.e. see if the pre-Python 3.8 behavior of just appending to PATH is how the Conda distribution handles things

[unlinking]

The Anaconda distribution is the difference, right?

Exactly, it's issus with anaconda.

[unlinking]

only os.add_dll_directory without LoadLibrary, does not work for conda, current beizer alreay has handled this in add_dll_directory.

Maybe ImportError can be appended with an new condition to cover below error in windows to fallback, or add some notices to README.md to warn this issue.

    import bezier
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\bezier\__init__.py", line 41, in <module>
    __config__.handle_import_error(exc, "_speedup")
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\bezier\__config__.py", line 136, in handle_import_error
    raise caught_exc
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\bezier\__init__.py", line 37, in <module>
    import bezier._speedup  # noqa: F401
ImportError: DLL load failed while importing _speedup: The specified module could not be found.

def handle_import_error(caught_exc, name):
    expected_msg = TEMPLATE.format(name)
    if caught_exc.args == (expected_msg,) or caught_exc.msg.startswith('DLL load failed'):
        return

    raise caught_exc

[dhermes]

only os.add_dll_directory without LoadLibrary, does not work for conda, current beizer alreay has handled this in add_dll_directory.

@lulucas Yes I'm aware, that's why I suggested modifying os.environ["PATH"] as well. E.g. this is what SciPy does for pre-3.8 (and what bezier used to do):

$ wget https://files.pythonhosted.org/packages/52/8a/d53c6a64dd88ef80911a150478367567a9b1254d1926664524867c4d64e2/scipy-1.7.3-cp310-cp310-win_amd64.whl
$ unzip scipy-1.7.3-cp310-cp310-win_amd64.whl
$ cat scipy/__config__.py | grep -B 1 -A 6 win32

if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):
    if sys.version_info >= (3, 8):
        os.add_dll_directory(extra_dll_dir)
    else:
        os.environ.setdefault('PATH', '')
        os.environ['PATH'] += os.pathsep + extra_dll_dir

[hafizmtalha]

I hope people worked this out but I am facing the same issue on windows..
python = 3.9
bazier is installed.. but as mentioned in this thread, importing this "bezier-b9fda8dc.dll" should help.. but I checked at this location and I don't have this dll.. dll present in my folder is "bezier-2a44d276.dll"

Requirement already satisfied: bezier[full] in c:\users\hafiz\anaconda3\lib\site-packages (2021.2.12)
Requirement already satisfied: numpy>=1.20.1 in c:\users\hafiz\anaconda3\lib\site-packages (from bezier[full]) (1.21.5)
Requirement already satisfied: scipy>=1.6.0 in c:\users\hafiz\anaconda3\lib\site-packages (from bezier[full]) (1.7.3)
Requirement already satisfied: sympy>=1.7.1 in c:\users\hafiz\anaconda3\lib\site-packages (from bezier[full]) (1.10.1)
Requirement already satisfied: matplotlib>=3.3.4 in c:\users\hafiz\anaconda3\lib\site-packages (from bezier[full]) (3.5.1)
Requirement already satisfied: fonttools>=4.22.0 in c:\users\hafiz\anaconda3\lib\site-packages (from matplotlib>=3.3.4->bezier[full]) (4.25.0)
Requirement already satisfied: cycler>=0.10 in c:\users\hafiz\anaconda3\lib\site-packages (from matplotlib>=3.3.4->bezier[full]) (0.11.0)
Requirement already satisfied: pillow>=6.2.0 in c:\users\hafiz\anaconda3\lib\site-packages (from matplotlib>=3.3.4->bezier[full]) (9.0.1)
Requirement already satisfied: pyparsing>=2.2.1 in c:\users\hafiz\anaconda3\lib\site-packages (from matplotlib>=3.3.4->bezier[full]) (3.0.4)
Requirement already satisfied: python-dateutil>=2.7 in c:\users\hafiz\anaconda3\lib\site-packages (from matplotlib>=3.3.4->bezier[full]) (2.8.2)
Requirement already satisfied: packaging>=20.0 in c:\users\hafiz\anaconda3\lib\site-packages (from matplotlib>=3.3.4->bezier[full]) (21.3)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\hafiz\anaconda3\lib\site-packages (from matplotlib>=3.3.4->bezier[full]) (1.3.2)
Requirement already satisfied: six>=1.5 in c:\users\hafiz\anaconda3\lib\site-packages (from python-dateutil>=2.7->matplotlib>=3.3.4->bezier[full]) (1.16.0)
Requirement already satisfied: mpmath>=0.19 in c:\users\hafiz\anaconda3\lib\site-packages (from sympy>=1.7.1->bezier[full]) (1.2.1)

Errro::

---

ImportError Traceback (most recent call last)
Input In [3], in <cell line: 13>()
11 import numpy as np
12 from scipy.special import binom
---> 13 import bezier
14 import itertools
15 import requests

File ~\anaconda3\lib\site-packages\bezier_init_.py:41, in
39 _HAS_SPEEDUP = True
40 except ImportError as exc: # pragma: NO COVER
---> 41 config.handle_import_error(exc, "_speedup")
42 _HAS_SPEEDUP = False
43 # NOTE: The __version__ and __author__ are hard-coded here, rather
44 # than using pkg_resources.get_distribution("bezier").version
45 # and related. This is entirely to accomodate builds where
46 # bezier is imported from source (and not installed).

File ~\anaconda3\lib\site-packages\bezier_config_.py:136, in handle_import_error(caught_exc, name)
133 if caught_exc.args == (expected_msg,):
134 return
--> 136 raise caught_exc

File ~\anaconda3\lib\site-packages\bezier_init_.py:37, in
34 from bezier.triangle import Triangle
36 try:
---> 37 import bezier._speedup # noqa: F401
39 _HAS_SPEEDUP = True
40 except ImportError as exc: # pragma: NO COVER

ImportError: DLL load failed while importing _speedup: The specified module could not be found.

[rrryan2016]

After I follow your aforementioned install method as below,

I still got error information,

Traceback (most recent call last):
  File "c:\Users\ghostInSh3ll\Desktop\graph_May_6\draw.py", line 8, in <module>
    import bezier
  File "D:\Program\Anaconda3\lib\site-packages\bezier\__init__.py", line 29, in <module>
    from bezier import __config__
  File "D:\Program\Anaconda3\lib\site-packages\bezier\__config__.py", line 139, in <module>
    modify_path()
  File "D:\Program\Anaconda3\lib\site-packages\bezier\__config__.py", line 109, in modify_path
    extra_dll_dir = _get_extra_dll_dir(bezier_files)
  File "D:\Program\Anaconda3\lib\site-packages\bezier\__config__.py", line 84, in _get_extra_dll_dir
    raise ImportError("No DLL directory found", bezier_files)
ImportError: ('No DLL directory found', [PackagePath('LICENSE'), PackagePath('MANIFEST.in'), PackagePath('README.rst'), PackagePath('setup.cfg'), PackagePath('setup.py'), PackagePath('src/python/bezier/__config__.py'), PackagePath('src/python/bezier/__init__.py'), PackagePath('src/python/bezier/_base.py'), PackagePath('src/python/bezier/_curve.pxd'), PackagePath('src/python/bezier/_curve_helpers.py'), PackagePath('src/python/bezier/_curve_intersection.pxd'), PackagePath('src/python/bezier/_geometric_intersection.py'), PackagePath('src/python/bezier/_helpers.pxd'), PackagePath('src/python/bezier/_helpers.py'), PackagePath('src/python/bezier/_intersection_helpers.py'), PackagePath('src/python/bezier/_legacy.py'), PackagePath('src/python/bezier/_plot_helpers.py'), PackagePath('src/python/bezier/_speedup.c'), PackagePath('src/python/bezier/_status.pxd'), PackagePath('src/python/bezier/_symbolic.py'), PackagePath('src/python/bezier/_triangle.pxd'), PackagePath('src/python/bezier/_triangle_helpers.py'), PackagePath('src/python/bezier/_triangle_intersection.pxd'), PackagePath('src/python/bezier/_triangle_intersection.py'), PackagePath('src/python/bezier/curve.py'), PackagePath('src/python/bezier/curved_polygon.py'), PackagePath('src/python/bezier/triangle.py'), PackagePath('src/python/bezier.egg-info/PKG-INFO'), PackagePath('src/python/bezier.egg-info/SOURCES.txt'), PackagePath('src/python/bezier.egg-info/dependency_links.txt'), PackagePath('src/python/bezier.egg-info/requires.txt'), PackagePath('src/python/bezier.egg-info/top_level.txt'), PackagePath('src/python/bezier.egg-info/zip-safe'), PackagePath('src/python/bezier/hazmat/__init__.py'), PackagePath('src/python/bezier/hazmat/algebraic_intersection.py'), PackagePath('src/python/bezier/hazmat/clipping.py'), PackagePath('src/python/bezier/hazmat/curve_helpers.py'), PackagePath('src/python/bezier/hazmat/geometric_intersection.py'), PackagePath('src/python/bezier/hazmat/helpers.py'), PackagePath('src/python/bezier/hazmat/intersection_helpers.py'), PackagePath('src/python/bezier/hazmat/triangle_helpers.py'), PackagePath('src/python/bezier/hazmat/triangle_intersection.py')])

[dhermes]

@rrryan2016 Thanks for reporting! The issue was fixed in #255 but unfortunately I haven't cut a release in quite some time. I hope to cut one soon but I am unfortunately quite busy.

[YomikoR]

I built libbezier with Intel fortran compiler (classic) and encountered the same issue. It seems even if the compiler runtime libs are in PATH, python is not guaranteed to find (and load) them.
The following works for me:

import ctypes
ctypes.cdll.LoadLibrary(r'C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\redist\intel64_win\compiler\libifcoremd.dll')

[dhermes]

I'm going to close this in the hopes that later versions of Python / conda have sorted this out.

I am happy to re-open if we have new reports but it's been awhile since this initial discussion.

---

@YomikoR for cases like yours, the library now supports BEZIER_EXTRA_DLL to provide ;-delimited list of paths to add to the Python DLL search path (this is only relevant when building or installing from source, in pre-built wheels this will already be patched up via delvewheel)