ImportError: DLL load failed while importing cimpl: The specified module could not be found. #1186

mikesmith1611 · 2021-08-11T13:04:03Z

Description

Import error on Windows 10 pythnon 3.8 and 3.9 conda environments. Python 3.7 works as expected.
confluent_kafka versions 1.6.0 and 1.7.0
Using binary wheels via pip install.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-efab95385e4c> in <module>
----> 1 import confluent_kafka.cimpl

~\.conda\envs\myenv\lib\site-packages\confluent_kafka\__init__.py in <module>
     38 #
     39
---> 40 from .deserializing_consumer import DeserializingConsumer
     41 from .serializing_producer import SerializingProducer
     42 from .error import KafkaException, KafkaError

~\.conda\envs\myenv\lib\site-packages\confluent_kafka\deserializing_consumer.py in <module>
     17 #
     18
---> 19 from confluent_kafka.cimpl import Consumer as _ConsumerImpl
     20 from .error import (ConsumeError,
     21                     KeyDeserializationError,

ImportError: DLL load failed while importing cimpl: The specified module could not be found.

How to reproduce

conda create --name py38 python=3.8
conda activate py38
pip install confluent_kafka

start python terminal and import conlfluent kafka

Checklist

Please provide the following information:

confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 1.60&1.70 cannot get at libversion as cannot import
Apache Kafka broker version: not relevent
Client configuration: {...} - not relevent
Operating system: Windows 10
Provide client logs (with 'debug': '..' as necessary) - above
Provide broker log excerpts - not relevent
Critical issue: yes

The text was updated successfully, but these errors were encountered:

eminizer · 2021-08-23T18:59:26Z

I am currently experiencing this same issue exactly as described in the original posting, and I have been since I started working with the Kafka Python client on Windows 10 in February 2021. Would love a solution for this so I can leverage later versions of Python in my own code that uses confluent_kafka.

edenhill · 2021-08-24T09:08:23Z

If possible, could you try locating the librdkafka.dll in your Python installation location (e.g., site-packages/..) and then run dumpbin /dependents ..the..librdkafka....dll on the librdkafka.dll and the other dlls in that directry and provide the output?

dumpbin is typically part of the MSVC compiler suite.

eminizer · 2021-08-24T16:40:01Z

Hi,

Thanks for replying!

I could not find a librdkafka.dll file in any of the following locations:

C:[username]\Miniconda3\envs\py38
C:[username]\Miniconda3\envs\py38\Lib\site-packages
C:[username]\Miniconda3\envs\py38\Lib\site-packages\confluent_kafka
C:[username]\Miniconda3\DLLs
C:[username]\Miniconda3\Library\bin

My apologies if I'm not looking in the right locations; I'm not terribly familiar with Windows systems in general. Should I be looking someplace else?

Thank you for your help!

mikesmith1611 · 2021-09-02T09:40:56Z

If possible, could you try locating the librdkafka.dll in your Python installation location (e.g., site-packages/..) and then run dumpbin /dependents ..the..librdkafka....dll on the librdkafka.dll and the other dlls in that directry and provide the output?

dumpbin is typically part of the MSVC compiler suite.

>dumpbin /dependents librdkafka-a2007a74.dll
Microsoft (R) COFF/PE Dumper Version 14.28.29337.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file librdkafka-a2007a74.dll

File Type: DLL

  Image has the following dependencies:

    libzstd-d6bdd78c.dll
    zlib-306d40ae.dll
    KERNEL32.dll
    Secur32.dll
    WS2_32.dll
    CRYPT32.dll
    USER32.dll
    ADVAPI32.dll

  Summary

       5B000 .data
       17000 .pdata
       C0000 .rdata
        9000 .reloc
        1000 .rsrc
      22B000 .text
        5000 .tls
        1000 redll

--------------------------------------

>dumpbin /dependents libzstd-d6bdd78c.dll
Microsoft (R) COFF/PE Dumper Version 14.28.29337.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file libzstd-d6bdd78c.dll

File Type: DLL

  Image has the following dependencies:

    KERNEL32.dll
    msvcrt.dll
    USER32.dll

  Summary

        1000 .CRT
        2000 .bss
        1000 .data
        1000 .debug_abbrev
        1000 .debug_aranges
        1000 .debug_frame
        2000 .debug_info
        1000 .debug_line
        2000 .edata
        1000 .idata
        3000 .pdata
        7000 .rdata
        1000 .reloc
       C9000 .text
        1000 .tls
        3000 .xdata

--------------------------------------

>dumpbin /dependents msvcr120.dll
Microsoft (R) COFF/PE Dumper Version 14.28.29337.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file msvcr120.dll

File Type: DLL

  Image has the following dependencies:

    KERNEL32.dll

  Summary

        8000 .data
        8000 .pdata
       34000 .rdata
        2000 .reloc
        1000 .rsrc
       A7000 .text

--------------------------------------

>dumpbin /dependents vcruntime140.dll
Microsoft (R) COFF/PE Dumper Version 14.28.29337.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file vcruntime140.dll

File Type: DLL

  Image has the following dependencies:

    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-convert-l1-1-0.dll
    KERNEL32.dll

  Summary

        1000 .data
        1000 .pdata
        4000 .rdata
        1000 .reloc
        1000 .rsrc
       10000 .text
        1000 _RDATA

--------------------------------------

>dumpbin /dependents zlib-306d40ae.dll
Microsoft (R) COFF/PE Dumper Version 14.28.29337.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file zlib-306d40ae.dll

File Type: DLL

  Image has the following dependencies:

    MSVCR120.dll
    KERNEL32.dll

  Summary

        1000 .data
        1000 .pdata
        6000 .rdata
        1000 .reloc
        1000 .rsrc
        C000 .text

.dll files were found here:
.conda\envs\myenv\Lib\site-packages\confluent_kafka.libs

edenhill · 2021-09-02T09:49:21Z

Thanks @mikesmith1611, that looks good.
Can you do the same for msvcrt.dll and also provide a dir listing of the dir containing the dlls?

mikesmith1611 · 2021-09-02T09:52:39Z

There is no msvcrt.dll there is an msvcr120.dll as shown in above output. This is the list of files @edenhill :

02/09/2021  10:30                98 .load_order
02/09/2021  10:30         3,538,432 librdkafka-a2007a74.dll
02/09/2021  10:30           968,907 libzstd-d6bdd78c.dll
02/09/2021  10:30           963,232 msvcr120.dll
02/09/2021  10:30            96,144 vcruntime140.dll
02/09/2021  10:30            77,824 zlib-306d40ae.dll

edenhill · 2021-09-02T10:03:16Z

Thank you.
Is there an msvcrt.dll in your windows/system or windows/system32 dir?

mikesmith1611 · 2021-09-02T10:06:15Z

Thank you.
Is there an msvcrt.dll in your windows/system or windows/system32 dir?

C:\Windows\System32>dumpbin /dependents msvcrt.dll
Microsoft (R) COFF/PE Dumper Version 14.28.29337.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file msvcrt.dll

File Type: DLL

  Image has the following dependencies:

    ntdll.dll
    api-ms-win-core-console-l1-1-0.dll
    api-ms-win-core-console-l1-2-0.dll
    api-ms-win-core-datetime-l1-1-0.dll
    api-ms-win-core-debug-l1-1-0.dll
    api-ms-win-core-errorhandling-l1-1-0.dll
    api-ms-win-core-fibers-l1-1-0.dll
    api-ms-win-core-file-l1-1-0.dll
    api-ms-win-core-handle-l1-1-0.dll
    api-ms-win-core-heap-l1-1-0.dll
    api-ms-win-core-localization-l1-2-0.dll
    api-ms-win-core-libraryloader-l1-2-0.dll
    api-ms-win-core-memory-l1-1-0.dll
    api-ms-win-core-namedpipe-l1-1-0.dll
    api-ms-win-core-processenvironment-l1-1-0.dll
    api-ms-win-core-processthreads-l1-1-0.dll
    api-ms-win-core-profile-l1-1-0.dll
    api-ms-win-core-string-l1-1-0.dll
    api-ms-win-core-synch-l1-1-0.dll
    api-ms-win-core-synch-l1-2-0.dll
    api-ms-win-core-sysinfo-l1-1-0.dll
    api-ms-win-core-util-l1-1-0.dll
    KERNELBASE.dll

  Summary

        6000 .data
        2000 .idata
        4000 .reloc
        1000 .rsrc
       B1000 .text

edenhill · 2021-09-02T10:11:18Z

Thank you! So it seems all the libraries are in order, so maybe Python is looking for cimpl/librdkafka in the wrong location?
Can you find the cimpl library and do a dumpbin on that one too?

mikesmith1611 · 2021-09-02T10:18:16Z

Thank you! So it seems all the libraries are in order, so maybe Python is looking for cimpl/librdkafka in the wrong location?
Can you find the cimpl library and do a dumpbin on that one too?

This is the confluent_kafta directory containing cimpl.cp38-win_amd64.pyd. There are no dlls in here. Cannot find any other reference to cimpl library in my env.

 Directory of site-packages\confluent_kafka

02/09/2021  10:30    <DIR>          .
02/09/2021  10:30    <DIR>          ..
02/09/2021  10:30    <DIR>          admin
02/09/2021  10:30    <DIR>          avro
02/09/2021  10:30           129,536 cimpl.cp38-win_amd64.pyd
02/09/2021  10:30             8,181 deserializing_consumer.py
02/09/2021  10:30             4,708 error.py
02/09/2021  10:30    <DIR>          kafkatest
02/09/2021  10:30    <DIR>          schema_registry
02/09/2021  10:30    <DIR>          serialization
02/09/2021  10:30             9,734 serializing_producer.py
02/09/2021  10:30             5,220 __init__.py
02/09/2021  10:30    <DIR>          __pycache__

mikesmith1611 · 2021-09-02T10:29:00Z

@edenhill In PyCharm using python 3.7 I can navigate to cimpl.py via ctrl + click on the "from .cimpl import ..." line in the confluent_kafka.__init__.py but in 3.8 it says cannot find declaration.

edenhill · 2021-09-02T10:41:17Z

I think .pyd might actually be a DLL, so try dumpbin on it.

mikesmith1611 · 2021-09-02T10:46:17Z

I think .pyd might actually be a DLL, so try dumpbin on it.

Dump of file cimpl.cp38-win_amd64.pyd

File Type: DLL

  Image has the following dependencies:

    librdkafka-a2007a74.dll
    python38.dll
    KERNEL32.dll
    VCRUNTIME140.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll

  Summary

        D000 .data
        1000 .pdata
        5000 .rdata
        1000 .reloc
        1000 .rsrc
        E000 .text
        1000 redll

edenhill · 2021-09-02T10:55:59Z

🤷 Very weird. Everything looks fine.

mikesmith1611 · 2021-09-02T11:08:18Z

Can you recreate the issue using a python 3.8 conda virtual environment?

edenhill · 2021-09-02T11:49:58Z

I haven't used conda before, am I supposed to install it confluent-kafka with conda install python-confluent-kafka (which does not work because there does not seem to be any windows buidlds) or through pip in a conda environment?

mikesmith1611 · 2021-09-02T13:09:21Z

I haven't used conda before, am I supposed to install it confluent-kafka with conda install python-confluent-kafka (which does not work because there does not seem to be any windows buidlds) or through pip in a conda environment?

Through pip in a conda environment:

conda create --name py38 python=3.8
conda activate py38
pip install confluent_kafka

edenhill · 2021-09-02T14:57:14Z

Thanks! Reproduced. Non the wiser

edenhill · 2021-09-02T15:18:21Z

Seems like this is a problem with conda:
ContinuumIO/anaconda-issues#12475

The provided workaround works:

(py38) C:\WINDOWS\system32>set CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1
(py38) C:\WINDOWS\system32>echo %CONDA_DLL_SEARCH_MODIFICATION_ENABLE%
1

(py38) C:\WINDOWS\system32>python
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 06:25:23) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import confluent_kafka
>>> confluent_kafka.version()
('1.7.0', 17235968)
>>> confluent_kafka.libversion()
('1.7.0', 17236223)

mikesmith1611 · 2021-09-02T15:31:57Z

This worked! Thank you. Hopefully they put in a fix for this. At least it is an easy workaround.

mikesmith1611 closed this as completed Sep 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ImportError: DLL load failed while importing cimpl: The specified module could not be found. #1186

ImportError: DLL load failed while importing cimpl: The specified module could not be found. #1186

mikesmith1611 commented Aug 11, 2021 •

edited

eminizer commented Aug 23, 2021

edenhill commented Aug 24, 2021

eminizer commented Aug 24, 2021

mikesmith1611 commented Sep 2, 2021 •

edited

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021 •

edited

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021 •

edited

mikesmith1611 commented Sep 2, 2021 •

edited

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

edenhill commented Sep 2, 2021

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

ImportError: DLL load failed while importing cimpl: The specified module could not be found. #1186

ImportError: DLL load failed while importing cimpl: The specified module could not be found. #1186

Comments

mikesmith1611 commented Aug 11, 2021 • edited

Description

How to reproduce

Checklist

eminizer commented Aug 23, 2021

edenhill commented Aug 24, 2021

eminizer commented Aug 24, 2021

mikesmith1611 commented Sep 2, 2021 • edited

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021 • edited

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021 • edited

mikesmith1611 commented Sep 2, 2021 • edited

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

edenhill commented Sep 2, 2021

edenhill commented Sep 2, 2021

mikesmith1611 commented Sep 2, 2021

mikesmith1611 commented Aug 11, 2021 •

edited

mikesmith1611 commented Sep 2, 2021 •

edited

mikesmith1611 commented Sep 2, 2021 •

edited

mikesmith1611 commented Sep 2, 2021 •

edited

mikesmith1611 commented Sep 2, 2021 •

edited