`PyCall` cannot find `ctypes` on python 3.10.2 #963

EHBaozi · 2022-01-28T21:47:55Z

As the title says, despite the fact that ctypes is present. It also causes Julia to crash.

PyCall version 1.93.0, python version 3.10.2 (managed through a miniforge installation).

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

Full stacktrace:

julia> using PyCall
ERROR: InitError: PyError (PyImport_ImportModule
The Python package ctypes could not be imported by pyimport. Usually this means

that you did not install ctypes in the Python version being used by PyCall.
PyCall is currently configured to use the Python version at:
C:\Users\invis\scoop\apps\mambaforge\current\python.exe
and you should use whatever mechanism you usually use (apt-get, pip, conda,

etcetera) to install the Python package containing the ctypes module.
One alternative is to re-configure PyCall to use a different Python

version on your system: set ENV["PYTHON"] to the path/name of the python

executable you want to use, run Pkg.build("PyCall"), and re-launch Julia.
Another alternative is to configure PyCall to use a Julia-specific Python

distribution via the Conda.jl package (which installs a private Anaconda

Python distribution), which has the advantage that packages can be installed

and kept up-to-date via Julia.  As explained in the PyCall documentation,

set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then,

To install the ctypes module, you can use pyimport_conda("ctypes", PKG),

where PKG is the Anaconda package that contains the module ctypes,

or alternatively you can use the Conda package directly (via

using Conda followed by Conda.add etcetera).
) <class 'ImportError'>

ImportError('DLL load failed while importing _ctypes: The specified module could not be found.')
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.

Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffb4706a8c5 -- PyObject_Call at C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\python310.dll (unknown line)

in expression starting at none:0

PyObject_Call at C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\python310.dll (unknown line)

macro expansion at C:\Users\invis.julia\packages\PyCall\L0fLP\src\exception.jl:95 [inlined]

#107 at C:\Users\invis.julia\packages\PyCall\L0fLP\src\pyfncall.jl:43 [inlined]

disable_sigint at .\c.jl:458 [inlined]

__pycall! at C:\Users\invis.julia\packages\PyCall\L0fLP\src\pyfncall.jl:42 [inlined]

_pycall! at C:\Users\invis.julia\packages\PyCall\L0fLP\src\pyfncall.jl:29

_pycall! at C:\Users\invis.julia\packages\PyCall\L0fLP\src\pyfncall.jl:11 [inlined]

#pycall#112 at C:\Users\invis.julia\packages\PyCall\L0fLP\src\pyfncall.jl:80 [inlined]

pycall at C:\Users\invis.julia\packages\PyCall\L0fLP\src\pyfncall.jl:80 [inlined]

show at C:\Users\invis.julia\packages\PyCall\L0fLP\src\exception.jl:43

showerror at .\errorshow.jl:30 [inlined]

#showerror#813 at .\errorshow.jl:88

showerror##kw at .\errorshow.jl:87

unknown function (ip: 000000003cf2a7d3)

#showerror#815 at .\errorshow.jl:103

showerror##kw at .\errorshow.jl:102

unknown function (ip: 000000003cf27883)

show_exception_stack at .\errorshow.jl:866

display_error at .\client.jl:104

jfptr_display_error_48751.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)

jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]

jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757

#invokelatest#2 at .\essentials.jl:716 [inlined]

invokelatest at .\essentials.jl:714 [inlined]

print_response at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:286

#45 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:275

jfptr_YY.45_49620.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)

with_repl_linfo at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:508

print_response at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:273

do_respond at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:844

jfptr_do_respond_49916.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)

jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]

jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757

#invokelatest#2 at .\essentials.jl:716 [inlined]

invokelatest at .\essentials.jl:714 [inlined]

run_interface at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\LineEdit.jl:2493

jfptr_run_interface_50086.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)

run_frontend at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:1230

#49 at .\task.jl:423

jfptr_YY.49_49929.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)

jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]

start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:877

Allocations: 13768873 (Pool: 13762844; Big: 6029); GC: 16

The text was updated successfully, but these errors were encountered:

stevengj · 2022-01-29T13:12:26Z

Probably you have some path problem. What is PyCall.python, are you letting PyCall install its own Python distro via Conda or are you using some other Python distro?

EHBaozi · 2022-01-29T15:54:09Z

I'm using my own python installation (using miniconda). THe path is properly set, i.e. I have the following before using PyCall:

julia> println.(split(ENV["PATH"], ";"));
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\Library\mingw-w64\bin
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\Library\usr\bin
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\Library\bin
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\Scripts
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\bin
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\condabin
...

and PyCall is also built with this python installation.

I cannot get PyCall.python since using PyCall crashes Julia.

tkf · 2022-01-30T01:07:32Z

You can look at the file C:\Users\invis.julia\packages\PyCall\L0fLP\deps\deps.jl (the L0fLP part may change if you use different PyCall version). You can also try typing ]build --verbose PyCall in the REPL.

I cannot get PyCall.python since using PyCall crashes Julia.

@stevengj This is why I think #945 should have a separated config package.

EHBaozi · 2022-01-30T01:23:25Z

deps.jl:

const python = "C:\\Users\\invis\\scoop\\apps\\mambaforge\\4.10.1-4\\python.exe"
const libpython = "C:\\Users\\invis\\scoop\\apps\\mambaforge\\4.10.1-4\\python310.dll"
const pyprogramname = "C:\\Users\\invis\\scoop\\apps\\mambaforge\\4.10.1-4\\python.exe"
const pyversion_build = v"3.10.2"
const PYTHONHOME = "C:\\Users\\invis\\scoop\\apps\\mambaforge\\4.10.1-4"

"True if we are using the Python distribution in the Conda package."
const conda = false

The paths are set up ok.

I did a bit more digging and it turns out that this problem doesn't happen with python 3.9.4. I switched to another conda environment w/ python 3.9.4, rebuilt PyCall with it and now it's working just fine. So something in python 3.10 is making PyCall unhappy. Don't know what exactly though.

metab0t · 2022-01-30T06:26:10Z

I met the same problem.
I use Python package from conda-forge and PyCall.jl 1.93.0 on Windows 10.

Python 3.9.7 works fine
Python 3.9.10 errors that it fails to import _ctypes but I can import it in Python

It seems to be some issues on conda-forge side.
See
conda-forge/python-feedstock#539
pre-commit/pre-commit#1329
The same error ImportError: DLL load failed while importing _ctypes: The specified module could not be found.

Currently, I pin Python package at 3.9.7 as a temporary solution.
https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning

isuruf · 2022-01-30T07:11:03Z

Can you send the logs with CONDA_DLL_SEARCH_MODIFICATION_DEBUG=1 env variable set?

metab0t · 2022-01-30T08:45:12Z

@isuruf
When Python 3.9.10 errors, I get following log

CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\Scripts - ExePrefix)

CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\Library\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\Library\usr\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\Library\mingw-w64\bin - ExePrefix)

I think it should search miniforge3 prefix rather than Julia directory.

I find your PR conda-forge/python-feedstock#546.
Thanks for quick response and patch!

isuruf · 2022-01-30T20:31:42Z

Should be fixed now. Update your python and it should work.

EHBaozi · 2022-01-30T22:28:28Z

Can confirm. Thanks!

EHBaozi changed the title ~~PyCall cannot find ctypes~~ PyCall cannot find ctypes on python 3.10.2 Jan 30, 2022

EHBaozi closed this as completed Jan 30, 2022

tomaarsen mentioned this issue Jul 27, 2022

Pre commit hook fails nltk/nltk#3027

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`PyCall` cannot find `ctypes` on python 3.10.2 #963

`PyCall` cannot find `ctypes` on python 3.10.2 #963

EHBaozi commented Jan 28, 2022

stevengj commented Jan 29, 2022

EHBaozi commented Jan 29, 2022

tkf commented Jan 30, 2022

EHBaozi commented Jan 30, 2022

metab0t commented Jan 30, 2022 •

edited

isuruf commented Jan 30, 2022

metab0t commented Jan 30, 2022 •

edited

isuruf commented Jan 30, 2022

EHBaozi commented Jan 30, 2022

PyCall cannot find ctypes on python 3.10.2 #963

PyCall cannot find ctypes on python 3.10.2 #963

Comments

EHBaozi commented Jan 28, 2022

stevengj commented Jan 29, 2022

EHBaozi commented Jan 29, 2022

tkf commented Jan 30, 2022

EHBaozi commented Jan 30, 2022

metab0t commented Jan 30, 2022 • edited

isuruf commented Jan 30, 2022

metab0t commented Jan 30, 2022 • edited

isuruf commented Jan 30, 2022

EHBaozi commented Jan 30, 2022

`PyCall` cannot find `ctypes` on python 3.10.2 #963

`PyCall` cannot find `ctypes` on python 3.10.2 #963

metab0t commented Jan 30, 2022 •

edited

metab0t commented Jan 30, 2022 •

edited