Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pybind11 extension module cannot be imported on windows when compiling with GCC #10454

Closed
atw1020 opened this issue Jun 3, 2022 · 3 comments
Closed

Comments

@atw1020
Copy link

atw1020 commented Jun 3, 2022

Describe the bug
When tiring to compile a test project that uses PyBind11 on a Windows computer using G++, meson is able to compile the program. However, if you try to import the resulting module in python, the program will error out with a DLL error:

>>> import adder
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing adder: The specified module could not be found.

To Reproduce

  1. On a windows machine install GCC (I used MSYS2)
  2. Ensure that G++ is in the system path
  3. Create a project directory with 2 files and an empty subprojects directory. The two files should be named "meson.build" and "module.cpp" (see below)

C:.
└───subprojects
└───meson.build
└───module.cpp

Contents of meson.build:

project('adder', 'cpp',
        default_options: ['cpp_std=c++17'], version: '0.0.1')

# ===========================================================
# Python 3
# ===========================================================

# python 3 import for extension_module
python_module = import('python')
inst = python_module.find_installation('python')

# ===========================================================
# PyBind11
# ===========================================================

# pybind11 dependency
pybind11_dep = dependency('pybind11')

inst.extension_module('adder', 'module.cpp',
                      dependencies: [pybind11_dep, inst.dependency()])

Contents of module.cpp:

#include <pybind11/pybind11.h>

int add(int a, int b){
    return a + b;
}

PYBIND11_MODULE(adder, module){

    module.doc() = "this is an example";

    module.def("add", &add);

}
  1. install pybind11 using meson wrap install pybind11
  2. run meson setup build to create a build directory
  3. examine the setup log and ensure that meson has detected the G++ compiler

pybind11| C++ compiler for the host machine: c++ (gcc 11.3.0 "c++ (Rev1, Built by MSYS2 project) 11.3.0")

  1. run meson compile -C build to compile the project
  2. cd build into the build directory
  3. run python to open an interactive shell.
  4. try to import the module "adder" using import adder

Expected behavior
Python imports the adder module successfully and a python shell user is able to call the add function provided by the library

system parameters
Build: Native
Operating System: Windows 11
Python version: 3.9.7
Meson version: 0.62.1
Ninja version: 1.10.2.git.kitware.jobserver-1

Other notes & Workarounds

  • Using MSVC instead of GCC fixes the issue.
  • The issue appears to be related to the linking of DLL libraries, examining the .pyd file that is created with dependency walker shows a few missing DLL files, but I have been unable to find these files on the internet
@atw1020
Copy link
Author

atw1020 commented Jun 5, 2022

Further research has led me to discover that this happens because the windows version of Python is built with MSVC, and therefore can't link anything compiled with GCC, which is not a bug.

@atw1020 atw1020 closed this as completed Jun 5, 2022
@jeandet
Copy link
Member

jeandet commented Jun 6, 2022

@atw1020, for the record, this is not totally exact, you can build python packages on windows with Gcc but you might want to static link gcc provided libs as much as possible like here:
https://github.com/SciQLop/CDFpp/blob/master/meson.build#L38-L47
This is distributed on PyPi and built with Gcc on windows. Note also that you might only be able to build in release mode :/.
For such issues, Dependency Walker is a good friend.

@atw1020
Copy link
Author

atw1020 commented Jun 6, 2022

I tried release mode to try to get it to link statically but it still gives the DLL error, using static linking flags could also help though, thanks for the example!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants