Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Passing both python functions and std::function as arguments interchangeably #11

Closed
Vayu opened this issue Nov 16, 2015 · 3 comments
Closed

Comments

@Vayu
Copy link

Vayu commented Nov 16, 2015

I noticed that including pybind11/functional.h has some (probably) unintended side-effects.

If I include pybind11/functional.h I cannot pass std::function instances which I get from C++ code back to C++ functions. On the other hand without pybind11/functional.h, I cannot pass Python methods to C++ functions expecting std::function.

Is it possible to get both working, or are they mutually exclusive? I'm attaching a short example

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>  // comment/uncomment this

struct Functor {
    bool operator() (int a, int b) const {
      return a < b;
    }
    std::function<bool(int, int)> cast() {
      return std::function<bool(int, int)>(*this);
    }
};

struct Caller {
    bool call(std::function<bool(int, int)> func, int a, int b) {
      return func(a, b);
    }
};

namespace py = pybind11;

PYBIND11_PLUGIN(tmp) {
    py::module m("tmp", "pybind11 tmp plugin");

    py::class_<Caller>(m, "Caller").def(py::init<>())
                                   .def("call", &Caller::call);

    py::class_<std::function<bool(int, int)>>(m, "FuncType").def(py::init<>());

    py::class_<Functor>(m, "Functor").def(py::init<>())
                                     .def("__call__", [](const Functor& f, int a, int b) { return f(a, b); })
                                     .def("cast", &Functor::cast);
    return m.ptr();
}
import tmp

cppfunc = tmp.Functor()
caller = tmp.Caller()

def pyfunc(a, b):
    return a < b

try:
    print 'Calling C++ functor {0!r}'.format(cppfunc),
    res = cppfunc(1, 2)
    print 'OK'
except Exception as error:
    print 'FAILED with error {1!r}'.format(cppfunc, error)

try:
    print 'Calling C++ with Python function {0!r}'.format(pyfunc),
    caller.call(pyfunc, 1, 2)
    print 'OK'
except Exception as error:
    print 'FAILED with error {1!r}'.format(cppfunc, error)

stdfunc = cppfunc.cast()
try:
    print 'Calling C++ with C++ function {0!r}'.format(stdfunc),
    caller.call(stdfunc, 1, 2)
    print 'OK'
except Exception as error:
    print 'FAILED with error {1!r}'.format(cppfunc, error)

Without pybind11/functional.h

Calling C++ functor <tmp.Functor object> OK
Calling C++ with Python function <function pyfunc> FAILED with error TypeError('Incompatible function arguments. The following argument types are supported:\n    1. (Caller, std::function<bool (int, int)>, int32_t, int32_t) -> bool\n',)
Calling C++ with C++ function <tmp.FuncType object> OK

With pybind11/functional.h

Calling C++ functor <tmp.Functor object> OK
Calling C++ with Python function <function pyfunc> OK
Calling C++ with C++ function <built-in method  of PyCapsule object> FAILED with error TypeError('Incompatible function arguments. The following argument types are supported:\n    1. (Caller, function<(int32_t, int32_t) -> bool>, int32_t, int32_t) -> bool\n',)
@wjakob
Copy link
Member

wjakob commented Nov 16, 2015

Fixed in commit eb7c0b8 -- all cases now work when including functional.h. One thing you'll have to keep in mind though is that when passing functions back and forth is that each roundtrip will add one layer of Python -> C++ -> Python instrumentation that gets called every time you invoke the 2x wrapped function :). To avoid that, you will have to split your codebase into parts that include functional.h and those that don't.

@wjakob wjakob closed this as completed Nov 16, 2015
@Vayu
Copy link
Author

Vayu commented Nov 16, 2015

Thanks!

Perhaps that is worth adding to documentation, so people who want minimum overhead would stick to "non-functional" approach.

By splitting into parts, you mean separate classes at least? It doesn't look like same class could have both methods with "functional" semantics and "object" semantics.

@wjakob
Copy link
Member

wjakob commented Nov 17, 2015

Ok, I added a warning.

You can actually do the aforementioned split, for instance by passing a pybind11::class_ to another translation unit (that doesn't include functional), which adds further functions.

nbigaouette pushed a commit to nbigaouette/pybind11 that referenced this issue Jan 12, 2016
This should insure that both the PythonLibs and PythonInterp points to
the same python version.

Without this, the Python library and interpreter might not match version.
For example, if both Python 2.7 and 3.4 is installed, PythonLibs will
find /usr/lib/x86_64-linux-gnu/libpython3.4m.so while
PythonInterp will find /usr/bin/python2.7 (at least on Ubuntu 14.04).

When PythonLibs and PythonInterp don't point to the same Python version,
the examples will all fail:

$ cmake ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Setting build type to 'MinSizeRel' as none was specified.
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.4m.so (found suitable version "3.4.3", minimum required is "2.7")
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.6", minimum required is "2.7")
-- Performing Test HAS_LTO_FLAG
-- Performing Test HAS_LTO_FLAG - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/nbigaouette/pybind11/build
$ make test
Running tests...
Test project /home/nbigaouette/pybind11/build
      Start  1: example1
 1/12 Test  pybind#1: example1 .........................***Failed    0.02 sec
      Start  2: example2
 2/12 Test  pybind#2: example2 .........................***Failed    0.03 sec
      Start  3: example3
 3/12 Test  pybind#3: example3 .........................***Failed    0.02 sec
      Start  4: example4
 4/12 Test  pybind#4: example4 .........................***Failed    0.02 sec
      Start  5: example5
 5/12 Test  pybind#5: example5 .........................***Failed    0.02 sec
      Start  6: example6
 6/12 Test  pybind#6: example6 .........................***Failed    0.02 sec
      Start  7: example7
 7/12 Test  pybind#7: example7 .........................***Failed    0.02 sec
      Start  8: example8
 8/12 Test  pybind#8: example8 .........................***Failed    0.02 sec
      Start  9: example9
 9/12 Test  pybind#9: example9 .........................***Failed    0.02 sec
      Start 10: example10
10/12 Test pybind#10: example10 ........................***Failed    0.02 sec
      Start 11: example11
11/12 Test pybind#11: example11 ........................***Failed    0.03 sec
      Start 12: example12
12/12 Test pybind#12: example12 ........................***Failed    0.02 sec

0% tests passed, 12 tests failed out of 12

Total Test time (real) =   0.25 sec

The following tests FAILED:
          1 - example1 (Failed)
          2 - example2 (Failed)
          3 - example3 (Failed)
          4 - example4 (Failed)
          5 - example5 (Failed)
          6 - example6 (Failed)
          7 - example7 (Failed)
          8 - example8 (Failed)
          9 - example9 (Failed)
         10 - example10 (Failed)
         11 - example11 (Failed)
         12 - example12 (Failed)
Errors while running CTest
make: *** [test] Error 8


By adding the EXACT version to the find_package() calls, the version
discrepency is at least caught at the cmake call:

$ cmake ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Setting build type to 'MinSizeRel' as none was specified.
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
  Could NOT find PythonLibs: Found unsuitable version "3.4.3", but required
  is exact version "2.7" (found /usr/lib/x86_64-linux-gnu/libpython3.4m.so)
Call Stack (most recent call first):
  /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:313 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-2.8/Modules/FindPythonLibs.cmake:208 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:27 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/nbigaouette/pybind11/build/CMakeFiles/CMakeOutput.log".
nbigaouette pushed a commit to nbigaouette/pybind11 that referenced this issue Jan 12, 2016
…ts to

the same python version.

Without this, the Python library and interpreter might not match version.
For example, if both Python 2.7 and 3.4 is installed, `PythonLibs` will
find /usr/lib/x86_64-linux-gnu/libpython3.4m.so while
`PythonInterp` will find /usr/bin/python2.7 (at least on Ubuntu 14.04).

When `PythonLibs` and `PythonInterp` don't point to the same Python version,
the examples will all fail:

```bash
$ cmake ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Setting build type to 'MinSizeRel' as none was specified.
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.4m.so (found suitable version "3.4.3", minimum required is "2.7")
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.6", minimum required is "2.7")
-- Performing Test HAS_LTO_FLAG
-- Performing Test HAS_LTO_FLAG - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/nbigaouette/pybind11/build
$ make test
Running tests...
Test project /home/nbigaouette/pybind11/build
      Start  1: example1
 1/12 Test  pybind#1: example1 .........................***Failed    0.02 sec
      Start  2: example2
 2/12 Test  pybind#2: example2 .........................***Failed    0.03 sec
      Start  3: example3
 3/12 Test  pybind#3: example3 .........................***Failed    0.02 sec
      Start  4: example4
 4/12 Test  pybind#4: example4 .........................***Failed    0.02 sec
      Start  5: example5
 5/12 Test  pybind#5: example5 .........................***Failed    0.02 sec
      Start  6: example6
 6/12 Test  pybind#6: example6 .........................***Failed    0.02 sec
      Start  7: example7
 7/12 Test  pybind#7: example7 .........................***Failed    0.02 sec
      Start  8: example8
 8/12 Test  pybind#8: example8 .........................***Failed    0.02 sec
      Start  9: example9
 9/12 Test  pybind#9: example9 .........................***Failed    0.02 sec
      Start 10: example10
10/12 Test pybind#10: example10 ........................***Failed    0.02 sec
      Start 11: example11
11/12 Test pybind#11: example11 ........................***Failed    0.03 sec
      Start 12: example12
12/12 Test pybind#12: example12 ........................***Failed    0.02 sec

0% tests passed, 12 tests failed out of 12

Total Test time (real) =   0.25 sec

The following tests FAILED:
          1 - example1 (Failed)
          2 - example2 (Failed)
          3 - example3 (Failed)
          4 - example4 (Failed)
          5 - example5 (Failed)
          6 - example6 (Failed)
          7 - example7 (Failed)
          8 - example8 (Failed)
          9 - example9 (Failed)
         10 - example10 (Failed)
         11 - example11 (Failed)
         12 - example12 (Failed)
Errors while running CTest
make: *** [test] Error 8
```

By adding the `EXACT` version to the `find_package()` calls, the version
discrepancy is at least caught at the cmake call:

```bash
$ cmake ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Setting build type to 'MinSizeRel' as none was specified.
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
  Could NOT find PythonLibs: Found unsuitable version "3.4.3", but required
  is exact version "2.7" (found /usr/lib/x86_64-linux-gnu/libpython3.4m.so)
Call Stack (most recent call first):
  /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:313 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-2.8/Modules/FindPythonLibs.cmake:208 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:27 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/nbigaouette/pybind11/build/CMakeFiles/CMakeOutput.log".
```
nbigaouette pushed a commit to nbigaouette/pybind11 that referenced this issue Jan 12, 2016
This should insure that both the `PythonLibs` and `PythonInterp` points to
the same python version.

Without this, the Python library and interpreter might not match version.
For example, if both Python 2.7 and 3.4 is installed, `PythonLibs` will
find /usr/lib/x86_64-linux-gnu/libpython3.4m.so while
`PythonInterp` will find /usr/bin/python2.7 (at least on Ubuntu 14.04).

When `PythonLibs` and `PythonInterp` don't point to the same Python version,
the examples will all fail:

```bash
$ cmake ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Setting build type to 'MinSizeRel' as none was specified.
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.4m.so (found suitable version "3.4.3", minimum required is "2.7")
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.6", minimum required is "2.7")
-- Performing Test HAS_LTO_FLAG
-- Performing Test HAS_LTO_FLAG - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/nbigaouette/pybind11/build
$ make test
Running tests...
Test project /home/nbigaouette/pybind11/build
      Start  1: example1
 1/12 Test  pybind#1: example1 .........................***Failed    0.02 sec
      Start  2: example2
 2/12 Test  pybind#2: example2 .........................***Failed    0.03 sec
      Start  3: example3
 3/12 Test  pybind#3: example3 .........................***Failed    0.02 sec
      Start  4: example4
 4/12 Test  pybind#4: example4 .........................***Failed    0.02 sec
      Start  5: example5
 5/12 Test  pybind#5: example5 .........................***Failed    0.02 sec
      Start  6: example6
 6/12 Test  pybind#6: example6 .........................***Failed    0.02 sec
      Start  7: example7
 7/12 Test  pybind#7: example7 .........................***Failed    0.02 sec
      Start  8: example8
 8/12 Test  pybind#8: example8 .........................***Failed    0.02 sec
      Start  9: example9
 9/12 Test  pybind#9: example9 .........................***Failed    0.02 sec
      Start 10: example10
10/12 Test pybind#10: example10 ........................***Failed    0.02 sec
      Start 11: example11
11/12 Test pybind#11: example11 ........................***Failed    0.03 sec
      Start 12: example12
12/12 Test pybind#12: example12 ........................***Failed    0.02 sec

0% tests passed, 12 tests failed out of 12

Total Test time (real) =   0.25 sec

The following tests FAILED:
          1 - example1 (Failed)
          2 - example2 (Failed)
          3 - example3 (Failed)
          4 - example4 (Failed)
          5 - example5 (Failed)
          6 - example6 (Failed)
          7 - example7 (Failed)
          8 - example8 (Failed)
          9 - example9 (Failed)
         10 - example10 (Failed)
         11 - example11 (Failed)
         12 - example12 (Failed)
Errors while running CTest
make: *** [test] Error 8
```

By adding the `EXACT` version to the `find_package()` calls, the version
discrepancy is at least caught at the cmake call:

```bash
$ cmake ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Setting build type to 'MinSizeRel' as none was specified.
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
  Could NOT find PythonLibs: Found unsuitable version "3.4.3", but required
  is exact version "2.7" (found /usr/lib/x86_64-linux-gnu/libpython3.4m.so)
Call Stack (most recent call first):
  /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:313 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-2.8/Modules/FindPythonLibs.cmake:208 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:27 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/nbigaouette/pybind11/build/CMakeFiles/CMakeOutput.log".
```
EricCousineau-TRI referenced this issue in EricCousineau-TRI/pybind11 Mar 10, 2018
unique_ptr: Fix ownership issues for failed overloads (RobotLocomotion/drake#8160)
rwgk pushed a commit that referenced this issue Jun 18, 2024
* Support free-threaded CPython (PEP 703)

Some additional locking is added in the free-threaded build when
`Py_GIL_DISABLED` is defined:

- Most accesses to internals are protected by a single mutex
- The registered_instances uses a striped lock to improve concurrency

Pybind11 modules can indicate support for running with the GIL disabled
by calling `set_gil_not_used()`.

* refactor: use PYBIND11_MODULE (#11)

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

* Address code review

* Suppress MSVC warning

* Changes from review

* style: pre-commit fixes

* `py::mod_gil_not_used()` suggestion.

* Update include/pybind11/pybind11.h

---------

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants