Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix builtin exception handlers to work across modules #951

Closed
wants to merge 1 commit into from

Conversation

jagerman
Copy link
Member

The builtin exception handler currently doesn't work across modules under clang/libc++ for builtin pybind exceptions like pybind11::error_already_set or pybind11::stop_iteration: under RTLD_LOCAL module loading clang considers each module's exception classes distinct types (this is essentially the same as issue #912). This then means that the base exception translator fails to catch the exceptions and falls through to the generic std::exception handler, which completely breaks things like stop_iteration: only the stop_iteration of the first module loaded actually works properly. Later modules raise a RuntimeError with no message when their iterators try to signal the end.

For example, two modules defined like this exhibit the behaviour under clang++/libc++:

z1.cpp:

#include <pybind11/pybind11.h>
#include <pybind11/stl_bind.h>
namespace py = pybind11;
PYBIND11_MODULE(z1, m) {
    py::bind_vector<std::vector<long>>(m, "IntVector");
}

z2.cpp:

#include <pybind11/pybind11.h>
#include <pybind11/stl_bind.h>
namespace py = pybind11;
PYBIND11_MODULE(z2, m) {
    py::bind_vector<std::vector<double>>(m, "FloatVector");
}

Python:

import z1, z2
for i in z2.FloatVector():
    pass

results in:

Traceback (most recent call last):
  File "zs.py", line 2, in <module>
    for i in z2.FloatVector():
RuntimeError

while if you remove z1 from the import the code runs. (You could also change the order of the import to z2, z1 which lets it run, but would break z1.IntVector's iterator).

This commit fixes the issue by adding a new exception translator each time the internals pointer is initialized from python builtins: this generally means the internals data was initialized by some other module. (The extra translator(s) are skipped under libstdc++).

The builtin exception handler currently doesn't work across modules
under clang/libc++ for builtin pybind exceptions like
`pybind11::error_already_set` or `pybind11::stop_iteration`: under
RTLD_LOCAL module loading clang considers each module's exception
classes distinct types.  This then means that the base exception
translator fails to catch the exceptions and the fall through to the
generic `std::exception` handler, which completely breaks things like
`stop_iteration`: only the `stop_iteration` of the first module loaded
actually works properly; later modules raise a RuntimeError with no
message when trying to invoke their iterators.

For example, two modules defined like this exhibit the behaviour under
clang++/libc++:

z1.cpp:
    #include <pybind11/pybind11.h>
    #include <pybind11/stl_bind.h>
    namespace py = pybind11;
    PYBIND11_MODULE(z1, m) {
        py::bind_vector<std::vector<long>>(m, "IntVector");
    }

z2.cpp:
    #include <pybind11/pybind11.h>
    #include <pybind11/stl_bind.h>
    namespace py = pybind11;
    PYBIND11_MODULE(z2, m) {
        py::bind_vector<std::vector<double>>(m, "FloatVector");
    }

Python:
    import z1, z2
    for i in z2.FloatVector():
        pass

results in:
    Traceback (most recent call last):
      File "zs.py", line 2, in <module>
        for i in z2.FloatVector():
    RuntimeError

This commit fixes the issue by adding a new exception translator each
time the internals pointer is initialized from python builtins: this
generally means the internals data was initialized by some other
module.  (The extra translator(s) are skipped under libstdc++).
@jagerman jagerman force-pushed the multimod-builtin-exceptions branch from b5d0246 to a5ccb5f Compare July 19, 2017 16:41
This was referenced Jul 19, 2017
@dean0x7d
Copy link
Member

Looks good to me, but adding a test would be nice. See my comment in #949 about using the new tests module for cross-module testing.

@jagerman
Copy link
Member Author

Closing (this will be merged via #949, which depends on this fix, but also adds the cross module testing apparatus this PR needs).

@jagerman jagerman closed this Jul 29, 2017
@dean0x7d dean0x7d added this to the v2.2 milestone Aug 13, 2017
@jagerman jagerman deleted the multimod-builtin-exceptions branch August 14, 2017 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants