Compile without exceptions #1703

av8ramit · 2019-02-20T19:50:55Z

Wrapping exception logic in #ifdefs to allow for compilation without exceptions.

Context: We'd like to use pybind, but are in an environment where -fno-exceptions is required (Google). The suggested changes are sufficient to make it work, but if you prefer a different approach that'd be fine too.

…exceptions.

…etail namespace.

av8ramit · 2019-02-21T17:54:24Z

I believe the two failures on Travis are unrelated.

E: Unable to locate package python3.6-dev
E: Couldn't find any package by glob 'python3.6-dev'
E: Couldn't find any package by regex 'python3.6-dev'

Installing pytest, numpy, scipy...

Looking into appveyor, but I'm not sure why this should affect those.

av8ramit · 2019-02-25T18:41:33Z

I can confirm with this latest pull request (#1709) that the failing presubmits are unrelated to my change.

av8ramit · 2019-03-26T00:07:12Z

@wjakob sorry to bug! Would it be possible to provide an ETA on any review?

bstaletic

I'm not an official maintainer, but this needs a change log entry and it would be nice to have tests to show what happens to these pieces of code when compiled without exceptions.

bstaletic · 2019-04-26T20:00:45Z

include/pybind11/detail/common.h

+  try {                                                   \
+    PYBIND11_CONCAT(pybind11_init_, name)(module);        \
+    return module.ptr();                                  \
+  } catch (pybind11::error_already_set & e) {             \


If we're replacing PYBIND11_CATCH_INIT_EXCEPTIONS with this function, let's do it consistently and remove the macro completely, which means replacing the macro in the deprecated PYBIND11_PLUGIN as well.

That was not intentional. Merged the latest changes which should make it more clear.

bstaletic · 2019-04-26T20:00:50Z

include/pybind11/detail/common.h

@@ -369,6 +396,23 @@ constexpr size_t instance_simple_holder_in_ptrs() {
    return size_in_ptrs(sizeof(std::shared_ptr<int>));
 }

+inline bool Py_VersionCheckPassed() {


Similar to my other comment, we should consistently replace the PYBIND11_CHECK_PYTHON_VERSION macro.

Done. Thanks!

…exceptions.

…etail namespace.

…ceptions

Merging the latest changes.

…o the deprecated PYBIND11_PLUGIN as well.

av8ramit · 2019-05-31T14:28:24Z

@bstaletic do you mind please providing more insight into the change log entry? Any specific file I should modify or add a short note in the README?

bstaletic

The changelog is in docs/changelog.rst.

bstaletic · 2019-05-31T14:54:11Z

include/pybind11/detail/common.h

+    static void PYBIND11_CONCAT(pybind11_init_, name)(pybind11::module &);     \
+    PYBIND11_PLUGIN_IMPL(name) {                                               \
+        PYBIND11_CHECK_PYTHON_VERSION                                          \
+        auto m = pybind11::module(PYBIND11_TOSTRING(name));                    \


So what happens now if constructing an extension module throws?

Imagine a line inside a user's PYBIND11_MODULE that says py::module foo = py::import("throwy_module"); and actually having a exception thrown from the throwy_module.

If you've set the flag PYBIND11_NOEXCEPTIONS and you run without enable exceptions I think it will fail to compile.

I was talking about raising an exception from pure python, which can't be detected at compile time. I gave it a test and here are the two similar cases that I've tried:

// foo.cpp #include <pybind11/pybind11.h> PYBIND11_MODULE(foo, m) { pybind11::module x = pybind11::module::import("boom"); }

If there is no boom.py in python's path, the following happens:

Python 3.7.3 (default, Mar 27 2019, 19:19:44) [GCC 8.2.1 20181127] on linux Type "help", "copyright", "credits" or "license" for more information. >>> try: ... import foo ... except Exception: ... KeyboardInterrupt >>> try: ... import foo ... except Exception as e: ... print(e) ... terminate called after throwing an instance of 'pybind11::error_already_set' what(): ModuleNotFoundError: No module named 'boom' zsh: abort python

And if we add a boom.py that looks like this:

raise RuntimeError

the error becomes more verbose:

Python 3.7.3 (default, Mar 27 2019, 19:19:44) [GCC 8.2.1 20181127] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import boom Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/bstaletic/Temp/pybind11/boom.py", line 1, in <module> raise RuntimeError RuntimeError >>> try: ... import foo ... except Exception as e: ... print(e) ... terminate called after throwing an instance of 'pybind11::error_already_set' what(): RuntimeError: At: /home/bstaletic/Temp/pybind11/boom.py(1): <module> <frozen importlib._bootstrap>(219): _call_with_frames_removed <frozen importlib._bootstrap_external>(728): exec_module <frozen importlib._bootstrap>(677): _load_unlocked <frozen importlib._bootstrap>(967): _find_and_load_unlocked <frozen importlib._bootstrap>(983): _find_and_load <frozen importlib._bootstrap>(219): _call_with_frames_removed <frozen importlib._bootstrap_external>(1043): create_module <frozen importlib._bootstrap>(583): module_from_spec <frozen importlib._bootstrap>(670): _load_unlocked <frozen importlib._bootstrap>(967): _find_and_load_unlocked <frozen importlib._bootstrap>(983): _find_and_load <stdin>(2): <module> zsh: abort python

Either way, it quickly calls std::terminate(), which is completely fine, but let's mention it in the changelog too.

Ah whoops, I totally misread that. Okay I'll update the changelog. Thanks @bstaletic

bstaletic · 2019-06-10T06:23:10Z

I'm still unable to compile pybind with -fno-exceptions.

In file included from include/pybind11/pytypes.h:12,
                 from include/pybind11/cast.h:13,
                 from include/pybind11/attr.h:13,
                 from include/pybind11/pybind11.h:44,
                 from foo.cpp:1:
include/pybind11/detail/common.h: In function ‘void pybind11::pybind11_fail(const char*)’:
include/pybind11/detail/common.h:702:111: error: exception handling disabled, use ‘-fexceptions’ to enable
  702 | [[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); }
      |                                                                                                               ^
In file included from foo.cpp:1:
include/pybind11/pybind11.h: In static member function ‘static PyObject* pybind11::cpp_function::dispatcher(PyObject*, PyObject*, PyObject*)’:
include/pybind11/pybind11.h:663:11: error: ‘...’ handler must be the last handler for its try block [-fpermissive]
  663 |         } catch (error_already_set &e) {
      |           ^~~~~
In file included from include/pybind11/pytypes.h:12,
                 from include/pybind11/cast.h:13,
                 from include/pybind11/attr.h:13,
                 from include/pybind11/pybind11.h:44,
                 from foo.cpp:1:
include/pybind11/detail/common.h: In function ‘void pybind11::pybind11_fail(const char*)’:
include/pybind11/detail/common.h:702:114: warning: ‘noreturn’ function does return
  702 | [[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); }
      |                                                                                                                  ^
include/pybind11/detail/common.h: In function ‘void pybind11::pybind11_fail(const string&)’:
include/pybind11/detail/common.h:703:121: warning: ‘noreturn’ function does return
  703 | [[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); }
      |                                                                                                                         ^

wjakob · 2019-06-10T19:28:27Z

Hi @av8ramit,

apologies for the long delay -- this is a side project of mine, and my time for supporting it is limited.

I'm not super-excited about this PR: exceptions are really central to the internals of pybind11. They can of course be commented out like what is done here, but this will break any kind of error handling in software using pybind11. I'll close this for now but am open to further discussion.

Best,
Wenzel

martinwicke · 2019-06-11T23:31:33Z

@wjakob thanks for taking a look!

The purpose of the PR is to allow users to use pybind who for unrelated reasons cannot use C++ exceptions. Case in point, google compiles with -fno-exceptions as a matter of policy. I don't want take any position on the wisdom of this policy, but without this or a similar patch, we cannot use pybind at Google (we could fork, but I don't like forks if we can avoid them).

I believe that pybind provides huge value, and I would like to be able to use it, and in the environment that we're in, I do not have to worry about exceptions (since they don't exist). This is true for anyone who would be using -fno-exceptions. I don't think this would break error handling, since by definition users using -fno-exceptions would not be able to create exceptions. It is very possible that I'm missing something here.

We figured this would be the minimally intrusive way to make pybind work in an environment without exceptions. If you have suggestions that would make this more palatable, we're open to any sort of changes.

Thank you!

wjakob · 2019-06-11T23:40:18Z

Hi @martinwicke,

This PR simply turns off exceptions but punts on addressing the underlying problem, which is fundamental: Python makes copious use of exceptions — not just explicit ones but also implicit ones e.g. when querying an iterator. All of these exceptions need to be converted into something meaningful when crossing the Python -> C++ boundary. If exceptions are forbidden, what is pybind11 to do? The old school thing would be to use return values or manual error checks everywhere, but that is far too intrusive of a change to both pybind11 and projects using pybind11.

FWIW Compiling with -fno-exception as a policy strikes me as pretty terrible default choice, especially for binding code that needs to interoperate with Python.

Best,
Wenzel

martinwicke · 2019-06-12T15:00:10Z

Yeah -fno-exceptions isn't my first choice either.

Your point is very valid. Right now, IIUC, any Python exception that gets exposed to C++ code via tha patched pybind would simply terminate. I agree stringing error handling code through everything is not desirable.

Basically, the way that it is now (modulo bugs that need to be fixed) would be that crippling C++ by removing exceptions means that it is now illegal for an exception raised in python to be exposed to C++ via pybind (if that happens, the program terminates). I think this is pretty consistent. I understand that limits how good the Python integration can be (e.g., we won't be writing C++ code handling iterators), but it is still a big value-add to be able to conveniently expose C++ classes/functions to Python.

wjakob · 2019-06-12T16:01:10Z

Terminating the application when a Python exception reaches C++ sounds absolutely nuts to me! I think you should give some thought on how to get out of this situation that an unreasonable policy is forced on to you (rather than pushing it to dependency libraries). If your only recourse is to modify pybind11 in this way, then I would rather you did so in a fork.

bstaletic · 2019-06-12T16:34:03Z

I have demonstrated what happens if a python exception reaches C++ when importing the module - it ends up calling std::terminate().

What happens when importing doesn't fail, but then python layer calls a C++ function?

foo.cpp

#include "include/pybind11/pybind11.h"

void f() {
        pybind11::module x = pybind11::module::import("boom");
}

PYBIND11_MODULE(foo, m) {
        m.def("f", f);
}

boom.py

raise RuntimeError

Calling f() results in:

>>> import foo
>>> foo.f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: <built-in method f of PyCapsule object at 0x7f83e3d85d50> returned NULL without setting an error
>>>

So it's not always clear what happens when a python exception reaches C++.

Amit Patankar added 3 commits February 20, 2019 11:47

Wrapping exception logic in #ifdefs to allow for compilation without …

cfb7185

…exceptions.

Moving the Py_VersionCheckPassed function to the pybind11 namespace.

453663e

Making the version function an inline function and adding it to the d…

d642120

…etail namespace.

Amit Patankar added 2 commits April 25, 2019 16:49

Trigger CI.

4fa14c5

Test presubmits again.

5540fad

bstaletic reviewed Apr 26, 2019

View reviewed changes

Amit Patankar added 7 commits April 26, 2019 17:21

Wrapping exception logic in #ifdefs to allow for compilation without …

4f49b32

…exceptions.

Moving the Py_VersionCheckPassed function to the pybind11 namespace.

ed773c1

Making the version function an inline function and adding it to the d…

65dca2c

…etail namespace.

Merge branch 'noexceptions' of github.com:av8ramit/pybind11 into noex…

e3c54bc

…ceptions

Merge remote-tracking branch 'upstream/master' into noexceptions

f58351a

Merging the latest changes.

Applying the no exception logic to the new plugins and extending it t…

7184def

…o the deprecated PYBIND11_PLUGIN as well.

Removing unecessary macro.

4a3ef83

bstaletic reviewed May 31, 2019

View reviewed changes

Amit Patankar added 2 commits June 7, 2019 11:07

Adding a changelog.

50d0473

Add a minor note to the changelog.

6348def

wjakob closed this Jun 10, 2019

superbobry mentioned this pull request Aug 28, 2019

error_already_set::what() is now constructed lazily #1895

Merged

rwgk mentioned this pull request Feb 10, 2023

FWD pybind11 google/pybind11clif#1703

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compile without exceptions #1703

Compile without exceptions #1703

av8ramit commented Feb 20, 2019 •

edited

Loading

av8ramit commented Feb 21, 2019

av8ramit commented Feb 25, 2019

av8ramit commented Mar 26, 2019

bstaletic left a comment

bstaletic Apr 26, 2019

av8ramit May 30, 2019

bstaletic Apr 26, 2019

av8ramit May 30, 2019

av8ramit commented May 31, 2019

bstaletic left a comment

bstaletic May 31, 2019

av8ramit Jun 7, 2019

bstaletic Jun 7, 2019

av8ramit Jun 7, 2019

bstaletic commented Jun 10, 2019

wjakob commented Jun 10, 2019

martinwicke commented Jun 11, 2019

wjakob commented Jun 11, 2019

martinwicke commented Jun 12, 2019

wjakob commented Jun 12, 2019

bstaletic commented Jun 12, 2019

Compile without exceptions #1703

Compile without exceptions #1703

Conversation

av8ramit commented Feb 20, 2019 • edited Loading

av8ramit commented Feb 21, 2019

av8ramit commented Feb 25, 2019

av8ramit commented Mar 26, 2019

bstaletic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

av8ramit commented May 31, 2019

bstaletic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bstaletic commented Jun 10, 2019

wjakob commented Jun 10, 2019

martinwicke commented Jun 11, 2019

wjakob commented Jun 11, 2019

martinwicke commented Jun 12, 2019

wjakob commented Jun 12, 2019

bstaletic commented Jun 12, 2019

av8ramit commented Feb 20, 2019 •

edited

Loading