Python exception translation

[foundrytom]

What

Ensure information is not lost from Python exceptions when bridged to C++

Also see #866 (duplicate issue)

Why

We currently lose information about Python exceptions when caught in C++.

When Python code calls a C++ function, and that C++ function raises an exception, there is a built-in hook via pybind11 to translate the C++ exception to a Python exception (see usages of register_exception_translator).

However, when C++ calls a Python function, and that Python function raises an exception, pybind11 will detect the Python exception and throw an (unhelpfully named) error_already_set exception. There is no pybind11 hook available to modify this.

In this case, consumers (hosts or manager plugins) must catch std::exception and interrogate the what() string to disambiguate.

We really need to devise a way to translate Python exceptions to the appropriate C++ exceptions, so that consumers can disambiguate using the exception type (i.e. in catch blocks).

An unfortunate wrinkle is in the special status that pybind11's error_already_set exception has within pybind11 itself. If this exception propagates back into Python, then the original Python exception is restored, complete with stack trace etc.

So ideally we would only conditionally translate an error_already_set exception into a more appropriate C++ exception, by somehow figuring out whether the exception will be caught in C++ or Python code.

ACs

• Define approach for the above

[feltech]

I'd imagine any pybind11 trampoline class (i.e. anywhere we have a Python class that can inherit from C++) needs to have its member functions wrapped in try/catch, plus any other functions where we call out to Python (e.g. createPythonPluginSystemManagerImplementationFactory)

I wonder if multiple inheritance could help us here.

class openassetio::PluginError : std::runtime_error { ... };

class openassetio::PyPluginError : openassetio::PluginError, py::error_already_set { ... };

void rethrowCppException(std::exception_ptr pexc) {
  try {
    std::rethrow_exception(std::move(pexc))
  } catch (py::error_already_set& exc) {
    // ... disambiguate ... then:
   throw PyPluginError{exc};  // Absorb `error_already_set` and construct `PluginError`.
  }
}

void initialize(InfoDictionary managerSettings, const HostSessionPtr& hostSession) override {
  try {
    PYBIND11_OVERRIDE_PURE(void, ManagerInterface, initialize, std::move(managerSettings),
                           hostSession);
  } catch (...) {
    rethrowCppException(std::current_exception());
  }
}

Then C++ consumers can catch PluginError, and Python consumers will get the original Python exception. This might also work with catching in C++ then rethrowing back to Python.

[foundrytom]

Dont know if its relevant, but some prior art here too https://github.com/ImageEngine/cortex/blob/main/include/IECorePython/ExceptionAlgo.h

[feltech]

This issue duplicates #866. Closed that other issue in favour of this one.

[feltech]

Looking specifically at the project linked above, ImageEngine's Cortex:

(Boost.Python) error_already_set catches are found around most of the Boost equivalent of pybind11's trampoline methods, translated to either a generic exception or using an exception_ptr stashed on the Python exception object.

[feltech]

Quick Compiler Explorer mockup of the sketch I did above, seems to be viable.