Use pybind11 in ttconv module #25253

thomasjpfan · 2023-02-18T20:34:06Z

PR Summary

This PR migrates the ttconv module to using pybind11. Most of the conversion uses pybind11 data structures. Some notes for reviewers:

The Python C-API was required to converting bytes into Unicode: pybind11 reference.
pybind11::handle is a thin wrapper around PyObject * that does not perform any reference counting.

github-actions

Thank you for opening your first PR into Matplotlib!

If you have not heard from us in a while, please feel free to ping @matplotlib/developers or anyone who has commented on the PR. Most of our reviewers are volunteers and sometimes things fall through the cracks.

You can also join us on gitter for real-time discussion.

For details on testing, writing docs, and our review process, please see the developer guide

We strive to be a welcoming and open project. Please follow our Code of Conduct.

oscargus · 2023-02-20T08:11:23Z

src/_ttconv.cpp

-            Py_DECREF(result);
+        PyObject* decoded = PyUnicode_DecodeLatin1(a, strlen(a), "");
+        if (decoded == NULL) {
+            throw py::exception();


Thank you very much for this! There are more suitable reviewers than me (@ianthomas23 and @anntzer comes to mind), but I think it would be good to replace this exception with "something else" (to get rid of the py_exceptions.h import, which we want to get rid of as part of the PyBind11 conversion). I think that std::exception() is the way to go, possibly with some sensible error message. Although, of course, that general exception is not good etc, but at least it will be consistent with the current behavior.

(This exception is not tested in the original code either.)

This should be py::error_already_set (https://pybind11.readthedocs.io/en/stable/advanced/exceptions.html#handling-exceptions-from-python-in-c).

Agreed this should be made to work without py_exceptions.h.

We should probably switch the whole thing to write to binary streams (and do something like fh.flush(); convert_ttf_to_ps(..., fh.buffer, ...); fh.buffer.flush() on the Python calling side), but that can be another PR.

I switched to using error_already_set. Note that this PR is keeping the original behavior on main which catches this exception and ignores it:

matplotlib/src/_ttconv.cpp

Lines 139 to 142 in de8e9d0

catch (const py::exception &)

{

return NULL;

}

catches this exception and ignores it

No, returning NULL is the way for C-API functions to signal that an exception has been set and should be propagated to python-land...

Ah I see. I misinterpreted the original implementation. Thank you for the explanation!

Missed something.

anntzer · 2023-02-20T22:31:22Z

src/_ttconv.cpp

+        pybind11::arg("filename"),
+        pybind11::arg("output"),
+        pybind11::arg("fonttype"),
+        pybind11::arg("glyph_ids"),


default value for glyph_ids is missing.

anntzer · 2023-02-20T22:34:34Z

src/_ttconv.cpp

+    const char *filename,
+    pybind11::object &output,
+    int fonttype,
+    pybind11::iterable glyph_ids)


Just make this argument a std::vector<int> and don't bother with doing the conversion yourself?

With the default value of None, I think this needs to be pybind11::iterable*, to account for the nullptr. If we need to use std::vector<int>, then we need std::optional, which requires C++17:

XREF: The note section from: https://pybind11.readthedocs.io/en/stable/advanced/functions.html#allow-prohibiting-none-arguments

anntzer · 2023-02-20T22:56:09Z

Frankly, I would not bother with changing ttconv; I'd rather just see some work on reviving #20866 to move everything to fonttools (which we already did for pdf in #20391).

oscargus · 2023-02-21T07:48:15Z

I would not bother with changing ttconv;

Although I agree in principle, the question is if/when that will happen... If it is before 3.8 is released (i.e., in July(?)), the current PR may not be worthwhile, but considering that it has been around for 14 months it is not obvious that it will happen.

Anyway, good that you bring it up, so your call @thomasjpfan . We appreciate your work, but there is a risk that it will be in vain in the sense that it will be removed before the next release. On the other hand, I still think it is worthwhile to merge this if finished as one do not know if it will be replaced and there are only details remaining. Sorry we didn't make it clear in #23846 that for this particular module there was another possible route.

thomasjpfan · 2023-02-21T17:45:57Z

but there is a risk that it will be in vain in the sense that it will be removed before the next release.

I am okay with the changes made in this PR being removed because of #20866.

On the other hand, I still think it is worthwhile to merge

I agree that it makes sense to move forward with this PR anyways to get the benefits of using pybind11 as stated in #23846.

ianthomas23

I've left a few comments but I'd be happy for this to merged as it is.

src/_ttconv.cpp

ianthomas23 · 2023-02-23T10:34:59Z

src/_ttconv.cpp

-    catch (const py::exception &)
-    {
-        return NULL;
+        throw std::runtime_error(e.getMessage());


I am concerned that we don't exercise this line in the test suite as it would be good to confirm that TTExceptions propagate through pybind11 as expected. But given that we've evidently never had such a test it doesn't have to be done here, it can be punted to "future work".

ianthomas23 · 2023-02-23T10:37:30Z

src/_ttconv.cpp

    }
    catch (...)
    {
-        PyErr_SetString(PyExc_RuntimeError, "Unknown C++ exception");
-        return NULL;
+        throw std::runtime_error("Unknown C++ exception");


This catch all isn't really needed as pybind11 catches C++ exceptions and converts them to Python ones. But I like the approach that support for pybind11 should happen with minimal changes to C++ code so it is fine to leave it as it is.

oscargus · 2023-03-30T14:02:55Z

I fixed the merge conflict (in the browser...). Is there anything left here @anntzer? (Now, assuming that we are not fully sure that #20866 will be implemented in time for 3.8 and that we want to swap to PyBind11 "ASAP".)

anntzer · 2023-03-30T15:25:54Z

From a quick look it seems fine. Let's just get this in modulo CI.

oscargus · 2023-03-30T20:55:44Z

Thanks @thomasjpfan !

github-actions bot reviewed Feb 18, 2023

View reviewed changes

oscargus reviewed Feb 20, 2023

View reviewed changes

oscargus added the topic: text/fonts label Feb 20, 2023

anntzer mentioned this pull request Feb 20, 2023

[MNT]: Make _ttconv.convert_ttf_to_ps take a binary file as output argument, not a text file #25265

Open

thomasjpfan force-pushed the ttconv_pybind11_pr branch 3 times, most recently from a53da51 to f14a566 Compare February 20, 2023 15:57

jklymak requested a review from anntzer February 20, 2023 22:21

anntzer previously approved these changes Feb 20, 2023

View reviewed changes

anntzer reviewed Feb 20, 2023

View reviewed changes

jklymak added the status: needs revision label Feb 20, 2023

thomasjpfan force-pushed the ttconv_pybind11_pr branch from f14a566 to b074140 Compare February 21, 2023 17:35

thomasjpfan force-pushed the ttconv_pybind11_pr branch from b074140 to 8da08a7 Compare February 23, 2023 03:21

ianthomas23 approved these changes Feb 23, 2023

View reviewed changes

thomasjpfan force-pushed the ttconv_pybind11_pr branch from 8da08a7 to dcb5e31 Compare February 23, 2023 14:54

Use pybind11 in ttconv module

c38348b

thomasjpfan force-pushed the ttconv_pybind11_pr branch from dcb5e31 to c38348b Compare February 24, 2023 11:54

github-actions bot added the status: needs rebase label Mar 25, 2023

Merge branch 'main' into ttconv_pybind11_pr

3205f85

github-actions bot removed the status: needs rebase label Mar 30, 2023

anntzer approved these changes Mar 30, 2023

View reviewed changes

oscargus merged commit e5fc57c into matplotlib:main Mar 30, 2023
37 of 38 checks passed

tacaswell added this to the v3.8.0 milestone Mar 30, 2023

tacaswell removed the status: needs revision label Mar 30, 2023

oscargus mentioned this pull request Mar 30, 2023

[MNT]: Pybind11 transition plan #23846

Open

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use pybind11 in ttconv module #25253

Use pybind11 in ttconv module #25253

thomasjpfan commented Feb 18, 2023

github-actions bot left a comment

oscargus Feb 20, 2023 •

edited

anntzer Feb 20, 2023 •

edited

thomasjpfan Feb 20, 2023

anntzer Feb 20, 2023 •

edited

thomasjpfan Feb 20, 2023

anntzer Feb 20, 2023

anntzer Feb 20, 2023 •

edited

thomasjpfan Feb 21, 2023

anntzer commented Feb 20, 2023

oscargus commented Feb 21, 2023

thomasjpfan commented Feb 21, 2023

ianthomas23 left a comment

ianthomas23 Feb 23, 2023

ianthomas23 Feb 23, 2023

oscargus commented Mar 30, 2023

anntzer commented Mar 30, 2023

oscargus commented Mar 30, 2023

Use pybind11 in ttconv module #25253

Use pybind11 in ttconv module #25253

Conversation

thomasjpfan commented Feb 18, 2023

PR Summary

github-actions bot left a comment

Choose a reason for hiding this comment

oscargus Feb 20, 2023 • edited

Choose a reason for hiding this comment

anntzer Feb 20, 2023 • edited

Choose a reason for hiding this comment

thomasjpfan Feb 20, 2023

Choose a reason for hiding this comment

anntzer Feb 20, 2023 • edited

Choose a reason for hiding this comment

thomasjpfan Feb 20, 2023

Choose a reason for hiding this comment

anntzer Feb 20, 2023

Choose a reason for hiding this comment

anntzer Feb 20, 2023 • edited

Choose a reason for hiding this comment

thomasjpfan Feb 21, 2023

Choose a reason for hiding this comment

anntzer commented Feb 20, 2023

oscargus commented Feb 21, 2023

thomasjpfan commented Feb 21, 2023

ianthomas23 left a comment

Choose a reason for hiding this comment

ianthomas23 Feb 23, 2023

Choose a reason for hiding this comment

ianthomas23 Feb 23, 2023

Choose a reason for hiding this comment

oscargus commented Mar 30, 2023

anntzer commented Mar 30, 2023

oscargus commented Mar 30, 2023

oscargus Feb 20, 2023 •

edited

anntzer Feb 20, 2023 •

edited

anntzer Feb 20, 2023 •

edited

anntzer Feb 20, 2023 •

edited