Interoperability with cython

[ianhbell]

I am sure I am not the only one, but I have historically developed a number of interlinking packages based on cython interfaces, cython headers, etc.

Having discovered pybind11, I am totally enamored. I especially like how wrapping a class to python involves just a single C++ method definition if you are lucky, and you are good to go. I've been doing lots of work with pybind11 recently.

I especially like how easy it is to drop into a visual studio/Xcode debugger inside the C++ code called from the python module. That has drastically improved my compile-debug-compile cycle throughput. Previously it was all just print commands in my cython code (yuck!)...

But the crux of my problem is that I would like to switch over my API to use pybind11 instead of cython while retaining the ability to make the module(s) interoperable with cython. I know this is a big ask, but I thought I would put it out there as something I am very interested in and would like to know if there is any way to make it work.

[aldanor]

Do you have a particular (perhaps hypothetical) example in mind? It would be a lot easier to discuss a specific case as opposed to pb11 vs cython in general.

[ianhbell]

Here is a concrete (and rather simple, at least for cython) example: https://github.com/CoolProp/CoolProp/tree/master/wrappers/Python . The .pyx and .pxd files are in the CoolProp folder. This is an open-source thermophysical property library that I use extensively in my research (I was the primary developer of CoolProp, so I know its guts pretty well). I've got several projects ongoing that make use of CoolProp.

This cython module for CoolProp is linked into another tool I developed: https://github.com/ibell/pdsim.

Core desire: I would like to switch CoolProp over completely to using pybind11, but not have to change the interface of pdsim one iota.

[wjakob]

I'm not sure I fully understand the problem (I don't have any experience with Cython).

Types bound with pybind11 are just regular Python types with native functions wired up to functions instead of byte code. What framework was used to create particular bindings shouldn't matter as long as the implementations behave as expected as far as Duck typing is concerned. For instance, I'm sure that pybind11 can make calls to Cython-bound functions without problems. Why it wouldn't work the other way?

[aldanor]

I would like to switch CoolProp over completely to using pybind11, but not have to change the interface of pdsim one iota.

Do you mean "not have to change the Cython interface"? So you'd like to replace CoolProp with a pybind11-based library, but have pdsim expose everything via Cython?

[wjakob]

Closing this due to lack of a response. Feel free to comment though, and we can reopen the ticket.

[fried]

For future explorers
The issue is cython name mangles its cdefs so from c++ you can't just call cython functions.
The solution is to use Cython API, that way in imports the python module and exposes the c++ functions as human readable names.

[ianhbell]

I’m a future explorer that has been entirely flummoxed by trying to hook pybind11 and cython together. I’m now pretty invested in both cython and pybind11, but so far have not been able to get them to seamlessly interoperate. Could you provide a MWE of interoperability of cython and pybind11? If so, I’d be forever appreciative. From: Jason Fried [mailto:notifications@github.com] Sent: Thursday, March 16, 2017 7:11 PM To: pybind/pybind11 <pybind11@noreply.github.com> Cc: Bell, Ian (Fed) <ian.bell@nist.gov>; Author <author@noreply.github.com> Subject: Re: [pybind/pybind11] Interoperability with cython (#522) For future explorers The issue is cython name mangles its cdefs so from c++ you can't just call cython functions. The solution is to use Cython API, that way in imports the python module and exposes the c++ functions as human readable names. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#522 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALaZLT-C8SsMY4RTB6wBi67Mn8IYMhWEks5rmd20gaJpZM4K522v>.

[fried]

if you mark a cdef "api" like cdef api void callback(int num): cython will generate a module_api.h file that you can include in your c++ that include will provide the following function import_package__module() it uses __ underscores instead of '.' but its the python import path. after you do that you get a callback function

…

On Fri, Mar 17, 2017 at 08:57 Ian Bell ***@***.***> wrote: I’m a future explorer that has been entirely flummoxed by trying to hook pybind11 and cython together. I’m now pretty invested in both cython and pybind11, but so far have not been able to get them to seamlessly interoperate. Could you provide a MWE of interoperability of cython and pybind11? If so, I’d be forever appreciative. From: Jason Fried ***@***.*** Sent: Thursday, March 16, 2017 7:11 PM To: pybind/pybind11 ***@***.***> Cc: Bell, Ian (Fed) ***@***.***>; Author ***@***.***> Subject: Re: [pybind/pybind11] Interoperability with cython (#522) For future explorers The issue is cython name mangles its cdefs so from c++ you can't just call cython functions. The solution is to use Cython API, that way in imports the python module and exposes the c++ functions as human readable names. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub< #522 (comment)>, or mute the thread< https://github.com/notifications/unsubscribe-auth/ALaZLT-C8SsMY4RTB6wBi67Mn8IYMhWEks5rmd20gaJpZM4K522v>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#522 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMPCZyNKbG20dGU8Kd9_A3koybdDI4Hks5rmq1igaJpZM4K522v> .

[ianhbell]

Brilliant - I’ll experiment with this! How about the other way around (calling pybind11 interface classes from cython)? From: Jason Fried [mailto:notifications@github.com] Sent: Saturday, March 25, 2017 9:10 PM To: pybind/pybind11 <pybind11@noreply.github.com> Cc: Bell, Ian (Fed) <ian.bell@nist.gov>; Author <author@noreply.github.com> Subject: Re: [pybind/pybind11] Interoperability with cython (#522) if you mark a cdef "api" like cdef api void callback(int num): cython will generate a module_api.h file that you can include in your c++ that include will provide the following function import_package__module() it uses __ underscores instead of '.' but its the python import path. after you do that you get a callback function

On Fri, Mar 17, 2017 at 08:57 Ian Bell ***@***.******@***.***>> wrote: I’m a future explorer that has been entirely flummoxed by trying to hook pybind11 and cython together. I’m now pretty invested in both cython and pybind11, but so far have not been able to get them to seamlessly interoperate. Could you provide a MWE of interoperability of cython and pybind11? If so, I’d be forever appreciative. From: Jason Fried ***@***.*** Sent: Thursday, March 16, 2017 7:11 PM To: pybind/pybind11 ***@***.******@***.***>> Cc: Bell, Ian (Fed) ***@***.******@***.***>>; Author ***@***.******@***.***>> Subject: Re: [pybind/pybind11] Interoperability with cython (#522) For future explorers The issue is cython name mangles its cdefs so from c++ you can't just call cython functions. The solution is to use Cython API, that way in imports the python module and exposes the c++ functions as human readable names. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub< #522 (comment)>, or mute the thread< https://github.com/notifications/unsubscribe-auth/ALaZLT-C8SsMY4RTB6wBi67Mn8IYMhWEks5rmd20gaJpZM4K522v>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#522 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMPCZyNKbG20dGU8Kd9_A3koybdDI4Hks5rmq1igaJpZM4K522v> .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#522 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALaZLQBSHfruB_xWKujbMdcu2hwhPTEQks5rpdbygaJpZM4K522v>.

[davidalbertonogueira]

I've discovered this project now (pybind11), and I am trying to understand what it offers that cython doesn't. Moreover, I couldn't find a single mention/reference to cython in the read me description.
So, here I am, searching issues with the keyword 'cython', that can shed some light.

Btw,
@fried , what do you mean with:

The issue is cython name mangles its cdefs so from c++ you can't just call cython functions.

I've been using cython in some projects, and I can definitely define objects and functions both in c++ and in python and call them from the other side:
C++ calling python:
https://github.com/Priberam/TurboTextAnalysis/blob/master/TurboTextAnalysisPython/CppToPyTurboSink.cpp#L22
python calling c++:
https://github.com/Priberam/TurboTextAnalysis/blob/master/TurboTextAnalysisPython/turboparser.pyx#L234

[fried]

well you are calling cython defs using the python-c api. Well that api won't allow you to call cython cdefs since they are just c/c++ methods and not callable from python. You can directly call them from c++ if you know its name which is this issue. cython will generate a name like _pyx_blah_blah_real_name to prevent name collisions. the 'api' keyword i was talking about produces a .h files to help. It also takes care of importing the symbols dynamically using the python import system

…

On Sun, Jul 16, 2017 at 04:50 David Nogueira ***@***.***> wrote: I've discovered this project now (pybind11), and I am trying to understand what it offers that cython doesn't. Moreover, I couldn't find a single mention/reference to cython in the read me description. So, here I am, searching issues with the keyword 'cython', that can shed some light. Btw, @fried <https://github.com/fried> , what do you mean with: The issue is cython name mangles its cdefs so from c++ you can't just call cython functions. I've been using cython in some projects, and I can definitely define objects and functions both in c++ and in python and call them from the other side: C++ calling python: https://github.com/Priberam/TurboTextAnalysis/blob/master/TurboTextAnalysisPython/CppToPyTurboSink.cpp#L22 python calling c++: https://github.com/Priberam/TurboTextAnalysis/blob/master/TurboTextAnalysisPython/turboparser.pyx#L234 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#522 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMPCegZ34V3Cw7ceBz0dJ2EBrcoBkuPks5sOfj7gaJpZM4K522v> .

[cvanelteren]

Apologies for reviving this old thread. @fried that api keyword is a neat feature, however this does not work for extension classes, is there a similar trick for cdef classes?