Pybind11 extended functions and classes support in no python mode

[sivansh11]

Feature request

Pybind11 extended functions and classes support in no python mode,
these are either made into their own binaries (modules) by pybind or are included while emmbdeding (PYBIND11_EMBEDDED_MODULE).

Why use something like this ?
For using python as a scripting language in game engines. Python is super easy to use by anyone, but every one quickly learns about it being slower than compiled languages by orders of magnitudess. But what if someone is able to compile python while being able to use the sea of libraries (my main usecase is with machine learning libraries) that are easily accessible.

I am extremely sorry for not following the proper github issue template, this is my first issue ever.
Please tell me what all I should add to the issue for it to be considered valid issue, I have done a basic search of pybind11 in the issue search bar and didnt get any related result.

[esc]

@sivansh11 thank you for raising this. I would like to clarify, what it is you are trying to access. Are you trying to use Python modules that were written with pybind11 from Numba? Or, are you trying to use the C/C++ libraries that were exposed with pybind11?

This sounds like the following use-case: https://numba.readthedocs.io/en/latest/user/cfunc.html?highlight=cfunc#calling-c-code-from-numba

[sivansh11]

@esc
Not exactly, I dont have a shared object that I can load in the script, the main executable itself has an embedded library (take a look at https://pybind11.readthedocs.io/en/stable/advanced/embedding.html#adding-embedded-modules).
I wanna be able to call these embedded functions and classes in nopython mode.

I may be wrong, but I did find this http://numba.pydata.org/numba-doc/0.50.0/extending/low-level.html#typing
from my understanding this is useful for creating custom numba understandable types ?

I couldn't get this to work, hence my post for adding a some form of pybind11 support so extending python and numba becomes easier and more performant.

If you need some more information, please feel free to ask, ill try to reply to everything.

[esc]

@sivansh11 ok, thank you for clarifying. This post on our Discourse may potentially be related: https://numba.discourse.group/t/how-to-extend-numba-so-it-supports-pybind11-wrapped-c-struct/1419

[esc]

@sivansh11 I am so sorry, Github showed me two copies of your latest post and after I tried to delete one of them it seems like Github has deleted both. This was a glitch and entirely unintentional, so sorry, please ask again.

[sivansh11]

@esc
It's no problem

I don't exactly remember what I wrote, but it was something like this

The link you shared was really helpful,
It showed me i have a bit of learning to do
(About numbas internals and llvm in general)

So just to test my understandings, the link explained how to "tell" numba about my function, what are the expected input parameters and it's types and the expected output parameters and it's types right ?

Also a follow up question, so technically the functions backend doesnt matter, it could be pure python (no decorator jit or njit) it could be pybind11 embedded etc and it would still work in the nopython mode ?

[esc]

@esc It's no problem

I don't exactly remember what I wrote, but it was something like this

The link you shared was really helpful, It showed me i have a bit of learning to do (About numbas internals and llvm in general)

So just to test my understandings, the link explained how to "tell" numba about my function, what are the expected input parameters and it's types and the expected output parameters and it's types right ?

Yes. You tell Numba about your native function, and then you can call it from Numba.

Also a follow up question, so technically the functions backend doesnt matter, it could be pure python (no decorator jit or njit) it could be pybind11 embedded etc and it would still work in the nopython mode ?

I am not sure exactly, but I don't think so.