Document how to embed Cython modules in C/C++ applications

[scoder]

The documentation on embedding is inherently lacking. Even the one in the CPython docs is incomplete at best.

There are some hints in the Wiki, there's the --embed option to the cython and cythonize frontends, the embedding demo, the PyImport_AppendInittab() function in CPython and the documentation of the module init function in CPython.

All of these are incomplete and none of them gives the whole picture in one place. There should be a dedicated documentation page in the Cython docs for this, and the documentation in the CPython docs is also worth improving.

[scoder]

I created a stub page in https://github.com/cython/cython/blob/master/docs/src/tutorial/embedding.rst – improvements welcome!

[brightening-eyes]

hi,
the distribution of the embedded application should be documented as well, like how python modules are searched, in order to copy them beside the compiled executable. something like wxpython for example.

[scoder]

Sure, investigation and PRs welcome.

[jokoon]

https://github.com/cython/cython/blob/master/Demos/embed/Makefile#L38

I think this embedded.c file got removed...

[da-woods]

@jokoon No - it's created by Cythonizing the embedded.pyx file, 4 lines later:

cython/Demos/embed/Makefile

Lines 42 to 43 in 3ee0662

	embedded.c: embedded.pyx
	@$(PYTHON) $(CYTHON) --embed embedded.pyx

[jokoon]

Oh ok.

I'm still unsure if that .msc can be used in visual studio.

I'm also confused if a cython module requires compilation each time it's edited, or if cython effectively still interprets a .py script at runtime though python runtime. I'm guessing cython acts as some kind of glue, but I'm not sure I understand.

The context being I want to use python a scripting language for a game, and I don't know if cython can help for this or not.

[TeamSpen210]

Cython translates your code into a set of C functions which use Python's C-API to do all the operations. Cython doesn't really help in executing scripts at runtime, though you could use it to make the extension modules to provide access to your game in Python code. You'd really want to just use eval(), or the direct C-API function which does the same.

[scoder]

Please don't use the bug tracker for general discussions. We have the cython-users mailing list for this.

[da-woods]

FWIW I have a working example of how to call PyInit_* on a multi-phase init module therefore avoiding the need to use PyImport_AppendInittab. Essentially it's running through the multi-phase init steps manually. I'm not sure if we want to put it in any updated embedding docs, but it's posted below in case anyone wants to document it properly:

I suspect this comes under

Most importantly, DO NOT call the module init function instead of importing the module. This is not the right way to initialise an extension module. (It was always wrong but used to work before, but since Python 3.5, it is wrong and no longer works.)

though

#include <Python.h>

#include "embed_example.h"

int main() {
    int result = 1;
    PyObject *spec = NULL, *spec_globals = NULL, *mod = NULL;
    Py_Initialize();
    PyObject *maybe_mod = PyInit_embed_example();
    if (!maybe_mod) goto bad;
    if (Py_IS_TYPE(maybe_mod, &PyModuleDef_Type)) {
        // multi-phase init
        spec_globals = PyDict_New();
        if (!spec_globals) goto bad;
        PyObject *res = PyRun_String(
            "import importlib.machinery as im\n"
            // Note that Cython doesn't actually use the loader
            // so it can be None. It'd be better to
            // provide something more useful though.
            "spec = im.ModuleSpec('embed_example', None)\n",
            Py_file_input, spec_globals, spec_globals);
        Py_XDECREF(res); // don't use res whether or not it's set
        if (!res) goto bad;
        spec = PyDict_GetItemString(spec_globals, "spec");
        if (!spec) goto bad;               
        
        mod = PyModule_FromDefAndSpec(
            (PyModuleDef*)maybe_mod,
            spec);
        if (!mod) goto bad;
        int execRes = PyModule_ExecDef(mod, (PyModuleDef*)maybe_mod);
        if (execRes) goto bad;
    } else {
        mod = maybe_mod;
    }
    
    func(); // cdef public in the module being embedded.
    
    result = 0;
    if (0) {
        bad:
        PyErr_Print();
    }
    // The moduledef isn't an owned reference so doesn't get decref'd
    Py_XDECREF(mod);
    Py_XDECREF(spec);
    Py_XDECREF(spec_globals);
    Py_Finalize();
    return result;    
}

I remain sceptical of whether embedding Cython in C/C++ applications is actually useful (because people expect it to embed their module and all its dependencies), but multi-phase init is definitely something that people fight with, and additional ways to work with it are probably good.

[sohampirale]

Is this open?
Can I work on this?

[da-woods]

@sohampirale yes I think the documentation here could still be improved. I couldn't tell you exactly what needs doing though