Migrating project to C++ and Python

[paulocoutinhox]

Hi,

First, nice project 👍

Number 1:
How i can stop/terminate a python execution?

I want migrate my CI project (goci) from Go and JS to C++ and Python.

But i need some way to stop/terminate the python execution, because it will run in a thread and the user can stop that execution.

Number 2:
Basically my current github project (goci) work in this way. A go program control tasks that is javascript files executed from it. From go i can expose some functions to javascript (change progress, send message to console, set status etc). After finished that thread dead and the log, status, progress is saved to history.

I read about this project and see that i can make the same thing. I want only confirm it please.

Thanks.

Thanks.

[YannickJadoul]

@prsolucoes Can you expand on the first item? First of all, are you writing an extension module, or embedding the interpreter in an executable? Can you sketch what kind of "python execution you want to stop/terminate"?

I'm also not sure I understand your problem. You want to run the Python interpreter from your own program?

[paulocoutinhox]

Hi,

Is better my current project:
https://github.com/prsolucoes/goci

The problem with my current project is that i need export all functions from go to js (download, create dir, read file etc).

So after i search for something that i can use python and its libraries i found your project.

Then i decide make the same approach, but using c++ and python instead of go and javascript.

The project is a continuous integration system and the user can start tasks, that is a javascript file.

I want make the same thing but using c++ and python. I will embed all necessary things in an executable and will execute the python file associated with that "task".

After read your examples and code i see that the only thing that i can't do is stop a python execution. Example: My file build_android.py" will start "gradlew buildRelease" and after show a success message. But after the user start the task they forgot something and want stop this task, so they only need press stop button and internally i will stop that python execution and gradlew build process will stop too.

I will start the code using this example:
https://pybind11.readthedocs.io/en/stable/advanced/embedding.html

#include <pybind11/embed.h>
#include <iostream>

namespace py = pybind11;
using namespace py::literals;

int main() {
    py::scoped_interpreter guard{};

    auto locals = py::dict("name"_a="World", "number"_a=42);
    py::exec(R"(
        message = "Hello, {name}! The answer is {number}".format(**locals())
    )", py::globals(), locals);

    auto message = locals["message"].cast<std::string>();
    std::cout << message;
}

1 - I want know if i can stop it (py:exec) any time

2 - I want know if i can use any installed module with pip

Thanks.

[paulocoutinhox]

Hi,

@YannickJadoul , did you my message?

Thanks.

[YannickJadoul]

@prsolucoes Yes, I did, but it's been a busy week.

I want know if i can stop it (py:exec) any time

If I understand correctly, you want something like this? https://docs.python.org/3/c-api/exceptions.html#c.PyErr_SetInterrupt
(found through https://stackoverflow.com/questions/54441587/how-to-interrupt-python-interpreter-embedded-in-c-application)

I want know if i can use any installed module with pip

Should be, yes, if you install it to the version of Python that you're embedding. Try it out, I'd say, and you'll know.

[paulocoutinhox]

Hi,

Im trying execute a sample application with pybind11 and python 3.9:

I made a simple project that will compile with Github Actions:
https://github.com/prsolucoes/pybind11-test

In getting this error:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/Users/runner/work/pybind11-test/pybind11-test/build/./pybind11_test'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/runner/work/pybind11-test/pybind11-test/build/./pybind11_test'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python39.zip',
    '/usr/local/lib/python3.9',
    '/usr/local/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

https://github.com/prsolucoes/pybind11-test/runs/1393534782?check_suite_focus=true#step:11:29

Can you help me understand it?

Thanks.

[YannickJadoul]

Well, start out by not doing this all manually: paulocoutinhox/pybind11-test@e53bd82#diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20a

There's a whole section in the docs on how to embed the interpreter, which includes a minimal example of a build system: https://pybind11.readthedocs.io/en/stable/advanced/embedding.html

I'd start simple, with that, before doing complicated stuff and manually building against Python.

[YannickJadoul]

I'd start simple, with that, before doing complicated stuff and manually building against Python.

To be clear: most likely, something goes wrong there, which make Python not find those correct library locations.

[paulocoutinhox]

Hi,

1 - I dont need download python specific version?
2 - That embed example work on any system (windows, linux and macos)?
3 - I made pybind11 update to conan:
conan-io/conan-center-index#3600

Thanks.

[YannickJadoul]

1 - I dont need download python specific version?

I don't know what you mean. The version of Python you link against is used. Test out the example and play around with it, I'd say.

2 - That embed example work on any system (windows, linux and macos)?

Not sure what you mean again. Yes, the example in the documentation work. Just try it out.
Of course you'll have to recompile for every platform.

3 - I made pybind11 update to conan:
conan-io/conan-center-index#3600

Thanks! Not sure how that's related to this issue, and I'm not very familiar with conan, but good to see the latests update being used in package managers.