Redirecting python output with embedded interpreter

[SunCat95]

I'm using pybind11 to execute python scripts inside a C++ dll with custom embedded modules, which is working perfectly, but I can't figure out how to redirect the python output globally.

Ideally I would like any output from python to be redirected as a string into a function.

Is this at all possible?

[madebr]

You can replace sys.stdout and sys.stderr by custom filelike object.

In this code, I replace them with io.StringIO objects.

class PyStdErrOutStreamRedirect {
    py::object _stdout;
    py::object _stderr;
    py::object _stdout_buffer;
    py::object _stderr_buffer;
public:
    PyStdErrOutStreamRedirect() {
        auto sysm = py::module::import("sys");
        _stdout = sysm.attr("stdout");
        _stderr = sysm.attr("stderr");
        auto stringio = py::module::import("io").attr("StringIO");
        _stdout_buffer = stringio();  // Other filelike object can be used here as well, such as objects created by pybind11
        _stderr_buffer = stringio();
        sysm.attr("stdout") = _stdout_buffer;
        sysm.attr("stderr") = _stderr_buffer;
    }
    std::string stdoutString() {
        _stdout_buffer.attr("seek")(0);
        return py::str(_stdout_buffer.attr("read")());
    }
    std::string stderrString() {
        _stderr_buffer.attr("seek")(0);
        return py::str(_stderr_buffer.attr("read")());
    }
    ~PyStdErrOutStreamRedirect() {
        auto sysm = py::module::import("sys");
        sysm.attr("stdout") = _stdout;
        sysm.attr("stderr") = _stderr;
    }
};

This class can be used as follows:

{
    PyStdErrOutStreamRedirect pyOutputRedirect{};
    py::print("hello world");
    // Other noisy python functions can be put here
    assert(pyOutputRedirect.stdoutString() == "hello world\n")
}
// sys.stdout is back to its original state

[bstaletic]

@madebr Thanks for answering. We can close the issue.

[asmaloney]

I realize this is an older issue, but when using pybind11 embedded, this would probably be a FAQ.

Should it be added to the docs? Maybe in the Utilities section with the other i/o issues?

[YannickJadoul]

Except that we're not really providing a utility class/function to do this :-/

This is a nice class and useful implementation, but to some degree, it's basically the same as you would need to do in pure Python to capture output (e.g. https://stackoverflow.com/a/1218951 or https://stackoverflow.com/a/16571630) So I'm a bit hesitant to add it to the pybind11 docs (and in particular to that page, as I'm not sure it will be noticed?).

[asmaloney]

it's basically the same as you would need to do in pure Python to capture output

Which one might be aware of or understand if you're a Python person trying to integrate some C++, but if you're a C++ person trying to embed Python it's not at all obvious (in my (recent) experience 😄 ).

I'm not sure it will be noticed?

I mentioned that page because I went looking everywhere for it and that was the page I landed on - I guess because it kind of describes the "opposite" - Capturing standard output from ostream.

Maybe just a comment about it with a pointer to the things you mention?

[YannickJadoul]

Which one might be aware of or understand if you're a Python person trying to integrate some C++, but if you're a C++ person trying to embed Python it's not at all obvious (in my (recent) experience smile ).

Well, yes, but that doesn't mean pybind11 should become the tutorial for C++ users learning Python (OK, that's a bit too harsh and unfair, I realize that, but you get my point, I hope ;-) )

Maybe just a comment about it with a pointer to the things you mention?

I guess that could work. Something short, like this, then?

..note:

    Doing the reverse and capturing Python's `stdout` and `stderr` in C++, can be done in the same way as in pure Python, by (temporarily) replacing `sys.stdout` and `sys.stderr` with an `io.StringIO` object.

[asmaloney]

pybind11 should become the tutorial for C++ users learning Python

Not even remotely what I was suggesting.

That simple note would have helped a lot and saved a bunch of time, yes.

[YannickJadoul]

Not even remotely what I was suggesting.

I know, that's not what I was claiming either. But so, we need to be careful not to put "normal Python stuff" here, for things that should actually be read in the Python docs ;-)

[silent-dxx]

My implementation is as follows

The principle is to implement a module to replace sys.stdout

#include <iostream>
#include <pybind11/embed.h>

PYBIND11_EMBEDDED_MODULE(my_sys, m) {
    struct my_stdout {
        my_stdout() = default;
        my_stdout(const my_stdout &) = default;
        my_stdout(my_stdout &&) = default;
    };

    py::class_<my_stdout> my_stdout(m, "my_stdout");
    my_stdout.def_static("write", [](py::object buffer) {
        std::cout << buffer.cast<std::string>();
    });
    my_stdout.def_static("flush", []() {
        std::cout << std::flush;
    });

    m.def("hook_stdout", []() {
        auto py_sys = py::module::import("sys");
        auto my_sys = py::module::import("my_sys");
        py_sys.attr("stdout") = my_sys.attr("my_stdout");
    });
}

auto main(int argc, char **argv) -> int {
    py::scoped_interpreter python;

#if 0
    py_stdout.attr("write") = py::cpp_function([](py::object info) {
        std::cout << "info" << std::endl;
    });
#else
    py::module::import("my_sys").attr("hook_stdout")();
#endif

    py::print("Hello, World!\n")

    return 0;
}