def_readwrite throws Unicode error when reading from const char *

[RyanJarv]

def_readwrite throws Unicode error when reading from const char *

Expected

Creating a class from a C struct with const char * and std::string properties should behave the same in both cases.

Reproducible example code

Example repo

https://github.com/RyanJarv/char_struct_bug

Code

#include <pybind11/pybind11.h>

struct string_struct {
    std::string a;
};

struct char_struct {
    const char * a;
};

namespace py = pybind11;

PYBIND11_MODULE(char_struct_bug, m) {
    py::class_<string_struct>(m, "string_struct")
        .def(py::init<>())
        .def_readwrite("a", &string_struct::a);

    py::class_<char_struct>(m, "char_struct")
        .def(py::init<>())
        .def_readwrite("a", &char_struct::a);
}

Output

>>> import char_struct_bug
>>> s = char_struct_bug.string_struct()
>>> s.a = 'asdf'
>>> s.a
'asdf'
>>> c = char_struct_bug.char_struct()
>>> c.a = 'asdf'
>>> c.a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 0: invalid continuation byte

[jagerman]

The core issue here is that char * values, when coming from Python, are only valid temporarily (essentially for the duration of the function call): a char * is a pointer to memory, which we ask Python to allocate as needed. So that char * gets assigned a temporary pointer, the pointer only stays valid as long as the setter, and then the later getter is accessing invalid memory.

As a workaround, you can use a full def_property to define your own getter/setter:

struct MyType {
    const char *a = nullptr;
    ~MyType() { if (a) delete a; }
};
py::class_<MyType>(m, "MyType")
    .def(py::init<>())
    .def_property("a",
        [](const MyType &self) { return self.a; },
        [](MyType &self, const char *new_a) { self.a = strdup(new_a); }
    );

but without the destructor, that will leak. With the destructor, you might end up freeing things you shouldn't, so it probably needs another flag to indicate whether the class actually owns the pointer. And by that point, you're probably better off just switching to a std::string.

[RyanJarv]

Thanks, yeah that seems to work.

I suppose I would be curious why char * values coming from python are only temporary. Is it getting cleaned up by python or am I missing something about how c++/pybind works?

Taking a step back though, I was probably going down the wrong path with this. The reason I didn't want to switch to std::string was because I was wrapping another API that has C headers. Which if it has C headers it probably makes more sense for me to use something like ctypes(?) especially since the library I'm using isn't all that complex.

Will go ahead and close this ticket since this seems to be more an issue with my lack of understanding of how c++/pybind works. Thanks again for the help, I'm sure this will help out other people as well.

[jagerman]

I suppose I would be curious why char * values coming from python are only temporary. Is it getting cleaned up by python or am I missing something about how c++/pybind works?

Well, a char * is a pointer, so it has to have some memory to point to. For a string coming from Python, there isn't a piece of memory to point to, there's just a Python string with some internal representation (which doesn't have to be a char *, and usually isn't). So we can ask Python to convert it to utf8, but then that's just a temporary that we have to copy somewhere to keep it around.