Python bindings

[shaxbee]

I need to be able to create plugins for blender and some conversion tools, thus Python binding is required.

I am currently working on it but just want to make sure someone else is not doing it already and I'm going in the right direction.

Ideas:

• structs are represented as namedtuple, whole struct is read at once
• offsets and vectors of scalar types are wrapped array module
• all subregions are mapped using memoryview, deserialization of scalars is done using struct module
• there is optional extension module which registers itself as import hook and allows to directly import schema without compiling it

I don't know anyone that's working on it.

I'm not familiar enough with Python to be able to comment, but certainly anything that just represents an array of bytes is a good starting point. Convenient reading/writing of little endian scalars is helpful.

[shaxbee]

cb91e03:

Basic primitives reading - scalar / vector / offsets implemented.
Enum and fixed structure generator implemented.

Pending: tables and union generator

Cool to see progress!

[shaxbee]

7a4c374:

Python code generator for tables
Skeleton of builder implementation
Unit tests for reading scalars

[rodrigob]

On which branch should we follow changes on this issue ?
https://github.com/shaxbee/flatbuffers/tree/python_support_v2 ?

Out the box with Ubuntu 14.04 and gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2 I get simple errors like
flatbuffers_python/src/idl_gen_python.cpp:500:32: error: ‘ostream_iterator’ is not a member of ‘std’

[rodrigob]

After fixing the ostream_iterator error (by including <iterator>), the generated python code is not accepted by the parser:

Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_test_fbs.py
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "my_test_fbs.py", line 53
    _format = Struct("
                     ^
SyntaxError: EOL while scanning string literal
>>> 

and

Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_test_fbs.py
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: source code string cannot contain null bytes
>>> 

[rodrigob]

The problem of line 53 is _format = Struct("\00\00f4xx") the \00 are the issue.

The corresponding structure definition is

struct MyTestStruct {
  idx1:ulong;
  idx2:ulong;
  thresh:float;
}

it seems like the python port has problems handling ulong types. Any idea of the proper fix ?

[rodrigob]

Traced back the issue to static char GenPyFmtChar(const Type& type) which does not cover the ulong case indeed.
Which format char should be returned for ulong ?
I guess this function should implement
https://docs.python.org/2/library/struct.html#format-characters ?

I also notice there
default: assert("Unexpected scalar type"); return 0;

notice that the assert will never raise. Is this a bug or a feature ?

[shaxbee]

Hi Rodrigo, it's still work in progress :-) assert is there in case some
new scalar type gets added / to suppress unhandled enum value warning.

On Tue, Nov 4, 2014, 18:02 Rodrigo Benenson notifications@github.com
wrote:

Traced back the issue to static char GenPyFmtChar(const Type& type) which
does not cover the ulong case indeed.
Which format char should be returned for ulong ?
I guess this function should implement
https://docs.python.org/2/library/struct.html#format-characters ?

I also notice there
default: assert("Unexpected scalar type"); return 0;

notice that the assert will never raise. Is this a bug or a feature ?

—
Reply to this email directly or view it on GitHub
#81 (comment).

[shaxbee]

Oh I see now. I guess I should signal error somehow. I'll look into that on
weekend.

On Tue, Nov 4, 2014, 18:11 Zbigniew Mandziejewicz shaxbee@gmail.com wrote:

Hi Rodrigo, it's still work in progress :-) assert is there in case some
new scalar type gets added / to suppress unhandled enum value warning.

On Tue, Nov 4, 2014, 18:02 Rodrigo Benenson notifications@github.com
wrote:

Traced back the issue to static char GenPyFmtChar(const Type& type)
which does not cover the ulong case indeed.
Which format char should be returned for ulong ?
I guess this function should implement
https://docs.python.org/2/library/struct.html#format-characters ?

I also notice there
default: assert("Unexpected scalar type"); return 0;

notice that the assert will never raise. Is this a bug or a feature ?

—
Reply to this email directly or view it on GitHub
#81 (comment).

[rodrigob]

@shaxbee thanks for the efforts on this feature.
Got things almost running (after adding the missing long/ulong entries in GenPyFmtChar):
a) The generated python code was missing import functools

b) Now I have no clue how to read my data, I tried

    input_data = open(path_to_fbs_dot_bin, "rb")
    my_struct = my_generated_fbs_library.MyRootDataStructure(input_data)
    print(len(my_struct.some_vector_entry))

and I get

File "flatbuffers_python/python/flatbuffers/reader.py", line 102, in read_field
    view, base, vtable = self
ValueError: too many values to unpack

any suggestion on how to read the data from the python side ?

[shaxbee]

@rodrigob I'm in the process of writing tests and ironing out bugs for reading - you'll see some working stuff on weekend.

[shaxbee]

@rodrigob I've pushed some bugfixes - any help w. r. t reading flatbuffers would be welcome - feel free to fork and/or report errors here :-)

[rodrigob]

With current head I get

my_generated_fbs.py", line 92, in forests
    return self._read_indirect_field(1, functools.partial(flatbuffers.indirectvector, Forest.unpack_from))
  File "python/flatbuffers/reader.py", line 111, in _read_indirect_field
    view, base, vtable = self
ValueError: too many values to unpack

not clear what is wrong there (similar bug than before).

[rodrigob]

But keep in mind I do not know how to instanciate the read. Maybe I am doing it all wrong.
For now I do: instance = MyClass(input_data.read()); print(len(instance.field_I_want_to_access))

Is MyClass(input_data.read()) the right thing to do ? Looking at the code it seems not.