Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Arrow/Flatbuffers incompatibility #6372
Arrow 0.9.0/0.10.0 (and possibly earlier) uses an old/incompatible/3rd-party version of flatbuffers, and we can't simply exclude this version and use the newer/official dependency.
The issue has been reported upstream here with more details: https://issues.apache.org/jira/browse/ARROW-3175
So far the only solution (other than downgrading everything to the old/unofficial arrow version - clearly not acceptable) is to shade both Arrow and the 3rd party flatbuffers dependency for use in DataVec (it's not possible to shade just the flatbuffers dependency alone)
So it looks like the API change here is pretty minimal on FlatBufferBuilder:
A couple of (fairly hackish) ideas here that might avoid shading the whole library are to modify just the FlatBufferBuilder class (to add createString(String)) or the Arrow Field class (to call createString(CharSequence s) instead). Either via ByteBuddy or excluding the classes from the build while substituting our own.
OK, so I've spent a few hours messing around with ByteBuddy, and it was a good idea, but I don't think it'll work.
Plan B: Shade flatbuffers in the following way: