-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[JavaScript] Flexbuffers support #5949
Comments
I was thinking about doing it anyways. So I can take it. |
@aardappel I am done with FlexBuffers decoder. Should I create a PR already, so it is not that much code to review, or should I write an encoder as well in single PR? |
@mzaks I'd say lets do a code review once you think it is ready to merge. |
I just created a PR: #5973 |
@mzaks @aardappel Thanks for the PR and review! I tried out the code for and for me it seems like // json data: {'channels_in': 64, 'dilation_height_factor': 1, 'dilation_width_factor': 1, 'fused_activation_function': 1, 'pad_values': 1, 'padding': 0, 'stride_height': 1, 'stride_width': 1}
var data = [99, 104, 97, 110, 110, 101, 108, 115, 95, 105, 110, 0, 100, 105, 108, 97, 116, 105, 111, 110, 95, 104, 101, 105, 103, 104, 116, 95, 102, 97, 99, 116, 111, 114, 0, 100, 105, 108, 97, 116, 105, 111, 110, 95, 119, 105, 100, 116, 104, 95, 102, 97, 99, 116, 111, 114, 0, 102, 117, 115, 101, 100, 95, 97, 99, 116, 105, 118, 97, 116, 105, 111, 110, 95, 102, 117, 110, 99, 116, 105, 111, 110, 0, 112, 97, 100, 95, 118, 97, 108, 117, 101, 115, 0, 112, 97, 100, 100, 105, 110, 103, 0, 115, 116, 114, 105, 100, 101, 95, 104, 101, 105, 103, 104, 116, 0, 115, 116, 114, 105, 100, 101, 95, 119, 105, 100, 116, 104, 0, 8, 130, 119, 97, 76, 51, 41, 34, 21, 8, 1, 8, 64, 1, 1, 1, 1, 0, 1, 1, 4, 4, 4, 4, 4, 4, 4, 4, 16, 36, 1];
flexbuffers.toObject(new Uint8Array(data).buffer); Which fails with: /Users/lgeiger/code/flatbuffers/js/flexbuffers.js:257
while (dataView.getUint8(keyIndirectOffset + length) !== 0) {
^
RangeError: Offset is outside the bounds of the DataView
at DataView.getUint8 (<anonymous>)
at keyForIndex (/Users/lgeiger/code/flatbuffers/js/flexbuffers.js:257:21)
at Object.toObject (/Users/lgeiger/code/flatbuffers/js/flexbuffers.js:403:23)
at Object.flexbuffers.toObject (/Users/lgeiger/code/flatbuffers/js/flexbuffers.js:435:42) Do you have an idea what's going on there? Let me know if I you would prefer me to open a new issue. |
Hi Lukas, what did you use to build the buffer? |
@lgeiger Ah sorry you just replied it was C++. Then let me check a bit further. |
So the buffer you got with C++ is 10 bytes shorter then the one I get if encode the same object with JS. I need a bit more time to get to the bottom of it and today is already a bit late. I will get back at you tomorrow. @lgeiger |
@mzaks Thanks for taking a look, I'll try to investigate further |
I double checked, and if I generate the flexbuffer with Python I get 160 bytes: fbb = flexbuffers.Builder()
with fbb.Map():
fbb.Int("channels_in", 64)
fbb.Int("dilation_height_factor", 1)
fbb.Int("dilation_width_factor", 1)
fbb.Int("fused_activation_function", 1)
fbb.Int("pad_values", 1)
fbb.Int("padding", 0)
fbb.Int("stride_height", 1)
fbb.Int("stride_width", 1)
data = fbb.Finish()
list(data)
# [99, 104, 97, 110, 110, 101, 108, 115, 95, 105, 110, 0, 100, 105, 108, 97, 116, 105, 111, 110, 95, 104, 101, 105, 103, 104, 116, 95, 102, 97, 99, 116, 111, 114, 0, 100, 105, 108, 97, 116, 105, 111, 110, 95, 119, 105, 100, 116, 104, 95, 102, 97, 99, 116, 111, 114, 0, 102, 117, 115, 101, 100, 95, 97, 99, 116, 105, 118, 97, 116, 105, 111, 110, 95, 102, 117, 110, 99, 116, 105, 111, 110, 0, 112, 97, 100, 95, 118, 97, 108, 117, 101, 115, 0, 112, 97, 100, 100, 105, 110, 103, 0, 115, 116, 114, 105, 100, 101, 95, 104, 101, 105, 103, 104, 116, 0, 115, 116, 114, 105, 100, 101, 95, 119, 105, 100, 116, 104, 0, 8, 130, 119, 97, 76, 51, 41, 34, 21, 8, 1, 8, 64, 1, 1, 1, 1, 0, 1, 1, 4, 4, 4, 4, 4, 4, 4, 4, 16, 36, 1] and 170 bytes if I convert it with JavaScript using: flexbuffers.encode({
channels_in: 64,
dilation_height_factor: 1,
dilation_width_factor: 1,
fused_activation_function: 1,
pad_values: 1,
padding: 0,
stride_height: 1,
stride_width: 1,
})
// [99, 104, 97, 110, 110, 101, 108, 115, 95, 105, 110, 0, 100, 105, 108, 97, 116, 105, 111, 110, 95, 104, 101, 105, 103, 104, 116, 95, 102, 97, 99, 116, 111, 114, 0, 100, 105, 108, 97, 116, 105, 111, 110, 95, 119, 105, 100, 116, 104, 95, 102, 97, 99, 116, 111, 114, 0, 102, 117, 115, 101, 100, 95, 97, 99, 116, 105, 118, 97, 116, 105, 111, 110, 95, 102, 117, 110, 99, 116, 105, 111, 110, 0, 112, 97, 100, 95, 118, 97, 108, 117, 101, 115, 0, 112, 97, 100, 100, 105, 110, 103, 0, 115, 116, 114, 105, 100, 101, 95, 104, 101, 105, 103, 104, 116, 0, 115, 116, 114, 105, 100, 101, 95, 119, 105, 100, 116, 104, 0, 0, 8, 0, 132, 0, 122, 0, 101, 0, 81, 0, 57, 0, 48, 0, 42, 0, 30, 0, 16, 2, 8, 64, 1, 1, 1, 1, 0, 1, 1, 4, 4, 4, 4, 4, 4, 4, 4, 16, 36, 1] |
@mzaks function indirect(dataView, offset, width) {
const step = readUInt(dataView, offset, width);
return offset - step;
} |
So @lutzroeder is correct and the bug surfaced a few more issues with this implementation. I fixed it all and submitted a PR: #6107 As mentioned in PR comment the buffers created with previous implementation are compatible with C++ and other correct implementations of FlexBuffers. They are just a bit bigger in size, because values which can fit in 1 byte as uint (127 < x < 255 ) were stored as 2 bytes. This is where the 10 bytes difference comes from, which I found in my first investigation. |
This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days. |
Are there any plans or existing projects working on support for Flexbuffers in JavaScript?
One of my projects would really benefit from a schema-less option, and the C++ implementation for Flexbuffers is very pleasant to use.
The text was updated successfully, but these errors were encountered: