Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Javascript] tableToIPC has off-by-one errors in the generated IPC byte length #14791

Closed
aarashy opened this issue Nov 30, 2022 · 2 comments · Fixed by #14881
Closed

[Javascript] tableToIPC has off-by-one errors in the generated IPC byte length #14791

aarashy opened this issue Nov 30, 2022 · 2 comments · Fixed by #14881

Comments

@aarashy
Copy link

aarashy commented Nov 30, 2022

Describe the bug, including details regarding any error messages, version, and platform.

Hello, I filed an issue with arrow-rs because panics were being triggered in their IPC byte parsing functions.
apache/arrow-rs#3215 (comment)

The input IPC bytes were the result of tableToIPC. According to the info I've gathered in my other issue, it seems like tableToIPC is not correctly padding and instead incorrectly truncating bits in cases where the length of a record batch is not divisible by 8.

For example, the Null Bit Buffer encodes whether each row is null or not using 1 bit. My record batch has length 518, which is not divisible by 8 (64.75). Some parsing/validation inside arrow-rs is therefore expecting the null bit buffer to be 65 bytes long, but it's only 64 bytes long.

Component(s)

Format, Integration, JavaScript

@aarashy
Copy link
Author

aarashy commented Dec 6, 2022

@trxcllnt - Hello! You seem to be the biggest contributor to this file https://github.com/apache/arrow/blob/5611f2bd0d6136b005d137a84b50709fc5c813bb/js/src/ipc/writer.ts

I wonder if you have any insight on what might have been happening here.

@trxcllnt
Copy link
Contributor

trxcllnt commented Dec 7, 2022

@aarashy thanks for the report, I'll take a look and try to repro.

@domoritz domoritz added javascript Pull requests that update Javascript code Component: JavaScript and removed javascript Pull requests that update Javascript code labels Dec 7, 2022
trxcllnt added a commit to trxcllnt/arrow that referenced this issue Dec 7, 2022
trxcllnt added a commit to trxcllnt/arrow that referenced this issue Dec 7, 2022
domoritz pushed a commit that referenced this issue Dec 9, 2022
* 730e9c5 updates `ts-jest` configuration to remove deprecation warnings
* e4d83f2 updates `print-buffer-alignment.js`  debug utility for latest APIs
* 3b9d18c updates `arrow2csv` to print zero-based rowIds
* b6c42f3 fixes #14791


* Closes: #14791

Authored-by: ptaylor <paul.e.taylor@me.com>
Signed-off-by: Dominik Moritz <domoritz@gmail.com>
@domoritz domoritz added this to the 11.0.0 milestone Dec 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants