Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files written with Julia's Arrow.jl in IPC format cannot be read by arrow-rs #1335

Closed
pcjentsch opened this issue Feb 18, 2022 · 1 comment · Fixed by #1631
Closed

Files written with Julia's Arrow.jl in IPC format cannot be read by arrow-rs #1335

pcjentsch opened this issue Feb 18, 2022 · 1 comment · Fixed by #1631
Labels
arrow Changes to the arrow crate bug

Comments

@pcjentsch
Copy link
Contributor

Describe the bug
Files written with Julia's Arrow.jl IPC format have a footer that causes arrow-rs to panic. I think this is a bug on the side of arrow-rs because arrow2 can read them.

To Reproduce
Reproduced in this small repository.

git clone https://github.com/pcjentsch/testing_ipc_issue.git
cd testing_ipc_issue/test_ipc
cargo run

output

Ok((Schema { fields: [Field { name: "a", data_type: Int64, is_nullable: false, metadata: {} }], metadata: {} }, [Chunk { arrays: [Int64[1, 2, 3]] }]))
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /home/workaccount/.cargo/registry/src/github.com-1ecc6299db9ec823/arrow-9.0.2/src/ipc/reader.rs:636:44
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Expected behavior
arrow-rs should open the file as arrow2 does.

@pcjentsch pcjentsch added the bug label Feb 18, 2022
@alamb
Copy link
Contributor

alamb commented Feb 28, 2022

Looks like there is a function that assumes there is a footer when there may not be -- looks pretty straight forward to me. I'll try and get a fix up in the next few days if no one else beats me to to it.

for block in footer.dictionaries().unwrap() {

Thank you for the great report @pcjentsch

pcjentsch added a commit to pcjentsch/arrow-rs that referenced this issue Apr 29, 2022
pcjentsch added a commit to pcjentsch/arrow-rs that referenced this issue Apr 29, 2022
viirya pushed a commit that referenced this issue May 3, 2022
* do not assume footer exists, fixes issue #1335

* fix cargo fmt and clippy errors
@alamb alamb added the arrow Changes to the arrow crate label May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants