Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not assume dictionaries exists in footer #1631

Merged
merged 2 commits into from
May 3, 2022

Conversation

pcjentsch
Copy link
Contributor

This is my first Rust PR, please let me know if I could do this in a more idiomatic way!

Which issue does this PR close?

Closes #1335, on the arrow-rs side at least, Julia's Arrow.jl also writes mismatched versions (apache/arrow-julia#320).

What changes are included in this PR?

IPC reader no longer assumes footer exists.

Are there any user-facing changes?

No.

"Expecting DictionaryBatch in dictionary blocks, found {:?}.",
t
)));
match footer.dictionaries() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dictionaries is not a required field in the footer. This looks good fix.

@viirya viirya changed the title do not assume footer exists, fixes issue #1335 Do not assume dictionaries exists in footer May 2, 2022
@viirya
Copy link
Member

viirya commented May 2, 2022

There are some format error.

@codecov-commenter
Copy link

Codecov Report

Merging #1631 (86cf000) into master (7d00e3c) will increase coverage by 0.00%.
The diff coverage is 73.07%.

@@           Coverage Diff           @@
##           master    #1631   +/-   ##
=======================================
  Coverage   83.02%   83.03%           
=======================================
  Files         193      193           
  Lines       55577    55580    +3     
=======================================
+ Hits        46145    46149    +4     
+ Misses       9432     9431    -1     
Impacted Files Coverage Δ
arrow/src/ipc/reader.rs 89.27% <73.07%> (-0.10%) ⬇️
arrow/src/array/transform/mod.rs 86.68% <0.00%> (-0.12%) ⬇️
parquet/src/encodings/encoding.rs 93.56% <0.00%> (+0.18%) ⬆️
parquet_derive/src/parquet_field.rs 66.43% <0.00%> (+0.22%) ⬆️
arrow/src/datatypes/datatype.rs 66.80% <0.00%> (+0.39%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d00e3c...86cf000. Read the comment docs.

@viirya
Copy link
Member

viirya commented May 2, 2022

You can run cargo fmt locally to check the format. There are also some clippy error. You can click the CI check links to see the details too.

@pcjentsch
Copy link
Contributor Author

thank you, should be good now!

@viirya
Copy link
Member

viirya commented May 2, 2022

Thank you @pcjentsch

@viirya viirya merged commit 8f24c45 into apache:master May 3, 2022
@alamb
Copy link
Contributor

alamb commented May 4, 2022

❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Files written with Julia's Arrow.jl in IPC format cannot be read by arrow-rs
4 participants