Made IPC ZSTD-compressed consumable by pyarrow

[jorgecarleitao]

I do not understand why the level must be zstd's default to work, but it works. This is either a bug on zstd's crate or on pyarrow's ZSTD's compressor :/

[codecov]

Codecov Report

Merging #675 (e50f12b) into main (2b86bc9) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #675   +/-   ##
=======================================
  Coverage   69.60%   69.60%           
=======================================
  Files         301      301           
  Lines       16765    16765           
=======================================
  Hits        11670    11670           
  Misses       5095     5095           

Impacted Files	Coverage Δ
src/io/ipc/compression.rs	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b86bc9...e50f12b. Read the comment docs.

[ritchie46]

That magic number goes somewhere in metadata I assume? Now we aligned with pyarrows 'level'?

[jorgecarleitao]

Yes, the thing is that the "level" is just a number that tradeoffs performance for compression, so the fact that only one level works is a bit worrisome imo - the ability to read and write ZSTD-compressed data should be independent of the level. This PR just matches the level, but why other numbers do not work is unclear to me.

[ritchie46]

Hmm.. a bit hard to test whats at fault here. As there are not many ipc implementations. xD

[jorgecarleitao]

exactly! xD The evidence (prior to this PR) is:

• pyarrow reads what pyarrow writes
• pyarrow does not read what arrow2 writes
• arrow2 reads what pyarrow writes
• arrow2 reads what arrow2 writes

since this only changes the arrow2 writer (and we continue to be able to read it), I am not very worried.