PARQUET-1622: Add BYTE_STREAM_SPLIT encoding

[martinradev]

As discussed in the Jira issue, it would be a great benefit for better compression ratio and speed to add an encoding for FP data.

[wesm]

I think we need to vote on this

[gszadovszky]

The vote is still not initialized. I think, it would be much harder to start a vote with both the stream split and the ZFP. I would handle them separately.
While I find the stream split encoding simple and ready for implementation (after a successful vote), ZFP would require additional discussions. Parquet did not have any lossy stuff before. If the community would like to accept it it is still open if we would like to handle it like the other compression codecs.

[martinradev]

I have not started a vote yet. I am currently considering a possible change to the encoding to make it adaptive. The reason for this is that it performs worse than plain encoding when the data is very repetitive. The initial results were somewhat of an outlier because the dataset ( https://userweb.cs.txstate.edu/~burtscher/research/datasets/ ) I used is not repetitive and difficult to compress for regular text compressors like gzip, zstd, etc. However, real-world data would likely be a mixture of repetitive and non-repetitive time-series data.

I'll do a bit more investigation and initiate a vote soon.

I apologize for the caused inconvenience and overhead.

[martinradev]

As for the ZFP stuff, this was my mistake. I will have to clearly work on it on a separate branch.
I am more familiar with the "Gerrit-style" of contributing to code. The "Github-style" is new to me.
I will clean-up the pull request soon.

[martinradev]

We could maybe have a vote on this. I will open a VOTE thread.

[martinradev]

@zivanfi @gszadovszky would you be able to review the proposed encoding and achieved results? The vote is initialized on the parquet mailing list and there are two +1s.

Results from the encoding: https://github.com/martinradev/arrow-fp-compression-bench
The results are based on an implementation in Arrow but I also already have unpolished patches for parquet-mr.

[pitrou]

For posterity, here is the archive of the VOTE thread for this feature: https://lists.apache.org/thread/xs5qt2odm299pxgqb22mty2csc1so5yr