Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds parquet2 #1

Draft
wants to merge 91 commits into
base: master
Choose a base branch
from
Draft

Adds parquet2 #1

wants to merge 91 commits into from

Conversation

jorgecarleitao
Copy link
Member

@jorgecarleitao jorgecarleitao commented Jun 19, 2021 •
edited
Loading

This PR is pending IP clearance. Do not merge until IP is cleared.

It contains the source code of parquet2 rebased on top of all commits in arrow-rs, so that history of arrow-rs is preserved.

@jorgecarleitao
Initial commit.
ff1dec4
@jorgecarleitao
Initial.
6d39de3
@jorgecarleitao
Improved.
5388706
@jorgecarleitao
Reading and writing thrift.
f60e88d
@jorgecarleitao
Reading pages.
d2597a4
@jorgecarleitao
Removed unsafe.
5d8e13e
@jorgecarleitao
Added basic encoding and deserialization.
05670a7
@jorgecarleitao
Refactoring decompression.
2ee8a6d
@jorgecarleitao
Fixed apache#1.
d412a82
@jorgecarleitao
Minor simplifications.
8d39265
@jorgecarleitao
Added more tests.
99ca948
@jorgecarleitao
Added python to generate parquet files.
c9d8690
@jorgecarleitao
Added apache license.
aecbe9b
@jorgecarleitao
Fixed error in RLE encoding.
f866713
@jorgecarleitao
Added bench.
844475a
@jorgecarleitao
Improved performance of reading to arrow (-45%).
a45841a
@jorgecarleitao
Added RLE decoding to primitive arrow.
c5721d4
@jorgecarleitao
Removed unused file.
ab8baf5
@jorgecarleitao
Fixed file names.
3c2b9da
@jorgecarleitao
Added support for boolean to arrow.
8f92068
@jorgecarleitao
Fixed error.
9dd979c
@jorgecarleitao
Moved module around.
a195de8
@jorgecarleitao
Removed old examples.
1b1d3c1
@jorgecarleitao
Added support to read from String.
9234acf
@jorgecarleitao
Simplified benchmarks.
ecd1379
@jorgecarleitao
Published some modules.
bd10983
@jorgecarleitao
Added missing impl
2fda3d4
@jorgecarleitao
Renamed module to match std.
478c93a
@jorgecarleitao
Removed arrow2. Arrow2 is responsible for consuming pages.
c0ef9af 
This way other in-memory formats can use this crate.
@jorgecarleitao
Added github actions.
782443f
jorgecarleitao and others added 23 commits June 19, 2021 05:44
@jorgecarleitao
Added delta-bitpacked decoding.
d3231ca
@jorgecarleitao
Added delta decoding.
7fff5b8 
- delta-byte-array
- delta-length-byte-array
@jorgecarleitao
Improved README.
0e42425
@jorgecarleitao
Unpinned compression versions
3d90311
@jorgecarleitao
Added #![forbid(unsafe_code)]
6ce8cef
@jorgecarleitao
Fixed wrong number of reported bytes in Column chunk.
530bd44
@jorgecarleitao
Added statistics for boolean.
e5115c8
@elferherrera @jorgecarleitao
cleaning folder
08bb3c4
@jorgecarleitao
Allowed writing metadata and created_by
5a0b06e
@jhorstmann @jorgecarleitao
Support for plain encoded binary in data pages
7e7a38b
@jhorstmann @jorgecarleitao
Fix reading of length and implement size_hint for plain binary decoder
12d95d1
@jhorstmann @jorgecarleitao
Add tests for string columns with and without dictionary encoding
23242c1
@jhorstmann @jorgecarleitao
Return upper bound in plain binary decoder
2e1fecd
@jorgecarleitao
Added method to read columns' statistics.
83c6005
@jorgecarleitao
Improved API to read statistics.
b7498b0
@jorgecarleitao
Made int96 into nanoseconds.
ea4d85d
@jorgecarleitao
Improved API to re-use page buffers (#14)
8452203
@jorgecarleitao
Added support for native binary.
039f5ae
@jorgecarleitao
Added integration tests for non-dict arrays.
4be4ac4
@jorgecarleitao
Allow statistics to be written in pages and columns (#16)
d4128a5
@jorgecarleitao
Added descriptor to statistics. (#17)
eaeb646
@jorgecarleitao
Allow writing a file from a stream. (#18)
d40af6b
@jorgecarleitao
Clippy warnings.
d934300
@jorgecarleitao jorgecarleitao changed the title Adds Adds parquet2 Jun 19, 2021
@codecov-commenter
Copy link

Codecov Report

❗ No coverage uploaded for pull request base (master@5518118). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 4dffc8d differs from pull request most recent head 9e7ca73. Consider uploading reports for the commit 9e7ca73 to get more accurate results
Impacted file tree graph

@@            Coverage Diff            @@
##             master       #1   +/-   ##
=========================================
  Coverage          ?   75.83%           
=========================================
  Files             ?       62           
  Lines             ?     3174           
  Branches          ?        0           
=========================================
  Hits              ?     2407           
  Misses            ?      767           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5518118...9e7ca73. Read the comment docs.

jimexist
jimexist reviewed Jun 29, 2021
View reviewed changes
.github/workflows/test.yml
apt update && apt install python3-pip python3-venv -y -q
python3 -m venv venv
venv/bin/pip install pip --upgrade
venv/bin/pip install pyarrow==3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use 4.0.1? https://pypi.org/project/pyarrow/

@jorgecarleitao
Copy link
Member Author

@elferherrera @jhorstmann , for Apache's IP clearance:

  • would it be ok for your contributions to parquet2 to be donated to Apache under the CLA?
  • if yes, could you sign and submit a CLA, so that we have the paperwork in order for this to proceed?

Sorry for the inconvenience, this does not seem to be needed in normal contributions, but for some reason for these clearances it is required.

@jhorstmann
Copy link
Contributor

I'm of course ok with the donation, will check the CLA requirements later

@jorgecarleitao
Copy link
Member Author

@jhorstmann , did you submitted a CLA?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jimexist jimexist jimexist left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@jorgecarleitao @codecov-commenter @jhorstmann @jimexist @elferherrera