Skip to content

Support RLE, DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY Parquet encodings#5457

Merged
Mytherin merged 10 commits into
duckdb:featurefrom
Mytherin:deltabytearray
Nov 23, 2022
Merged

Support RLE, DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY Parquet encodings#5457
Mytherin merged 10 commits into
duckdb:featurefrom
Mytherin:deltabytearray

Conversation

@Mytherin

Copy link
Copy Markdown
Collaborator

Fixes #3981

This should implement all remaining Parquet encoding types - aside from Byte Stream Split which does not appear to be used by anyone anywhere.

@Mytherin
Mytherin merged commit 49882de into duckdb:feature Nov 23, 2022
@dselivanov

Copy link
Copy Markdown

@Mytherin what is the way to try it in python? Should I follow build steps from "python" github actions or there is a simpler way?

@Mause

Mause commented Nov 25, 2022

Copy link
Copy Markdown
Contributor

@Mytherin what is the way to try it in python? Should I follow build steps from "python" github actions or there is a simpler way?

You can install the Python prerelease using the pip --pre flag

@dselivanov

dselivanov commented Nov 25, 2022

Copy link
Copy Markdown

Thanks @Mause. I believe --pre installs from master branch (and current feature is still in feature branch)? I've tried pip install duckdb --pre --upgrade and see Unsupported page encoding while reading parquet (duckdb=0.6.1.dev86)

@dselivanov

Copy link
Copy Markdown

I was able to build package on linux machine just by running python3 -m build from within ./tools/pythonpkg. Reading parquet files with DELTA_LENGTH_BYTE_ARRAY encoded columns works smooth now.
On my local OSX machine, however, package built from feature branch throws segmentation fault when running import duckdb. Mb something with my environment - not sure.

@Mytherin

Copy link
Copy Markdown
Collaborator Author

That sounds like something funky with your environment indeed, perhaps try uninstalling all versions of duckdb (repeat pip uninstall duckdb until no more versions remain) and doing a clean build?

@Mytherin
Mytherin deleted the deltabytearray branch January 7, 2023 15:07
@Tishj Tishj mentioned this pull request Apr 6, 2023
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants