Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate Arrow Parquet writer compression code into dwio::common::compression #6901

Open
yingsu00 opened this issue Oct 5, 2023 · 2 comments
Assignees
Labels
arrow enhancement New feature or request parquet

Comments

@yingsu00
Copy link
Collaborator

yingsu00 commented Oct 5, 2023

Description

In #6586, the Arrow Parquet writer was moved into Velox. Currently people are moving the compression codecs #6805
and #6835. But these codecs were already in dwio::common::compression and many of the copied in code are duplicate. After #6890, we want to consolidate these codecs in the Parquet writer to dwio::common::compression.

@karteekmurthys
Copy link
Contributor

I will take up the task of switching to Velox zstd codecs for arrow parquet writer.
cc: @yingsu00 @majetideepak @pedroerp

@yingsu00
Copy link
Collaborator Author

I will take up the task of switching to Velox zstd codecs for arrow parquet writer. cc: @yingsu00 @majetideepak @pedroerp

cc @nmahadevuni

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow enhancement New feature or request parquet
Projects
None yet
Development

No branches or pull requests

2 participants