Skip to content

Conversation

@dennybritz
Copy link
Contributor

Which issue does this PR close?

Closes #5327

What changes are included in this PR?

  • Support zstd compression
  • Refactor tests to remove boilerplate code
  • New dependencies when compression feature is enabled: zstd and async-compression/zstd

Are these changes tested?

  • I adjusted the tests to include the zstd variant and also tested manually

@github-actions github-actions bot added core Core DataFusion crate logical-expr Logical plan and expressions sql SQL Planner labels Feb 25, 2023
dennybritz and others added 3 commits February 26, 2023 00:24
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Copy link
Member

@jackwener jackwener left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great job👍.
Thank you @dennybritz

@alamb alamb changed the title Zstd compression Support Zstd compressed files Feb 26, 2023
@alamb alamb merged commit 8202a39 into apache:main Feb 27, 2023
@ursabot
Copy link

ursabot commented Feb 27, 2023

Benchmark runs are scheduled for baseline = 5ffa8d7 and contender = 8202a39. 8202a39 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@dennybritz dennybritz deleted the db/zstd-compression branch February 27, 2023 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for zstd-compressed files

6 participants