Skip to content

Enable configuration of the zstd compression level for feather store #2623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Nov 1, 2022

Conversation

dispanser
Copy link
Contributor

This came up during work on the parquet / feather benchmarks, where we want to measure the effect of various compression levels for both parquet and feather.

This change follows closely what we do for the parquet plugin to make the zstd compression level configurable.

📝 Reviewer Checklist

Review this pull request by ensuring the following items:

  • All user-facing changes have changelog entries
  • User-facing changes are reflected on vast.io

@dispanser
Copy link
Contributor Author

I guess I still need to add a changelog entry, extend vast.yaml.example, and find a place in our documentation to add the parameter.

NB, there is no documentation for parquet plugin configuration options. Where's a good place for that?

@dominiklohmann
Copy link
Member

NB, there is no documentation for parquet plugin configuration options. Where's a good place for that?

I'd say under tuning: https://vast.io/docs/setup/tune

@dispanser dispanser force-pushed the topic/feather-zstd-configurable branch 3 times, most recently from 18f2bd5 to 1c99ee3 Compare October 27, 2022 16:17
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is almost there. Happy to see this lands, especially in light of the data analysis from the recent blog post.

@dispanser dispanser force-pushed the topic/feather-zstd-configurable branch from d92f564 to 563d742 Compare October 27, 2022 18:09
@dispanser dispanser force-pushed the topic/feather-zstd-configurable branch from 0297a3f to e513569 Compare October 28, 2022 06:40
@dispanser dispanser marked this pull request as ready for review October 28, 2022 08:20
@dispanser dispanser enabled auto-merge November 1, 2022 03:26
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wordsmithing now. I think this is ready to go! 🚀

dispanser and others added 2 commits November 1, 2022 11:21
@dispanser dispanser merged commit ab96db7 into master Nov 1, 2022
@dispanser dispanser deleted the topic/feather-zstd-configurable branch November 1, 2022 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants