Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Zstandard Compression #438

Closed
milleruntime opened this issue Apr 24, 2018 · 2 comments

Comments

@milleruntime
Copy link
Contributor

commented Apr 24, 2018

Hadoop added support for the Zstd compression library in version 2.9.0 https://issues.apache.org/jira/browse/HADOOP-13578

Zstandard was open source'd by Facebook a few years ago and looks like a nice improvement over gzip (better compression ratio and speed) and snappy (better compression ratio).
https://facebook.github.io/zstd/

@milleruntime

This comment has been minimized.

Copy link
Contributor Author

commented Apr 26, 2018

It looks like the 2.9.0 and 3.0.2 binary distributions of Hadoop were not built with Zstandard library. So this will require building Hadoop with -Pdist,native and -Drequire.zstd flags.

@milleruntime milleruntime self-assigned this Apr 30, 2018

@milleruntime

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2018

Tips for building Hadoop (I was able to build 2.9.0 on Fedora 27):

  • Required manual compilation and install of Protocol buffers 2.5.0. First did a "dnf remove protobuf-compiler" to make things easier.
  • Will fail to build with OpenSSL 1.1.0 so either patch or use 1.0.2
  • Fedora packages required: cmake openssl-devel make automake gcc gcc-c++ libzstd
    libzstd-devel

@ctubbsii ctubbsii added v2.0.0 and removed v1.9.1 labels May 8, 2018

@ctubbsii ctubbsii added this to Done in 2.0.0 Jun 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
2 participants
You can’t perform that action at this time.