ORC-175: ORC-232: add jmh-generator-annprocess in pom.xml. improve zlib perfomance using isal api #159
Conversation
after add jmh-generator-annprocess in pom.xml, BenchmarkList was created.
iamhumanbeing
changed the title
iamhumanbeing
changed the title
| @@ -53,6 +54,8 @@ public OutputStream create(OutputStream out) throws IOException { | |||
| return new GZIPOutputStream(out); | |||
| case SNAPPY: | |||
| return new SnappyCodec().createOutputStream(out); | |||
| case ISAL: | |||
There was a problem hiding this comment.
ISAL is just the library for gzip? That doesn't need a new codec - have you tried LD_PRELOAD to load ISAL instead of libz?
There was a problem hiding this comment.
- igzip is only part of the ISAL.
- igzip only support level 0 - level 1 compression, it can not replace libz totally.
- igzip's API can not replace ligz's API directly.
ORC-175: improve ZLIB performance using isa-l's igzip api
|
LGTM |
|
Is the ISA-L zlib support sufficient to read and write the ORC files with zlib compression? I agree with Gopal that it doesn't feel like a separate compression codec. I'm don't think it is a good idea to build in support for proprietary compression formats. If it is just an optimization on performance, that might be workable. As a unique codec, it isn't. |
|
@omalley: How about we add a compiling option which use ISA-L ZLIB support for ZLIB compression&decompression? this option is just an optimization on performance. |
|
@iamhumanbeing did you compare it with zstd? Based on my experience, zstd is way better than igzip. I would expect a similar result with ISA-L. It doesn't seem to be adding a lot of value if we plan to support zstd in near future. |
after add jmh-generator-annprocess in pom.xml, BenchmarkList was created