Introduce more compression libraries and implementations

[mulugetam]

Lucene, and by extension OpenSearch, currently support only zlib and lz4 compression algorithms offering best compression ratio and best speed, respectively. There are a couple of limitations with this:

• Use cases that prefer a middle ground between best compression ratio and best speed, such as that offered by zstd, are not supported.
• The lz4 implementation is written entirely in Java and, based on our benchmark results, is less performant than a native lz4 implementation.

To address the limitations, we have implemented a small library that adds support for both zstd and lz4 (native) compressions to OpenSearch. The library requires a couple of source code changes to OpenSearch, EngineConfig.java and CodecService.java, but does not require any code changes to Lucene.

The charts below summarize the relative performance gain we measured for the different compression algorithms and implementations using the StoredFieldsBenchmark.

Indexing Time (relative, less is better):

Retrieval Time (relative, less is better):

Stored size (relative, less is better):

We have also used the OpenSearch-Benchmark with the nyc_taxis workload to compare the indexing performance of the different compression algorithms and implementations above and measured decent gains as well.

We'll be upstreaming our source code and patches in a couple of weeks and discuss on how best we can integrate this into OpenSearch and make them available for users. This post is intended to start the discussion.

[reta]

@mulugetam there is an issue on Apache Lucene to support ZSTD [1], [2], plus somewhat unrelated to index storage but ZSTD [3] for snapshots. I believe the main concern is related to usage of the JNI bindings, I suspect your implementation is also using that? Thank you.

[1] https://issues.apache.org/jira/browse/LUCENE-8739
[2] apache/lucene#439
[3] #2192

[dblock]

This is awesome @mulugetam, looking forward!

[dblock]

@reta on JNI, I think the issue is primarily native code, not the bindings themselves, as Lucene is strongly opposed to including native code? This isn't a limitation of OpenSearch, we have plugins, such as k-nn, that use native libraries.

I don't love JNI btw as it requires one to write JNI bindings in C. I would prefer to replace all native boilerplate code written in OpenSearch with https://github.com/java-native-access/jna (see opensearch-project/k-NN#35 for an example).

[reta]

thanks @dblock, yeah I think the native code + JNI bindings (as the consequence) is the concern for Apache Lucene (as per discussion thread), I am just curious what the compression implementation was used, as you said, OpenSearch has no such limitations or / and constraints.

[willyborankin]

Compression for snapshotting I added here: #2996. @mulugetam Do you implement Lucene codes with ZSTD compression support? Regarding lz4 Lucense has its own implementation the missing part is a support of lz4 InputStream.

[mulugetam]

@willyborankin Yes. From Lucene, you'd set your codec, e.g. zstd, using:

indexWriterConfig.setCodec(new Lucene92CustomCodec(Lucene91CustomCodec.Mode.ZSTD));

and lz4-jni (native) like:

indexWriterConfig.setCodec(new Lucene92CustomCodec(Lucene92CustomCodec.Mode.LZ4_NATIVE));

From OpenSearch, only the EngineConfig.java and CodecService.java source files need to be updated. You can then use it by setting the index.codec property.

Below is how I used it when running OpenSearch-Benchmark:

echo '{"index_settings":{"index.codec": "zstd"}}' > params.json
opensearch-benchmark execute_test \
--workload=nyc_taxis \
--workload-params=params.json \
--target-hosts=127.0.0.1:9200 \
--pipeline=benchmark-only

The current lz4 implementation in Lucene, which is written in pure Java, has a very poor performance and that is why we went for the native implementation.

Compression for snapshotting and lz4 InputStream is something we'd need to look into. I'll be upstreaming the patches as soon as I'm done with our internal process here.

[willyborankin]

@mulugetam This is perfect. I misread your comment about LZ4 looks really good. Regarding lz4 and InputStream, I can implement it like it was done for Kafka here: https://github.com/apache/kafka/tree/99b9b3e84f4e98c3f07714e1de6a139a004cbc5b/clients/src/main/java/org/apache/kafka/common/compress.

[willyborankin]

Maybe it makes sense to add a global parameter in opensearch.yml such this:
cluster.compression_type: which defines compression type both for indexes and snapshotting. WDYT?

[reta]

@willyborankin what would be the benefit to compress LZ4 indices with LZ4 again while snapshotting?

[willyborankin]

@reta Better compression but slow read even compare to ZSTD ... I think it is up to the end-user to decide. Wdyt?

[reta]

Ok, let me go back to your original question:

Maybe it makes sense to add a global parameter in opensearch.yml such this:
cluster.compression_type: which defines compression type both for indexes and snapshotting.

Let say the indices are compressed with LZ4 / ZSTD, what would be the benefit to compress them again with LZ4 / ZSTD while snapshotting? I am trying to understand this part.

[willyborankin]

The idea is to set this parameter in case you want to use the same compression for indexes and snapshotting otherwise use lets say a granular settings for indexes and snapshotting separately.

[mulugetam]

@willyborankin @reta @dblock Have created the PR here: #3577.

[dblock]

This is tangentially related to encryption, #3469, just putting it on the radar here so we can think of consistent behavior in indexes vs. snapshots.

[willyborankin]

tangentially

@dblock answered

[wbeckler]

Will this work on AWS graviton or AMD processors, or is it just going to work when a server is using an Intel chip? And is there a downside to the non-intel users?

[dblock]

I opened opensearch-project/documentation-website#3816 to document this and other codecs. @mulugetam Interested in drafting something?

[wbeckler]

Can someone explain what the impacts are on backward compatibility when a new compression codec is introduced? Does a cluster need to do a full restart and/or reindex? Is it possible to upgrade with zero downtime?

[dblock]

What would be the steps to take this feature out of experimental @mulugetam?

[willyborankin]

What would be the steps to take this feature out of experimental @mulugetam?

Hi @dblock, as far as I understand all you need to do is just move sandbox/plugins/custom-codecs into modules (maybe additional tests need to be added). @reta correct me if I'm wrong. I can do it after #2996 has been merged

[reta]

@mulugetam any tests have been performed, internally or externally, you could share results of? should we run opensearch-benchmarks workloads with this new codec?

[sarthakaggarwal97]

@reta you can view the results of benchmarks over at #7805