Add experimental support for zstd compression. #3577

mulugetam · 2022-06-14T09:06:53Z

This PR adds experimental support for lz4 (native) and zstd (with and without dictionary) compression algorithms. The rationale for adding this support is discussed here: #3354.

Users would be able to set the compression type by setting the index.codec with the following values:

"LZ4" -- lz4 (native) compression.
"ZSTD" -- zstd compression, with dictionary.
"ZSTDNODICT" -- zstd compression, without dictionary.

The core classes are packaged under org.apache.lucene.codecs.experimental.

The compression algorithms have been tested and benchmarked using OpenSearch-Benchmark and StoredFieldsBenchmark . The results show significant gains in performance.

Signed-off-by: Mulugeta Mammo mulugeta.mammo@intel.com

Description

A detailed description is available here: #3354

Issues Resolved

#3354

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot · 2022-06-14T09:18:37Z

❌ Gradle Check failure 9d3cbe0dbd8e41daef93385b6c77aa2ca6c95d6b
Log 5973

Reports 5973

willyborankin · 2022-06-14T09:20:45Z

Small thing so far:

You need to add signature files for new libs + licenses otherwise build fails :-)
Take a look here: ZSTD snapshotting compression #2996 it will reduce my changes.
./gradlew precommit/check will help

reta · 2022-06-14T12:48:14Z

@dblock @nknize I think we should take the sandbox approach for experimental features. We have the ability to provide CodecService from the plugin(s), using sandbox plugin for that would make sense. Thoughts?

mulugetam · 2022-06-14T17:05:36Z

@willyborankin I have added signature files and licenses for the dependencies.

opensearch-ci-bot · 2022-06-14T17:07:21Z

❌ Gradle Check failure 759642278598f5f17fe12af5a6efa052e06e9972
Log 5987

Reports 5987

mulugetam · 2022-06-14T17:17:12Z

@reta the PR is labelled "experimental" just because we'd like to make it available for users together with the snapshotting compression that @willyborankin has been doing. The codecs work well and have significant performance gains.

dblock · 2022-06-14T18:30:52Z

@dblock @nknize I think we should take the sandbox approach for experimental features. We have the ability to provide CodecService from the plugin(s), using sandbox plugin for that would make sense. Thoughts?

I really don't know anything about this stuff, deferring to @nknize, at a high level, yes.

Overall this PR looks good to me.

dblock · 2022-06-14T18:31:28Z

@mulugetam You should get it to green either way, fix DCO, etc.

opensearch-ci-bot · 2022-06-14T20:30:34Z

❌ Gradle Check failure 3de2f6b7a00dfe4f700f2d45e314f7a1ca53c7fa
Log 5997

Reports 5997

opensearch-ci-bot · 2022-06-14T20:30:47Z

❌ Gradle Check failure 6999a5496e1955e4c632044cee86cddcdb50654b
Log 5998

Reports 5998

opensearch-ci-bot · 2022-06-15T07:50:08Z

✅ Gradle Check success e95ae663dfd0cda0550d5bb7ffd589590b87312a
Log 6019

Reports 6019

dblock

I think license headers need updates, otherwise LGTM.

server/src/main/java/org/apache/lucene/codecs/experimental/LZ4CompressionMode.java

...er/src/main/java/org/apache/lucene/codecs/experimental/Lucene90CustomStoredFieldsFormat.java

opensearch-ci-bot · 2022-06-15T16:26:19Z

❌ Gradle Check failure 1a558e751816fb87ddf54b3e529c65ab1a8c3691
Log 6027

Reports 6027

willyborankin

Please see my comments.

...er/src/main/java/org/apache/lucene/codecs/experimental/Lucene90CustomStoredFieldsFormat.java

server/src/main/java/org/apache/lucene/codecs/experimental/Lucene92CustomCodec.java

server/src/main/java/org/apache/lucene/codecs/experimental/ZstdCompressionMode.java

server/src/main/java/org/apache/lucene/codecs/experimental/LZ4CompressionMode.java

...er/src/main/java/org/apache/lucene/codecs/experimental/Lucene90CustomStoredFieldsFormat.java

willyborankin · 2022-06-19T19:40:47Z

@dblock @nknize I think we should take the sandbox approach for experimental features. We have the ability to provide CodecService from the plugin(s), using sandbox plugin for that would make sense. Thoughts?

IMHO I think we can add them as a new experimental feature with default behaviour as it works now.

mulugetam · 2022-06-21T08:23:34Z

@willyborankin have fixed the issues your raised. Thanks :-)

opensearch-ci-bot · 2022-06-21T08:24:51Z

✅ Gradle Check success 9c02870d30fb33ae000402d2ace647beba069188
Log 6176

Reports 6176

opensearch-ci-bot · 2022-06-21T08:51:09Z

✅ Gradle Check success 566217cba00fa09825b71c0b8e2dcff414a1c371
Log 6177

Reports 6177

willyborankin

Added a small question.

server/src/main/java/org/apache/lucene/codecs/experimental/Lucene92CustomCodec.java

server/src/main/java/org/apache/lucene/codecs/experimental/LZ4CompressionMode.java

opensearch-ci-bot · 2022-06-21T19:40:40Z

✅ Gradle Check success 11376fadc35b0e83b90bbfe4a5c621963e0c48f9
Log 6191

Reports 6191

willyborankin

LGTM. @reta, @kartg, @dblock and @mch2 Could you please review this PR?

server/src/main/resources/org/opensearch/bootstrap/security.policy

github-actions · 2023-03-31T17:37:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/13154/
CommitID: 06443ea

dblock · 2023-04-03T21:14:38Z

I'm going to merge on 🟢

dblock · 2023-04-03T21:25:03Z

@nknize we need you to dismiss your review/approve to merge

nknize

Lots of good movement here. Still feels open but at least it's sandboxed. LGFN...

sandbox/modules/custom-codecs/src/main/plugin-metadata/plugin-security.policy

mulugetam · 2023-04-03T22:53:21Z

@dblock Only the bc-fips vulnerability issue outstanding now. What is the plan? Remove it?

dblock · 2023-04-04T17:37:12Z

@dblock Only the bc-fips vulnerability issue outstanding now. What is the plan? Remove it?

Separate issue from this PR. Care to open one?

dblock · 2023-04-04T17:38:24Z

I tagged this as backport to 2.x, watch that pass/fail/may need manual labor.

* Add experimental support for zstd and lz4 (native) compression. Add experimental support for zstd (with and without dictionary support) and lz4 (native) compressions as discussed in: #3354. Users would be able to set the index.codec setting with the values "lz4_native" (for lz4 native), "zstd" (for zstd with dictionary), and "zstd_no_dict" (for zstd without a dictionary). Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix license issues, add tests for zstd, zstd_no_dict, and lz4_native compressions. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix DCO and and issues with CodecTests.java. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix forbidden api violation error for lz4-java. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix license headers. Remove and fix unnecessary fields. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix magic numbers. Use more restrictive access modifiers. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Use protected access modifier for Zstd and LZ4 compression mode classes. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Allow negative compression levels for zstd. Use more restrictive access modifiers. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Use a more restrictive permission for loading zstd-jni and lz4-java libraries. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Rename a file (follow a consistent version naming convention). Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Refactor and create a new custom-codecs sandbox module. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Remove blank lines. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Restore Lucene92CustomCodec to extend from FilterCodec. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Make use of the compressionLevel argument. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Make Lucene92CustomCodec abstract and use a package-private access modifier. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix missing JavaDoc issues. Remove unused field in PerFieldMappingPostingFormatCodec. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix lint errors. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix the description for the custom-codecs plugin. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix wildcard import and improve documentation. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Access control exception fixed. Removed lz4-java support for now. - PRs were made to zstd-jni and lz4-java to use AccessController.doPrivileged. - The zstd-jni PR is merged since version 1.5.4-1. - The lz4-java support temporarily removed until the PR gets merged. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Upgrade plugin to use Lucene95Codec. Rename files accordingly. - Upgrade plugin to use Lucene95Codec. Rename files accordingly. - Fix lint issue with plugin-security. - Remove thridPartyAudit that was there for supporting lz4-java. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Add test cases for compression/decompression. Other minor changes. - add test cases for compression/decompression. - rename package. - add a CHANGELOG entry. - add more checks for signed integer arithmetic. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Remove ES grant in plugin-security.policy. Fix minor javadoc issue. - Remove ES grant in plugin-security.policy file. - Replace @link and @see to fix javadoc error. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Upgrade jettison version to 1.5.4. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Update SHA for jettison 1.5.4. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> --------- Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> (cherry picked from commit f071c9b) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

reta · 2023-04-04T17:42:03Z

@dblock Only the bc-fips vulnerability issue outstanding now. What is the plan? Remove it?

Separate issue from this PR. Care to open one?

@dblock it is already in work: #6629

* Add experimental support for zstd and lz4 (native) compression. Add experimental support for zstd (with and without dictionary support) and lz4 (native) compressions as discussed in: opensearch-project#3354. Users would be able to set the index.codec setting with the values "lz4_native" (for lz4 native), "zstd" (for zstd with dictionary), and "zstd_no_dict" (for zstd without a dictionary). Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix license issues, add tests for zstd, zstd_no_dict, and lz4_native compressions. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix DCO and and issues with CodecTests.java. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix forbidden api violation error for lz4-java. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix license headers. Remove and fix unnecessary fields. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix magic numbers. Use more restrictive access modifiers. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Use protected access modifier for Zstd and LZ4 compression mode classes. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Allow negative compression levels for zstd. Use more restrictive access modifiers. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Use a more restrictive permission for loading zstd-jni and lz4-java libraries. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Rename a file (follow a consistent version naming convention). Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Refactor and create a new custom-codecs sandbox module. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Remove blank lines. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Restore Lucene92CustomCodec to extend from FilterCodec. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Make use of the compressionLevel argument. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Make Lucene92CustomCodec abstract and use a package-private access modifier. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix missing JavaDoc issues. Remove unused field in PerFieldMappingPostingFormatCodec. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix lint errors. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix the description for the custom-codecs plugin. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Fix wildcard import and improve documentation. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Access control exception fixed. Removed lz4-java support for now. - PRs were made to zstd-jni and lz4-java to use AccessController.doPrivileged. - The zstd-jni PR is merged since version 1.5.4-1. - The lz4-java support temporarily removed until the PR gets merged. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Upgrade plugin to use Lucene95Codec. Rename files accordingly. - Upgrade plugin to use Lucene95Codec. Rename files accordingly. - Fix lint issue with plugin-security. - Remove thridPartyAudit that was there for supporting lz4-java. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Add test cases for compression/decompression. Other minor changes. - add test cases for compression/decompression. - rename package. - add a CHANGELOG entry. - add more checks for signed integer arithmetic. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Remove ES grant in plugin-security.policy. Fix minor javadoc issue. - Remove ES grant in plugin-security.policy file. - Replace @link and @see to fix javadoc error. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Upgrade jettison version to 1.5.4. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Update SHA for jettison 1.5.4. Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> --------- Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Signed-off-by: Valentin Mitrofanov <mitrofmep@gmail.com>

* Add experimental support for zstd and lz4 (native) compression. Add experimental support for zstd (with and without dictionary support) and lz4 (native) compressions as discussed in: #3354. Users would be able to set the index.codec setting with the values "lz4_native" (for lz4 native), "zstd" (for zstd with dictionary), and "zstd_no_dict" (for zstd without a dictionary). * Fix license issues, add tests for zstd, zstd_no_dict, and lz4_native compressions. * Fix DCO and and issues with CodecTests.java. * Fix forbidden api violation error for lz4-java. * Fix license headers. Remove and fix unnecessary fields. * Fix magic numbers. Use more restrictive access modifiers. * Use protected access modifier for Zstd and LZ4 compression mode classes. * Allow negative compression levels for zstd. Use more restrictive access modifiers. * Use a more restrictive permission for loading zstd-jni and lz4-java libraries. * Rename a file (follow a consistent version naming convention). * Refactor and create a new custom-codecs sandbox module. * Remove blank lines. * Restore Lucene92CustomCodec to extend from FilterCodec. * Make use of the compressionLevel argument. * Make Lucene92CustomCodec abstract and use a package-private access modifier. * Fix missing JavaDoc issues. Remove unused field in PerFieldMappingPostingFormatCodec. * Fix lint errors. * Fix the description for the custom-codecs plugin. * Fix wildcard import and improve documentation. * Access control exception fixed. Removed lz4-java support for now. - PRs were made to zstd-jni and lz4-java to use AccessController.doPrivileged. - The zstd-jni PR is merged since version 1.5.4-1. - The lz4-java support temporarily removed until the PR gets merged. * Upgrade plugin to use Lucene95Codec. Rename files accordingly. - Upgrade plugin to use Lucene95Codec. Rename files accordingly. - Fix lint issue with plugin-security. - Remove thridPartyAudit that was there for supporting lz4-java. * Add test cases for compression/decompression. Other minor changes. - add test cases for compression/decompression. - rename package. - add a CHANGELOG entry. - add more checks for signed integer arithmetic. * Remove ES grant in plugin-security.policy. Fix minor javadoc issue. - Remove ES grant in plugin-security.policy file. - Replace @link and @see to fix javadoc error. * Upgrade jettison version to 1.5.4. * Update SHA for jettison 1.5.4. --------- (cherry picked from commit f071c9b) Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

reta · 2023-04-05T13:05:54Z

@mulugetam @dblock for 2.7.0, we should wait for ZSTD 1.5.5 update (the JNI one), they fixed rare corruption bug [1], thanks

[1] https://github.com/facebook/zstd/releases/tag/v1.5.5

- Add documentation for the custom-codecs plugin (opensearch-project/OpenSearch#3577). Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Add documentation for the custom-codecs plugin. - Add documentation for the custom-codecs plugin (opensearch-project/OpenSearch#3577). Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Update _api-reference/index-apis/create-index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _install-and-configure/plugins.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _install-and-configure/plugins.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Add documentation for the custom-codecs plugin. - Add documentation for the custom-codecs plugin (opensearch-project/OpenSearch#3577). Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Update _api-reference/index-apis/create-index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _install-and-configure/plugins.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _install-and-configure/plugins.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> (cherry picked from commit 817b730) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add documentation for the custom-codecs plugin. - Add documentation for the custom-codecs plugin (opensearch-project/OpenSearch#3577). * Update _api-reference/index-apis/create-index.md * Update _install-and-configure/plugins.md * Update _install-and-configure/plugins.md --------- (cherry picked from commit 817b730) Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

) * Add documentation for the custom-codecs plugin. - Add documentation for the custom-codecs plugin (opensearch-project/OpenSearch#3577). Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> * Update _api-reference/index-apis/create-index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _install-and-configure/plugins.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _install-and-configure/plugins.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

mulugetam requested review from a team and reta as code owners June 14, 2022 09:06

mulugetam mentioned this pull request Jun 14, 2022

Introduce more compression libraries and implementations #3354

Open

mulugetam force-pushed the add-zstd-lz4-support branch from 3de2f6b to 6999a54 Compare June 14, 2022 19:57

dblock requested changes Jun 15, 2022

View reviewed changes

server/src/main/java/org/apache/lucene/codecs/experimental/LZ4CompressionMode.java Outdated Show resolved Hide resolved

...er/src/main/java/org/apache/lucene/codecs/experimental/Lucene90CustomStoredFieldsFormat.java Outdated Show resolved Hide resolved

dblock requested review from mch2 and kartg June 15, 2022 13:13

mulugetam requested a review from dblock June 15, 2022 22:29

willyborankin suggested changes Jun 19, 2022

View reviewed changes

willyborankin reviewed Jun 21, 2022

View reviewed changes

server/src/main/java/org/apache/lucene/codecs/experimental/Lucene92CustomCodec.java Outdated Show resolved Hide resolved

willyborankin reviewed Jun 21, 2022

View reviewed changes

server/src/main/java/org/apache/lucene/codecs/experimental/LZ4CompressionMode.java Outdated Show resolved Hide resolved

willyborankin approved these changes Jun 22, 2022

View reviewed changes

reta requested changes Jun 22, 2022

View reviewed changes

server/src/main/resources/org/opensearch/bootstrap/security.policy Outdated Show resolved Hide resolved

dblock changed the title ~~Add experimental support for zstd and lz4 (native) compression algorithms.~~ Add experimental support for zstd compression. Apr 3, 2023

dblock approved these changes Apr 3, 2023

View reviewed changes

nknize approved these changes Apr 3, 2023

View reviewed changes

sandbox/modules/custom-codecs/src/main/plugin-metadata/plugin-security.policy Show resolved Hide resolved

dblock merged commit f071c9b into opensearch-project:main Apr 4, 2023

dblock added the backport 2.x Backport to 2.x branch label Apr 4, 2023

opensearch-trigger-bot bot mentioned this pull request Apr 4, 2023

[Backport 2.x] Add experimental support for zstd compression. #6982

Merged

martin-gaievski mentioned this pull request Apr 5, 2023

[BUG] Zstd new codec is a breaking change for kNN plugin #7012

Closed

mulugetam mentioned this pull request Apr 6, 2023

Fix: avoid ZSTD codec from overriding service codec factory. #7037

Merged

5 tasks

mulugetam mentioned this pull request May 1, 2023

Add documentation for the custom-codecs plugin. opensearch-project/documentation-website#3913

Merged

1 task

luyuncheng mentioned this pull request May 23, 2023

Reduce memory copy in zstd compression #7681

Merged

mulugetam mentioned this pull request May 23, 2023

[BUG] OpenSearch-Benchmark silently falls back and uses default for invalid index.codec values. opensearch-project/opensearch-benchmark#268

Closed

This was referenced Aug 17, 2023

[BUG] After enabling zstd codec memory usage increased drastically #9312

Closed

[Demote] ZStd compression from GA / LTS to experimental and release a 2.9.1 patch #9422

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add experimental support for zstd compression. #3577

Add experimental support for zstd compression. #3577

mulugetam commented Jun 14, 2022 •

edited

Loading

opensearch-ci-bot commented Jun 14, 2022

willyborankin commented Jun 14, 2022

reta commented Jun 14, 2022

mulugetam commented Jun 14, 2022 •

edited

Loading

opensearch-ci-bot commented Jun 14, 2022

mulugetam commented Jun 14, 2022

dblock commented Jun 14, 2022

dblock commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 15, 2022

dblock left a comment

opensearch-ci-bot commented Jun 15, 2022

willyborankin left a comment •

edited

Loading

willyborankin commented Jun 19, 2022

mulugetam commented Jun 21, 2022

opensearch-ci-bot commented Jun 21, 2022

opensearch-ci-bot commented Jun 21, 2022

willyborankin left a comment

opensearch-ci-bot commented Jun 21, 2022

willyborankin left a comment

github-actions bot commented Mar 31, 2023

dblock commented Apr 3, 2023

dblock commented Apr 3, 2023

nknize left a comment

mulugetam commented Apr 3, 2023

dblock commented Apr 4, 2023

dblock commented Apr 4, 2023

reta commented Apr 4, 2023

reta commented Apr 5, 2023

Add experimental support for zstd compression. #3577

Add experimental support for zstd compression. #3577

Conversation

mulugetam commented Jun 14, 2022 • edited Loading

Description

Issues Resolved

Check List

opensearch-ci-bot commented Jun 14, 2022

willyborankin commented Jun 14, 2022

reta commented Jun 14, 2022

mulugetam commented Jun 14, 2022 • edited Loading

opensearch-ci-bot commented Jun 14, 2022

mulugetam commented Jun 14, 2022

dblock commented Jun 14, 2022

dblock commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 15, 2022

dblock left a comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 15, 2022

willyborankin left a comment • edited Loading

Choose a reason for hiding this comment

willyborankin commented Jun 19, 2022

mulugetam commented Jun 21, 2022

opensearch-ci-bot commented Jun 21, 2022

opensearch-ci-bot commented Jun 21, 2022

willyborankin left a comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 21, 2022

willyborankin left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 31, 2023

Gradle Check (Jenkins) Run Completed with:

dblock commented Apr 3, 2023

dblock commented Apr 3, 2023

nknize left a comment

Choose a reason for hiding this comment

mulugetam commented Apr 3, 2023

dblock commented Apr 4, 2023

dblock commented Apr 4, 2023

reta commented Apr 4, 2023

reta commented Apr 5, 2023

mulugetam commented Jun 14, 2022 •

edited

Loading

mulugetam commented Jun 14, 2022 •

edited

Loading

willyborankin left a comment •

edited

Loading