[BUG] After enabling zstd codec memory usage increased drastically

[shamil]

Describe the bug

After enabling zstd codec memory usage increased drastically.

To Reproduce

1. Create index with zstd codec
2. Ingest data to the index, we have around 300GB+ per index
3. Comparing memory usage before and after the index with zstd memory usage increases drastically

Expected behavior

Memory usage doesn't increase that drastically

Screenshots

Here are memory graphs for 2 nodes from our cluster, it's clearly visible memory usage after enabling zstd codec and new indexes created (on 8/10). The memory usage is from the OS metrics.

Host/Environment (please complete the following information):

• OS: Ubunttu 22.04
• Version 2.9.0

[mgodwan]

@shamil I observed similar behaviour when using ZSTD codec.
This can be related to #9403

[shamil]

@mgodwan yes, that might be it. Is this planned to be relased soon as part of 2.9.x version or will be only in 2.10+?

[nknize]

🤦🏻‍♂️ Oye. So I'm kind of confused here. Zstd codec was sandboxed for almost three months. Was this never used while sandboxed such that this regression could be benchmarked and tested before promoting as a LTS feature?

Do we need to now add an extra bake layer in the form of FEATURE_FLAG sandbox promoted features to prevent this from happening in the future?

[nknize]

Hmmm.... why not switch to direct memory and keep the pressure off the heap?

try (ZstdDictDecompress dictDecompress = new ZstdDictDecompress(ByteBuffer.wrap(bytes.bytes, 0, dictLength))) {

[nknize]

FTR introducing a core feature that relies on native libraries is SUPER trappy. I didn't realize the native dependencies when I reviewed the first experimental PR. Our portability could easily be burned by incompatibilities in glibc (we ran into this on kNN w/ FAISS not building on M1 macbooks). We need to seriously think about moving all zstd out as optional plugins. Perhaps we even cut a patch release that deprecates this w/ a DeprecationLogger warning that says zstd compression will be moving to an optional plugin in the next release.

[sandervandegeijn]

What is the solution? To pull the future?

[reta]

Hmmm.... why not switch to direct memory and keep the pressure off the heap?

try (ZstdDictDecompress dictDecompress = new ZstdDictDecompress(ByteBuffer.wrap(bytes.bytes, 0, dictLength))) {

It looks like the ZstdDictDecompress was leaking native memory, the heap seems to be fine (but we could double check for sure).

[sandervandegeijn]

What about current users, will demoting the feature break the cluster unless they enable the experimental feature? Maybe provide upgrade instructions

[dblock]

@andrross should we close this? I believe the issue itself is fixed in 2.10

[andrross]

The fix for this will be available in the 2.10 release

[dblock]

What about current users, will demoting the feature break the cluster unless they enable the experimental feature? Maybe provide upgrade instructions

With opensearch-project/opensearch-build#3971 this won't be the case for users of the official distribution.