Kafka compressions Snappy and LZ4 not supported with native builds

[solsson]

This limitation is documented in #977 but not in https://quarkus.io/guides/kafka-guide. My question is what level of effort went in to trying to support snappy and gzip? Is JNI with native somehow ruled out, or might it be worth a try after #2412?

Describe the bug

Native builds disable support for message compression types Snappy and Gzip. Any attempt to consume or produce will result in an exception where the compression type is reported as an integer. The enumeration is in https://github.com/apache/kafka/blob/2.2.1/clients/src/main/java/org/apache/kafka/common/record/CompressionType.java.

A review comment in #977 documents the "hack" in https://github.com/quarkusio/quarkus/blob/0.16/extensions/kafka-client/runtime/src/main/java/io/quarkus/kafka/client/runtime/graal/SubstituteSnappy.java.

Expected behavior

Kafka consumers typically handle compression transparently. Compression is per message, not per topic. The broker enumerates supported compressions. Also messages are immutable and AFAIK there is no other way to remove compression than to rewrite the topic.

Actual behavior

For example when trying to consume a message compressed with Snappy:

java.lang.IllegalArgumentException: Unknown or unsupported compression type id: 2
	at org.apache.kafka.common.record.CompressionType.forId(CompressionType.java:61)
	at org.apache.kafka.common.record.DefaultRecordBatch.compressionType(DefaultRecordBatch.java:211)

I haven't tested ZStandard yet. It isn't explicitly disabled, but I think it too requires a native lib.

To Reproduce

1. Run https://quarkus.io/guides/kafka-guide
2. Produce using a non-quarkus kafka client with snappy compression.
3. The aforementioned exception occurs

Environment (please complete the following information):

• Output of uname -a or ver: I've only run in docker.
• Output of java -version: 8 and 11
• GraalVM version (if different from Java): 1.0 rc15 and rc16 tested
• Quarkus version or git rev: 0.14 through 0.16 tested

Workaround

Use a jvm build. Note that with the FROM in the generated Dockerfile (fabric8/java-alpine-openjdk8-jre) you must add RUN apk add --no-cache libc6-compat in order to avoid the error java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.7-3d2397f3-516f-45ab-aad1-94be731682f5-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory.

[solsson]

There's https://github.com/airlift/aircompressor but I suppose a switch would involve a patch to Kafka client source.

[solsson]

Can we do something like #2794?

[cescoffier]

It's a bit more tricky as Graal does not support optional "substitutions".

In an ideal world, we would remove Snappy if we can't find it in the classpath. I might be somewhat possible for some of the substitution (methods) but not for all (class substitution).

[solsson]

Will #3060 help? How about https://github.com/quarkusio/quarkus/projects/10?

In an ideal world, we would remove Snappy if we can't find it in the classpath

Given how Kafka clients fail at runtime with the rather opaque error message described above, what's the rationale for removing the compression types at build time? I suppose an alternative could be to let it fail at runtime with an error that is either class loading or JNI, depending on how the build was produced. IMO the latter provides a way forward, while the current approach rules out any attempt at supporting these compressions.

Edit: to clarify, I'd be happy to experiment with JNI, but first I'd have to un-patch the Kafka client. I asked initially if support has somehow been ruled out? If so the un-patch would never land in the Quarkus code base and thus the experiments with JNI would be a waste of time.

[solsson]

From #2999 it looks like JNI support is being strengthened. I also noticed that +JNI is default now in native image builds.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you!
We are doing this automatically to ensure out-of-date issues does not stay around indefinitely.
If you believe this issue is still relevant please put a comment on it on why and if it truly needs to stay request or add 'pinned' label.

[gsmet]

I think we have optional substitutions now with onlyWith so it might be doable.

[gsmet]

@irenakezic this one might be doable now. See my last comment about adding conditions to the substitions depending on what we have in the classpath (there are a few examples in the code). It might be tricky to test though but you know Kafka so that might help :).

[irenakezic]

@gsmet ok thx! I will give it a try but probably not before the weekend.

[irenakezic]

@gsmet sorry for taking so long to look into this.
From what I can see there is a problem with MethodHandle in GraalVM (as described here oracle/graal#955) which is used for snappy compression type. I am not sure if I should try to make further substitutions for CompressionType to try to resolve this?
Is the idea to do the following:

1. If snappy is not on classpath use current implementation for CompressionType supstitution
2. If snappy is on classpath implement CompressionType substitiutions to avoid MethodHandle.

[marko-mets-gt]

Is the fix for supporting these compression algorithms doable or not? I also see that ZSTD is also not supported in quarkus.

[cescoffier]

It's now doable as GraalVM has added optional substitutions.

[spike83]

I was able to make a workaround.

Therefore I removed https://github.com/quarkusio/quarkus/blob/master/extensions/kafka-client/runtime/src/main/java/io/quarkus/kafka/client/runtime/graal/SubstituteSnappy.java#L26-L65 and builded snapshot

In my project I added the snappy java dependency:

implementation group: 'org.xerial.snappy', name: 'snappy-java', version: '1.1.8.4'

In addition the native library snappyjava needs to be loaded by:

System.loadLibrary("snappyjava");

This native library file libsnappyjava.so has to be in the java.library.path which includes the
/work directory of the docker container (ubi-minimal in my case), but it does not automatically go there.

Extract the snappy-java-x.y.z.jar and add the library by

COPY native/org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so /work/libsnappyjava.so
RUN chmod +x /work/libsnappyjava.so

The class org.xerial.snappy.SnappyInputStream.class has to be registered for reflection.

I would be happy to try to create a pr. But I would need some directions e.g. What would be the right way of adding the native libraries? How should this in general be approached?

Would it be an option to just remove the SubstituteSnappy class? One would get kind of helpful exceptions I think.

[cescoffier]

Thanks @spike83! That's a great step forward!
I will have a look this week.

Here are a few thoughts:

1. The native library could be handled by the extension. If you have the snappy dependency, we can add it to the native image instructions.
2. Once we have a proper integration, we can remove the SubstituteSnappy, but with your work, sounds almost done! We can also have a conditional substitution checking if the snappy classes are on the classpath.

Which version of GraalVM did you use?

[spike83]

@cescoffier I'm running the build in docker image quay.io/quarkus/ubi-quarkus-native-image:21.0.0-java11 (id: 31ccea2b17ae) so I would say 21.0.

[cescoffier]

Perfect thanks! MethodHandle support has been added in that version, which explains why you don't need any substitutions.

[mvrk57]

Hi guys :-) Im trying to use snappy in a native quarkus application too.
Is the only way to get it running to do it the way @spike83 described? Create a snapshot, remove SubstituteSnappy and load the precompiled lib?

Thanks for your help :)

[cescoffier]

At the moment yes. I will be working on it soon.

[mvrk57]

thanks a lot! for now I got it working with the workaround. So no need for a hurry ;-)
Have a good day!

[cescoffier]

#15174 provides an initial support.

The library is embedded in the native image (need to enable snappy and use GraalVM 21+) and loaded when the application starts.

The funny thing is that 4 days ago the MethodHandle used in the Kafka client have been removed... However, it would not enable the support on GraalVM 20.3 as it also needs serialization support.