kafka-client with SASL aws-msk-iam-auth #39548
Comments
|
/cc @Karm (mandrel), @alesj (kafka), @cescoffier (kafka), @galderz (mandrel), @geoand (kotlin), @ozangunalp (kafka), @zakkak (mandrel) |
|
I can privately share temp AWS credentials and MSK endpoints |
|
@jvdadda I can try. Email me at first name at redhat com. |
|
I've found the problem. The issue is related to this methods: public String[] getMechanismNames(Map<String, ?> props) {
return new String[]{IAMSaslClient.getMechanismNameForClassLoader(this.getClass().getClassLoader())};
}And: public static String getMechanismNameForClassLoader(ClassLoader classLoader) {
return "AWS_MSK_IAM." + classLoader.hashCode();
}The security providers Thankfully there's a very easy workaround that you can apply. Build the native with: Building it like this will make I will check with those maintaining the AWS integration to make the above permanent. |
|
Wow thanks @galderz, for the explanation, and for the fix ! Maintainers of the library are not really reactive, I made lot of modifications to make it work until this problem (that's why I included the jar, and not the dependency). I will try to fix it on my fork and propose a PR on the main repo |
Which maintainers are you talking about? Amazon AWS or the ones that contributed https://github.com/quarkiverse/quarkus-amazon-services? Do you have any links to public issues/discussions that have gone unanswered?
I have created quarkiverse/quarkus-amazon-services#1204 to get this fix permanently. The changes required are small and there are plenty of examples on how to do it in the main https://github.com/quarkusio/quarkus repo. |
I talk about AWS (https://github.com/aws/aws-msk-iam-auth), the library still uses the AWS v1 SDK that create lot of problems (mainly Random class used in static fields) and that have not support about native compilation (contrary to AWS SDK v2). The project I give you have a modified library jar (https://github.com/jvdadda/debug-aws-msk-iam-auth/tree/main/libs) with v2 usage (PR in the project are opened without answers). If you replace https://github.com/jvdadda/debug-aws-msk-iam-auth/blob/main/build.gradle.kts#L27 with: my example project will not work anymore.
|
Describe the bug
When using the aws-msk-iam-auth library, with native build, I am unable to connect to kafka broker, I have an exception
java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]All is working as expected with jvm build.
Expected behavior
Like jvm build, the producer should be able to connect to broker and then produce a message, here is the logs when it works:
Actual behavior
Here is the complete logs, including the exception:
How to Reproduce?
Use this project : https://github.com/jvdadda/debug-aws-msk-iam-auth
You need to replace a value in application.properties with the list of endpoints of a public MSK cluster with IAM enabled, then replace in Makefile your credentials that can access to the cluster.
Then, do a
make build/native startormake build/jvm startOutput of
uname -aorverDarwin mob-mac-53aa52 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64
Output of
java -versionopenjdk version "21.0.2" 2024-01-16 OpenJDK Runtime Environment (build 21.0.2+13-58) OpenJDK 64-Bit Server VM (build 21.0.2+13-58, mixed mode, sharing)
Mandrel or GraalVM version (if different from Java)
Same on Mandrel and GraalVM
Quarkus version or git rev
3.8.2
Build tool (ie. output of
mvnw --versionorgradlew --version)Additional information
No response
The text was updated successfully, but these errors were encountered: