Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes #22910

Merged

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Jun 14, 2024

Fixes #22909

Motivation

Netty's optimized direct byte buffer support is needed for high performance. Currently these debug messages will get logged when Netty library gets initialized:

  • "direct buffer constructor: unavailable: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module"
  • "jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module"
  • "java.nio.DirectByteBuffer.(long, {int,long}): unavailable {}"

Another detail is that Pulsar uses com.scurrilous.circe.checksum.Crc32cIntChecksum class for checksum calculation. For optimized checksum calculation Pulsar attempts to load libcirce-checksum native library. When that isn't available, com.scurrilous.circe.checksum.Java9IntHash is used. That only works when "--add-opens java.base/java.util.zip=ALL-UNNAMED" is passed on the command line.

Modifications

Pass these parameters in the Java runtime command line:

 -Dio.netty.tryReflectionSetAccessible=true
 -Dorg.apache.pulsar.shade.io.netty.tryReflectionSetAccessible=true
 -Dio.grpc.netty.shaded.io.netty.tryReflectionSetAccessible=true
 --add-opens java.base/java.nio=ALL-UNNAMED
 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED
 --add-opens java.base/java.util.zip=ALL-UNNAMED

remove --add-opens java.base/sun.net=ALL-UNNAMED added in #16423 since it's no longer needed after #22329

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Copy link
Contributor

@cbornet cbornet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@lhotari lhotari merged commit f3d4d5a into apache:master Jun 17, 2024
57 of 62 checks passed
lhotari added a commit that referenced this pull request Jun 19, 2024
lhotari added a commit that referenced this pull request Jun 19, 2024
lhotari added a commit that referenced this pull request Jun 19, 2024
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 21, 2024
…r Function runtimes (apache#22910)

(cherry picked from commit f3d4d5a)
(cherry picked from commit 20de952)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 24, 2024
…r Function runtimes (apache#22910)

(cherry picked from commit f3d4d5a)
(cherry picked from commit 20de952)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 25, 2024
…r Function runtimes (apache#22910)

(cherry picked from commit f3d4d5a)
(cherry picked from commit 20de952)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 1, 2024
…r Function runtimes (apache#22910)

(cherry picked from commit f3d4d5a)
(cherry picked from commit 20de952)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Pulsar Functions Runtime doesn't properly enable direct byte buffer access for Netty on Java 17+
3 participants