Open
Description
Code of Conduct
- I agree to follow this project's Code of Conduct
Search before asking
- I have searched in the issues and found no similar issues.
Describe the bug
Building a docker image via deploy/kubernetes/docker/build.sh --hadoop-version 3.2.4 and using that with Netty and Epoll fails because the image contains the Hadoop binaries, which contain a different netty-all jar then the RSS shuffle server.
This can be fixed by
- either not include the Hadoop binaries in the Docker image
- or by downgrading Netty version to match the version contained in the Hadoop binaries
Affects Version(s)
master
Uniffle Server Log Output
Exception in thread "main" java.lang.UnsatisfiedLinkError: failed to load the required native library
at io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:81)
at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:57)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:189)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:37)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:49)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:117)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:81)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:57)
at org.apache.uniffle.server.netty.StreamServer.<init>(StreamServer.java:88)
at org.apache.uniffle.server.ShuffleServer.initialization(ShuffleServer.java:300)
at org.apache.uniffle.server.ShuffleServer.<init>(ShuffleServer.java:113)
at org.apache.uniffle.server.ShuffleServer.main(ShuffleServer.java:131)
Caused by: java.lang.ExceptionInInitializerError
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:40)
... 15 more
Caused by: java.lang.IllegalStateException: Multiple resources found for 'META-INF/native/libnetty_transport_native_epoll_x86_64.so' with different content: [jar:file:/data/rssadmin/rss/jars/server/netty-transport-native-epoll-4.1.109.Final-linux-x86_64.jar!/META-INF/native/libnetty_transport_native_epoll_x86_64.so, jar:file:/data/rssadmin/hadoop/share/hadoop/hdfs/lib/netty-all-4.1.68.Final.jar!/META-INF/native/libnetty_transport_native_epoll_x86_64.so]
at io.netty.util.internal.NativeLibraryLoader.getResource(NativeLibraryLoader.java:301)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:334)
at io.netty.channel.epoll.Native.<clinit>(Native.java:96)
... 16 more
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
_No response_
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Metadata
Metadata
Assignees
Labels
No labels