Epoll native library does NOT load after upgrade between 4.1.39 and 4.1.40 #9758

ursaj · 2019-11-05T16:35:24Z

Expected behavior

Native epoll support keeps working after upgrade.

Actual behavior

Server apps fail on startup with the following stack trace:

...
Caused by: java.lang.UnsatisfiedLinkError: failed to load the required native library
	at io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:80)
	at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:52)
	at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142)
	at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68)
	at com.db.abos.common.api.transport.netty.NettyOsFactory.newGroup(NettyOsFactory.java:66)
	at com.db.abos.common.api.transport.netty.ServerWebSocketFactory.<init>(ServerWebSocketFactory.java:91)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
	... 19 common frames omitted
Caused by: java.lang.UnsatisfiedLinkError: /tmp/libnetty_transport_native_epoll_x86_643970988024581631659.so: /lib64/libc.so.6: version `GLIBC_2.12' not found (required by /tmp/libnetty_transport_native_epoll_x86_643970988024581631659.so)
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
	at java.lang.Runtime.load0(Runtime.java:810)
	at java.lang.System.load(System.java:1086)
	at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
	at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
	at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:203)
	at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:228)
	at io.netty.channel.epoll.Native.<clinit>(Native.java:57)
	at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
	... 36 common frames omitted
	Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_transport_native_epoll_x86_643970988024581631659.so: /lib64/libc.so.6: version `GLIBC_2.12' not found (required by /tmp/libnetty_transport_native_epoll_x86_643970988024581631659.so)
		at java.lang.ClassLoader$NativeLibrary.load(Native Method)
		at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
		at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
		at java.lang.Runtime.load0(Runtime.java:810)
		at java.lang.System.load(System.java:1086)
		at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
		at java.security.AccessController.doPrivileged(Native Method)
		at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
		at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
		... 40 common frames omitted
	Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
		at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
		at java.lang.Runtime.loadLibrary0(Runtime.java:871)
		at java.lang.System.loadLibrary(System.java:1122)
		at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
		at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
		... 39 common frames omitted
		Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
			at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
			at java.lang.Runtime.loadLibrary0(Runtime.java:871)
			at java.lang.System.loadLibrary(System.java:1122)
			at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
			at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
			at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
			at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.lang.reflect.Method.invoke(Method.java:498)
			at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
			at java.security.AccessController.doPrivileged(Native Method)
			at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
			at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
			... 40 common frames omitted
	Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:224)
		at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:231)
		... 38 common frames omitted
	Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll.so
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
		... 39 common frames omitted
		Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
			at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
			at java.lang.Runtime.loadLibrary0(Runtime.java:871)
			at java.lang.System.loadLibrary(System.java:1122)
			at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
			at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
			at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
			... 39 common frames omitted
			Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
				at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
				at java.lang.Runtime.loadLibrary0(Runtime.java:871)
				at java.lang.System.loadLibrary(System.java:1122)
				at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
				at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
				at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
				at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
				at java.lang.reflect.Method.invoke(Method.java:498)
				at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
				at java.security.AccessController.doPrivileged(Native Method)
				at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
				at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
				... 40 common frames omitted

Steps to reproduce

assertNull(Epoll.unavailabilityCause())

Netty version

4.1.39 - epoll works
4.1.40 - epoll does NOT work

JVM version (e.g. `java -version`)

% java -version
openjdk version "1.8.0_221"
OpenJDK Runtime Environment (Zulu 8.39.0.8-SA-linux64) (build 1.8.0_221-b08)
OpenJDK 64-Bit Server VM (Zulu 8.39.0.8-SA-linux64) (build 25.221-b08, mixed mode)

OS version (e.g. `uname -a`)

% uname -a
Linux gmafxu43 3.0.101-108.98-default #1 SMP Mon Jul 15 13:58:06 UTC 2019 (262a94d) x86_64 x86_64 x86_64 GNU/Linux

% cat /etc/os-release
NAME="SLES"
VERSION="11.4"
VERSION_ID="11.4"
PRETTY_NAME="SUSE Linux Enterprise Server 11 SP4"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:11:4"

The text was updated successfully, but these errors were encountered:

normanmaurer · 2019-11-05T16:56:24Z

What is the Linux distro and Version ?

…

Am 05.11.2019 um 17:53 schrieb ursa ***@***.***>: Expected behavior Native epoll support keeps working after upgrade. Actual behavior Server apps fail on startup with the following stack trace: ... Caused by: java.lang.UnsatisfiedLinkError: failed to load the required native library at io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:80) at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:52) at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142) at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) at com.db.abos.common.api.transport.netty.NettyOsFactory.newGroup(NettyOsFactory.java:66) at com.db.abos.common.api.transport.netty.ServerWebSocketFactory.<init>(ServerWebSocketFactory.java:91) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142) ... 19 common frames omitted Caused by: java.lang.UnsatisfiedLinkError: /tmp/libnetty_transport_native_epoll_x86_643970988024581631659.so: /lib64/libc.so.6: version `GLIBC_2.12' not found (required by /tmp/libnetty_transport_native_epoll_x86_643970988024581631659.so) at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at java.lang.Runtime.load0(Runtime.java:810) at java.lang.System.load(System.java:1086) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349) at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:203) at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:228) at io.netty.channel.epoll.Native.<clinit>(Native.java:57) at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39) ... 36 common frames omitted Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_transport_native_epoll_x86_643970988024581631659.so: /lib64/libc.so.6: version `GLIBC_2.12' not found (required by /tmp/libnetty_transport_native_epoll_x86_643970988024581631659.so) at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at java.lang.Runtime.load0(Runtime.java:810) at java.lang.System.load(System.java:1086) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369) at java.security.AccessController.doPrivileged(Native Method) at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339) ... 40 common frames omitted Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:871) at java.lang.System.loadLibrary(System.java:1122) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349) at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136) ... 39 common frames omitted Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:871) at java.lang.System.loadLibrary(System.java:1122) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369) at java.security.AccessController.doPrivileged(Native Method) at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339) ... 40 common frames omitted Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:224) at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:231) ... 38 common frames omitted Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll.so at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173) ... 39 common frames omitted Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:871) at java.lang.System.loadLibrary(System.java:1122) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349) at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136) ... 39 common frames omitted Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:871) at java.lang.System.loadLibrary(System.java:1122) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369) at java.security.AccessController.doPrivileged(Native Method) at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339) ... 40 common frames omitted Steps to reproduce assertNull(Epoll.unavailabilityCause()) Netty version 4.1.39 - epoll works 4.1.43 - epoll does NOT work JVM version (e.g. java -version) % java -version openjdk version "1.8.0_221" OpenJDK Runtime Environment (Zulu 8.39.0.8-SA-linux64) (build 1.8.0_221-b08) OpenJDK 64-Bit Server VM (Zulu 8.39.0.8-SA-linux64) (build 25.221-b08, mixed mode) OS version (e.g. uname -a) % uname -a Linux gmafxu43 3.0.101-108.98-default #1 SMP Mon Jul 15 13:58:06 UTC 2019 (262a94d) x86_64 x86_64 x86_64 GNU/Linux — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

ursaj · 2019-11-05T20:41:45Z

Updated description

carl-mastrangelo · 2019-11-06T01:17:56Z

Can you give the exact maven / gradle dependency you are using? I think you need to depend on the os specific version.

ursaj · 2019-11-06T09:27:57Z

Can you give the exact maven / gradle dependency you are using?

% mvn --projects api\serverapi dependency:tree | grep netty | tee build.log
5697 [BuilderThread 0] [INFO] |  +- io.netty:netty-transport:jar:4.1.43.Final:compile
5698 [BuilderThread 0] [INFO] |  |  \- io.netty:netty-resolver:jar:4.1.43.Final:compile
5698 [BuilderThread 0] [INFO] |  +- io.netty:netty-handler:jar:4.1.43.Final:compile
5698 [BuilderThread 0] [INFO] |  |  \- io.netty:netty-codec:jar:4.1.43.Final:compile
5699 [BuilderThread 0] [INFO] |  \- io.netty:netty-codec-http:jar:4.1.43.Final:compile
5701 [BuilderThread 0] [INFO] |     \- io.netty:netty-all:jar:4.1.43.Final:compile
5705 [BuilderThread 0] [INFO] +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.43.Final:compile
5706 [BuilderThread 0] [INFO] |  +- io.netty:netty-common:jar:4.1.43.Final:compile
5706 [BuilderThread 0] [INFO] |  +- io.netty:netty-buffer:jar:4.1.43.Final:compile
5707 [BuilderThread 0] [INFO] |  \- io.netty:netty-transport-native-unix-common:jar:4.1.43.Final:compile
5707 [BuilderThread 0] [INFO] +- io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.7.Final:compile
5723 [BuilderThread 0] [INFO] +- org.glassfish.jersey.containers:jersey-container-netty-http:jar:2.28:compile
5724 [BuilderThread 0] [INFO] |  \- org.glassfish.jersey.connectors:jersey-netty-connector:jar:2.28:compile

I think you need to depend on the os specific version.

it would be strange because with version 4.1.39 exactly the same dependencies work pretty fine.

normanmaurer · 2019-11-06T09:34:25Z

@ursaj can you show me your libc version ?

ursaj · 2019-11-06T18:18:12Z

sure, if you tell me, what should I be looking for. I'm not familiar with native libraries...

normanmaurer · 2019-11-06T18:21:22Z

Just run ldd-v

…

Am 06.11.2019 um 19:18 schrieb ursa ***@***.***>: sure, if you tell me, what should I be looking for. I'm not familiar with native libraries... — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

ursaj · 2019-11-06T22:40:34Z

% ldd -v /lib64/libc.so.6
		/lib64/ld-linux-x86-64.so.2 (0x00007f175f4c5000)
		linux-vdso.so.1 =>  (0x00007fff373d5000)

		Version information:
		/lib64/libc.so.6:
				ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
				ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

this version is used on ~250 boxes, so updating it is not an option.

carl-mastrangelo · 2019-11-07T01:32:20Z

glibc 2.3 was released in 2002. I think you need to upgrade. If not, you can compile the native portion of netty on box, with the correct linkage.

normanmaurer · 2019-11-07T07:38:17Z

@ursaj huh... 2.3 ? That is super strange and I don't think we changed anything in how we build stuff. Can you try to upgrade one version after the other and let me know in which version it exactly broke ?

ursaj · 2019-11-07T11:15:09Z

Checker:

package test;

import io.netty.channel.epoll.Epoll;

/**
 * Dependencies:<pre>
 *     io.netty:netty-transport-native-epoll:${project.version}:linux-x86_64
 * </pre>
 */
public class NettyEpollChecker {
    public static void main(String[] args) {
        System.out.println("Epoll.isAvailable: " + Epoll.isAvailable());
        Throwable cause = Epoll.unavailabilityCause();
        if (cause != null) {
            cause.printStackTrace(System.out);
        }
    }
}

Output:

% java -jar netty-epoll-checker-4.1.39.Final.jar
Epoll.isAvailable: true

% java -jar netty-epoll-checker-4.1.40.Final.jar
Epoll.isAvailable: false
java.lang.UnsatisfiedLinkError: /tmp/libnetty_transport_native_epoll_x86_641851005851198575174.so: /lib64/libc.so.6: version `GLIBC_2.12' not found (required by /tmp/libnetty_transport_native_epoll_x86_641851005851198575174.so)
		at java.lang.ClassLoader$NativeLibrary.load(Native Method)
		at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
		at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
		at java.lang.Runtime.load0(Runtime.java:810)
		at java.lang.System.load(System.java:1086)
		at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
		at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:203)
		at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:218)
		at io.netty.channel.epoll.Native.<clinit>(Native.java:59)
		at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
		at test.NettyEpollChecker.main(NettyEpollChecker.java:12)
		Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_transport_native_epoll_x86_641851005851198575174.so: /lib64/libc.so.6: version `GLIBC_2.12' not found (required by /tmp/libnetty_transport_native_epoll_x86_641851005851198575174.so)
				at java.lang.ClassLoader$NativeLibrary.load(Native Method)
				at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
				at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
				at java.lang.Runtime.load0(Runtime.java:810)
				at java.lang.System.load(System.java:1086)
				at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
				at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
				at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
				at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
				at java.lang.reflect.Method.invoke(Method.java:498)
				at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
				at java.security.AccessController.doPrivileged(Native Method)
				at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
				at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
				... 5 more
		Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
				at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
				at java.lang.Runtime.loadLibrary0(Runtime.java:871)
				at java.lang.System.loadLibrary(System.java:1122)
				at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
				at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
				at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
				... 4 more
				Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
						at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
						at java.lang.Runtime.loadLibrary0(Runtime.java:871)
						at java.lang.System.loadLibrary(System.java:1122)
						at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
						at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
						at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
						at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
						at java.lang.reflect.Method.invoke(Method.java:498)
						at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
						at java.security.AccessController.doPrivileged(Native Method)
						at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
						at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
						... 5 more
		Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll
				at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:224)
				at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:221)
				... 3 more
		Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll.so
				at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
				... 4 more
				Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
						at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
						at java.lang.Runtime.loadLibrary0(Runtime.java:871)
						at java.lang.System.loadLibrary(System.java:1122)
						at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
						at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
						at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
						... 4 more
						Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
								at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
								at java.lang.Runtime.loadLibrary0(Runtime.java:871)
								at java.lang.System.loadLibrary(System.java:1122)
								at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
								at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
								at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
								at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
								at java.lang.reflect.Method.invoke(Method.java:498)
								at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
								at java.security.AccessController.doPrivileged(Native Method)
								at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
								at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
								... 5 more

normanmaurer · 2019-11-07T13:17:11Z

ok interesting... let me try to find out what changed

normanmaurer · 2019-11-07T13:34:27Z

@ursaj can you paste the output of : ldd --version ?

ursaj · 2019-11-07T23:25:25Z

% ldd --version
ldd (GNU libc) 2.11.3
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

normanmaurer · 2019-11-08T08:12:15Z

hmm that's super strange. We use centos 6 to compile our native epoll transport which uses glibc. 2.12 since forever. I fail how there could be any difference :(

ursaj · 2019-11-08T19:41:39Z

--- % ldd -v libnetty_transport_native_epoll_x86_64_4.1.39.so
---         linux-vdso.so.1 =>  (0x00007ffd53dc5000)
---         librt.so.1 => /lib64/librt.so.1 (0x00007fa7c037d000)
---         libc.so.6 => /lib64/libc.so.6 (0x00007fa7bfffd000)
---         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa7bfddd000)
---         /lib64/ld-linux-x86-64.so.2 (0x00007fa7c07a5000)
+++ % ldd -v libnetty_transport_native_epoll_x86_64_4.1.40.so
+++ ./libnetty_transport_native_epoll_x86_64_4.1.40.so: /lib64/libc.so.6: version `GLIBC_2.12' not found (required by ./libnetty_transport_native_epoll_x86_64_4.1.40.so)
+++         linux-vdso.so.1 =>  (0x00007ffec3d8d000)
+++         librt.so.1 => /lib64/librt.so.1 (0x00007fda30f85000)
+++         libc.so.6 => /lib64/libc.so.6 (0x00007fda30c05000)
+++         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fda309e5000)
+++         /lib64/ld-linux-x86-64.so.2 (0x00007fda313ad000)

            Version information:
---         ./libnetty_transport_native_epoll_x86_64_4.1.39.so:
+++         ./libnetty_transport_native_epoll_x86_64_4.1.40.so:
                    librt.so.1 (GLIBC_2.2.5) => /lib64/librt.so.1
                    libc.so.6 (GLIBC_2.8) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_2.10) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_2.7) => /lib64/libc.so.6
+++                 libc.so.6 (GLIBC_2.12) => not found
                    libc.so.6 (GLIBC_2.5) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_2.3.2) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_2.9) => /lib64/libc.so.6
            /lib64/librt.so.1:
                    ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                    libpthread.so.0 (GLIBC_PRIVATE) => /lib64/libpthread.so.0
                    libpthread.so.0 (GLIBC_2.2.5) => /lib64/libpthread.so.0
                    libc.so.6 (GLIBC_2.3.2) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
            /lib64/libc.so.6:
                    ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                    ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
            /lib64/libpthread.so.0:
                    ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                    ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                    ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                    libc.so.6 (GLIBC_2.3.2) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
                    libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6

ursaj · 2019-11-08T20:41:43Z

a) Maybe there are some logs from build, that could clarify this (new) dependency on GLIBC_2.12 ?
b) Could it be a change in build environment? If you re-build from 4.1.39 tag, would it be linked to GLIBC_2.12?

ursaj · 2019-11-08T21:39:24Z

% strings /lib64/libc.so.6 | grep "^GLIBC_" | sort | uniq
GLIBC_2.10
GLIBC_2.11
GLIBC_2.2.5
GLIBC_2.2.6
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.5
GLIBC_2.6
GLIBC_2.7
GLIBC_2.8
GLIBC_2.9
GLIBC_PRIVATE

normanmaurer · 2019-11-22T07:52:18Z

@ursaj Let me try to build 4.1.39.Final on the docker image. That said I am pretty sure it is linked the same way.

normanmaurer · 2019-11-22T13:30:26Z

Alright I know what the problem is and I think I have a fix :) Stay tuned.

@weissi

Motivation: 394a1b3 introduced a hard dependency on GLIBC 2.12 which was not the case before. This had the effect of not be able to use the native epoll transports on platforms which ship with earlier versions of GLIBC. To make things a backward compatible as possible we should not introduce such changes in a bugfix release. Special thanks to @weissi with all the help to fix this. Modifications: - Use syscalls directly to remove dependency on GLIBC 2.12 - Make code consistent that needs newer GLIBC versions - Adjust scattering read test to only run if recvmmsg syscall is supported - Cleanup pom.xml as some stuff is not needed anymore after using syscalls. Result: Fixes #9758.

normanmaurer · 2019-11-22T13:39:16Z

@ursaj PTAL #9797

@weissi

Motivation: 394a1b3 introduced a hard dependency on GLIBC 2.12 which was not the case before. This had the effect of not be able to use the native epoll transports on platforms which ship with earlier versions of GLIBC. To make things a backward compatible as possible we should not introduce such changes in a bugfix release. Special thanks to @weissi with all the help to fix this. Modifications: - Use syscalls directly to remove dependency on GLIBC 2.12 - Make code consistent that needs newer GLIBC versions - Adjust scattering read test to only run if recvmmsg syscall is supported - Cleanup pom.xml as some stuff is not needed anymore after using syscalls. Result: Fixes #9758.

@weissi

Motivation: 394a1b3 introduced a hard dependency on GLIBC 2.12 which was not the case before. This had the effect of not be able to use the native epoll transports on platforms which ship with earlier versions of GLIBC. To make things a backward compatible as possible we should not introduce such changes in a bugfix release. Special thanks to @weissi with all the help to fix this. Modifications: - Use syscalls directly to remove dependency on GLIBC 2.12 - Make code consistent that needs newer GLIBC versions - Adjust scattering read test to only run if recvmmsg syscall is supported - Cleanup pom.xml as some stuff is not needed anymore after using syscalls. Result: Fixes #9758.

ursaj · 2019-11-25T02:17:26Z

@normanmaurer thanks a lot!

ursaj changed the title ~~Epoll native library does NOT load after upgrade between 4.1.39 and 4.1.43~~ Epoll native library does NOT load after upgrade between 4.1.39 and 4.1.40 Nov 7, 2019

normanmaurer mentioned this issue Nov 22, 2019

Remove dependency on GLIBC 2.12 by using syscalls directly #9797

Merged

normanmaurer closed this as completed in #9797 Nov 23, 2019

normanmaurer added this to the 4.1.44.Final milestone Nov 23, 2019

dzmitryk mentioned this issue Feb 15, 2020

Bug in netty 4.1.43 prevents from using Linux native Epoll transport apache/pulsar#6330

Closed

sijie mentioned this issue Feb 15, 2020

ISSUE-6330: Bug in netty 4.1.43 prevents from using Linux native Epoll transport streamnative/pulsar-archived#707

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Epoll native library does NOT load after upgrade between 4.1.39 and 4.1.40 #9758

Epoll native library does NOT load after upgrade between 4.1.39 and 4.1.40 #9758

ursaj commented Nov 5, 2019 •

edited

Loading

normanmaurer commented Nov 5, 2019 via email

ursaj commented Nov 5, 2019

carl-mastrangelo commented Nov 6, 2019

ursaj commented Nov 6, 2019 •

edited

Loading

normanmaurer commented Nov 6, 2019

ursaj commented Nov 6, 2019

normanmaurer commented Nov 6, 2019 via email

ursaj commented Nov 6, 2019 •

edited

Loading

carl-mastrangelo commented Nov 7, 2019 •

edited

Loading

normanmaurer commented Nov 7, 2019

ursaj commented Nov 7, 2019

normanmaurer commented Nov 7, 2019

normanmaurer commented Nov 7, 2019

ursaj commented Nov 7, 2019

normanmaurer commented Nov 8, 2019

ursaj commented Nov 8, 2019 •

edited

Loading

ursaj commented Nov 8, 2019 •

edited

Loading

ursaj commented Nov 8, 2019

normanmaurer commented Nov 22, 2019

normanmaurer commented Nov 22, 2019

normanmaurer commented Nov 22, 2019

ursaj commented Nov 25, 2019

Epoll native library does NOT load after upgrade between 4.1.39 and 4.1.40 #9758

Epoll native library does NOT load after upgrade between 4.1.39 and 4.1.40 #9758

Comments

ursaj commented Nov 5, 2019 • edited Loading

Expected behavior

Actual behavior

Steps to reproduce

Netty version

JVM version (e.g. java -version)

OS version (e.g. uname -a)

normanmaurer commented Nov 5, 2019 via email

ursaj commented Nov 5, 2019

carl-mastrangelo commented Nov 6, 2019

ursaj commented Nov 6, 2019 • edited Loading

normanmaurer commented Nov 6, 2019

ursaj commented Nov 6, 2019

normanmaurer commented Nov 6, 2019 via email

ursaj commented Nov 6, 2019 • edited Loading

carl-mastrangelo commented Nov 7, 2019 • edited Loading

normanmaurer commented Nov 7, 2019

ursaj commented Nov 7, 2019

normanmaurer commented Nov 7, 2019

normanmaurer commented Nov 7, 2019

ursaj commented Nov 7, 2019

normanmaurer commented Nov 8, 2019

ursaj commented Nov 8, 2019 • edited Loading

ursaj commented Nov 8, 2019 • edited Loading

ursaj commented Nov 8, 2019

normanmaurer commented Nov 22, 2019

normanmaurer commented Nov 22, 2019

normanmaurer commented Nov 22, 2019

ursaj commented Nov 25, 2019

ursaj commented Nov 5, 2019 •

edited

Loading

JVM version (e.g. `java -version`)

OS version (e.g. `uname -a`)

ursaj commented Nov 6, 2019 •

edited

Loading

ursaj commented Nov 6, 2019 •

edited

Loading

carl-mastrangelo commented Nov 7, 2019 •

edited

Loading

ursaj commented Nov 8, 2019 •

edited

Loading

ursaj commented Nov 8, 2019 •

edited

Loading