KQueue NativeIoException when profiling with JFR

[daschl]

Expected behavior

The following behavior is puzzling me to even look where to start debugging.

When using a combination Kqueue event loops and running a JFR recording on the running program, suddenly this error appears:

Apr 05, 2019 8:12:30 AM com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoop handleLoopException
WARNING: Unexpected exception in the selector loop.
com.couchbase.client.core.deps.io.netty.channel.unix.Errors$NativeIoException: kevent(..) failed: Invalid argument
	at com.couchbase.client.core.deps.io.netty.channel.unix.Errors.newIOException(Errors.java:122)
	at com.couchbase.client.core.deps.io.netty.channel.kqueue.Native.keventWait(Native.java:97)
	at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:147)
	at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:139)
	at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:208)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

The interesting piece is I can start my workload fine and it runs for minutes, but as soon as I start a JFR this error pops up in the logs and doesn't go away. Any idea what JFR does that could trip up the Kqueue implementation?

Actual behavior

No exception?

Netty version

4.1.34.Final

JVM version (e.g. java -version)

openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

OS version (e.g. uname -a)

(OSX Mojave 10.14.4)
Darwin 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64

[normanmaurer]

No idea... that said we just discovered the same lately but had no time yet to investigate .

[daschl]

Okay, let me know if I can be of help debugging/verifying this somehow @normanmaurer

[normanmaurer]

Will do... /cc @idelpivnitskiy @jayv

[normanmaurer]

@daschl just tried to replicate but no luck :/ Any special recording you did ? can you give me a step by step reproducer ?

[daschl]

Yes will try, the way I stumbled over this was by perf testing the new Couchbase SDK - will ping you if I can isolate it further

[jayv]

I have a reproducer @normanmaurer

[normanmaurer]

@jayv @daschl looking into it now

[normanmaurer]

@daschl #9023 PTAL

[daschl]

@normanmaurer well get this patch tested tomorrow. I'm curious, did you figure out how JFR triggers this? Or is this a side effect of something else slowing down with the JFR profiling

[normanmaurer]

it basically interrupts from time to time the native call which then produces the EINTR