Migrate multi node testkit to Netty 4.

[He-Pin]

refs: #462

I verified it locally withcluster/MultiJvm/test

[pjfanning]

lgtm - pekko-mult-node-testkit is documented to be 'API may change'

[He-Pin]

A new trait is added for help.

[He-Pin]

Thanks @pjfanning for the review , waiting another LGTM before merge it.

[mdedetrich]

@He-Pin So I am getting failures when running sbt cluster/MultiJvm/test, I have posted the entire output here https://gist.github.com/mdedetrich/9a8a1f7d73f3840c6378c3f549c284a2

I ran the test with JDK 11 + export JAVA_8_HOME=/usr/lib/jvm/java-8-openjdk/ and also a ran a sbt +clean beforehand. I noticed that some exceptions are being thrown regarding connection refused, maybe thats the reason? (I am not that familiar with these tests).

[He-Pin]

@mdedetrich
https://gist.github.com/He-Pin/c7232c8635a69b16e02b0dfc90b20f01
https://gist.github.com/He-Pin/f1349ef25c80cb4ed822b3a3fab7a4bf

these are my outputs and I don't have the errors in your gist, not sure why.

I just rebased on the current main.

[mdedetrich]

Im sorry but I have to block the PR at least for now because at least on my system its causing a regression in the cluster/MultiJvm/test.

I did an additional run with the latest rebase and its still failing (see https://gist.github.com/mdedetrich/5f68a3e3faa9cf8ee220547618fb89e2) however if I check out current main then the tests pass fine (see https://gist.github.com/mdedetrich/73ceedc04f51fda4381e9a3af7ec7bb4 ).

I had a deeper look, and it appears that the core issue is that its unable to resolve a connection, i.e.

[info] [JVM-3-MultiDcMultiJvmNode3] [ERROR] [08/01/2023 09:16:40.161] [MultiDc-pekko.actor.internal-dispatcher-3] [pekko://MultiDc/system/TestConductorClient] Connection refused: localhost/127.0.0.1:4711
[info] [JVM-3-MultiDcMultiJvmNode3] org.apache.pekko.actor.ActorInitializationException: pekko://MultiDc/system/TestConductorClient: exception during creation, root cause message: [Connection refused]
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.actor.ActorInitializationException$.apply(Actor.scala:206)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.actor.ActorCell.create(ActorCell.scala:679)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.actor.ActorCell.invokeAll$1(ActorCell.scala:523)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.actor.ActorCell.systemInvoke(ActorCell.scala:545)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:305)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:240)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[info] [JVM-3-MultiDcMultiJvmNode3] Caused by: java.lang.reflect.InvocationTargetException
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.util.Reflect$.instantiate(Reflect.scala:82)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.actor.ArgsReflectConstructor.produce(IndirectActorProducer.scala:111)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.actor.Props.newActor(Props.scala:236)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.actor.ActorCell.newActor(ActorCell.scala:626)
[info] [JVM-3-MultiDcMultiJvmNode3]     at org.apache.pekko.actor.ActorCell.create(ActorCell.scala:653)
[info] [JVM-3-MultiDcMultiJvmNode3]     ... 10 more
[info] [JVM-3-MultiDcMultiJvmNode3] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:4711
[info] [JVM-3-MultiDcMultiJvmNode3] Caused by: java.net.ConnectException: Connection refused
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[info] [JVM-3-MultiDcMultiJvmNode3]     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[info] [JVM-3-MultiDcMultiJvmNode3]     at java.base/java.lang.Thread.run(Thread.java:829)

I think netty/netty#6865 is related and you got pinged in that.

[mdedetrich]

@pjfanning @raboof @jrudolph If you have time do you mind checking out this PR's branch and running sbt cluster/MultiJvm/test to see if you have the same issues? I ran my tests on Linux system and I have a suspicion that the tests don't run in general on Linux OpenJDK but do run fine on Windows (which is what @He-Pin runs locally iirc)

[pjfanning]

I'm seeing lots of test failures - similar to what @mdedetrich reported.

I think we need to go back to the drawing board and start by getting the multi-jvm tests up and running in CI and sorting out the logging. The tests are reporting lots of slf4j binding issues - so we need to ensure that logback or something like it is set up when running the JVMs used in these tests.

[mdedetrich]

I think we need to go back to the drawing board and start by getting the multi-jvm tests up and running in CI and sorting out the logging. The tests are reporting lots of slf4j binding issues - so we need to ensure that logback or something like it is set up when running the JVMs used in these tests.

iirc Making these tests run in our current CI is problematic because of high flakiness due to noisy neighbour problem/VM's being very weak for the cluster tests, there is a reason why the tests were never enabled in CI for Akka. At least with Akka, before Lightbend would make a release/merge a PR like this they would test on machines like we are. Although this is another topic, this is one of the problems that dedicated hardware was meant to solve.

We can try temporarily enabling the tests only on this PR to see that it won't break main (on the presumption that we fixed the underlying issue we should be able to get at least a single pass with enough runs)

[He-Pin]

Will this multiNodeTestOnly JoinInProgressMultiJvmSpec test pass on your box @mdedetrich ?
I am using Windows box, if this connection refuse raise up, then the issue should be the bind is not sync I think.

[mdedetrich]

Will this multiNodeTestOnly JoinInProgressMultiJvmSpec test pass on your box @mdedetrich ? I am using Windows box, if this connection refuse raise up, then the issue should be the bind is not sync I think.

I am getting issues that I think are related to #486 (comment), see https://gist.github.com/mdedetrich/e4f0cec36d405bd0ba21921849d8a07a

[pjfanning]

cd1c74a makes org.apache.pekko.cluster.SplitBrainQuarantine work on my laptop - it was failing before this.

[He-Pin]

@mdedetrich @pjfanning Wired, I didn't see this on my Windows, and just saw it when try to run testOnly

[mdedetrich]

@He-Pin So I just re-ran the tests against latest state of the branch and I got the following https://gist.github.com/mdedetrich/d91a5d9e1731a20806ce6e04e58aa9ca.

Running multiNodeTestOnly JoinInProgressMultiJvmSpec doesn't actually do anything (you get "No tests run for MultiNode")

[He-Pin]

@mdedetrich Maybe it's bind on a different ip(not loopback) and try to connect with localhost/127.0.0.1:4711.

I mean, the multinode.server-host config is related.

[mdedetrich]

@mdedetrich Maybe it's bind on a different ip(not loopback) and try to connect with localhost/127.0.0.1:4711.

I mean, the multinode.server-host config is related.

Yes this definitely seems related although I wonder why its only occurring when updating to Netty 4, I guess Netty 4 may have changed loopback/resolution defaults?

[mdedetrich]

@He-Pin Here are the results of https://gist.github.com/mdedetrich/5c0acdba4cfb4bd180395e273033b880 at commit 0f525d8

[mdedetrich]

@He-Pin Here are the results of https://gist.github.com/mdedetrich/1f483dc433b32be6c9199c55c7de0b02 at commit aec80d4

[mdedetrich]

@He-Pin Here are the results for https://gist.github.com/mdedetrich/e1cf848f28a03861df1be3dd318e8c64 at commit f826ddc

[He-Pin]

I can 100% reproduce it on linux box, still find the root cause.

still some issues

[He-Pin]

Update: I will try to allocate sometime this weekend for this.

[He-Pin]

Close now,will reopen after I tested on linux box.