Issue with Akka (remoting) concurrent connections

[ecelgp]

Hello experts,

We see sporadic issue of member not joining the cluster after it restarts in OpenDaylight project (opendaylight.org), fyi we consume Akka 2.4.18 for our clustering solution:

After some debugging on multiple runs, it looks like the problem reproduces when restarting member and cluster leader establishes Akka TCP connection almost at the same time (1 connection is normally enough to setup the Akka channel). Please note the restarting member initiates the connection as part of the boot process while the cluster leader tries to connect the lost member every 5 secs, so this explains the sporadic behavior.

My question is has anyone reported/experienced the same issue with Akka concurrent connections? Do you have any advice to mitigate this?

Here is an example of the akka config we are using in opendaylight:

odl-cluster-data {
  akka {
    remote {
      artery {
        enabled = off
        canonical.hostname = "10.29.15.187"
        canonical.port = 2550
      }
      netty.tcp {
        hostname = "10.29.15.187"
        port = 2550
      }
    }

    cluster {
      seed-nodes = ["akka.tcp://opendaylight-cluster-data@10.29.15.187:2550",
				"akka.tcp://opendaylight-cluster-data@10.29.12.66:2550",
				"akka.tcp://opendaylight-cluster-data@10.29.15.35:2550"]

      roles = ["member-1"]

    }

    persistence {
      journal {
        leveldb {
        }
      }
    }
  }
}

More logs and discussion here:
https://jira.opendaylight.org/projects/CONTROLLER/issues/CONTROLLER-1751

Thanks in advance
Luis

[ktoso]

We're not quite sure what you mean here, could you provide a minimal reproducer?
What is "concurrent" about it?

[ktoso]

Could you also upgrade to 2.5 since 2.4.x is heading towards it's end of life (this month actually)

[ecelgp]

Concurrent means that 2 AKKA connections between same 2 nodes are active at the same time, for example:

tcp6 0 431 10.30.170.136:58204 10.30.170.104:2550 ESTABLISHED 12106/java
tcp6 0 0 10.30.170.136:42848 10.30.170.146:2550 ESTABLISHED 12106/java
tcp6 0 0 10.30.170.136:2550 10.30.170.146:40140 ESTABLISHED 12106/java

Member 1 (.136) above has simple unidirectional connection with member 2 (.104) but double bidirectional connection with member 3 (.146) after member 3 restarts.

This double or concurrent connection is always present when we see a restarting member failing to join the cluster so we wonder whether there is a relation between the double connection and the failure.

[ecelgp]

BTW we plan to upgrade to 2.5 in the next OpenDaylight release.

[ecelgp]

This issue has been fixed when we updated to akka_2.12-2.5.11, scala 2.12.5.

[raboof]

That's great to hear! Thanks for looping this back!