Connections setup from clients that use the additional native_trasport_shard_aware_port - are not all to that port

[slivne]

Please answer these questions before submitting your issue. Thanks!

What version of Scylla or Cassandra are you using?

• 2021.1.5 (scylla-cloud)

What version of Gocql are you using?

1.5.0

What version of Go are you using?

1.16.9

What did you do?

update scylla-bench to use gocql 1.5.0

connected with scylla-bench to a cluster

./scylla-bench -workload sequential -mode write -nodes XXX  -username XXX -password XXX

accessed one of the servers

support@ip-172-31-0-108:~$ netstat  -an | grep 9042
tcp        0      0 0.0.0.0:19042           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:9042            0.0.0.0:*               LISTEN     
tcp        0     15 ip1:19042      ip2:32810    ESTABLISHED
tcp        0     30 ip1:19042      ip2:32788    ESTABLISHED
tcp        0      0 ip1:19042      ip2:32821    ESTABLISHED
tcp        0      0 ip1:19042      ip2:32818    ESTABLISHED
tcp        0      0 ip1:19042      ip2:32820    ESTABLISHED
tcp        0     15 ip1:19042      ip2:32819    ESTABLISHED
tcp        0      0 ip1:9042       ip2:55714    ESTABLISHED
tcp        0     15 ip1:19042      ip2:32815    ESTABLISHED
tcp        0      0 ip1:19042      ip2:32823    ESTABLISHED
tcp        0      0 ip1:19042      ip2:32813    ESTABLISHED
tcp        0      0 ip1:9042       ip2:55712    ESTABLISHED
tcp        0     15 ip1:19042      ip2:32822    ESTABLISHED
tcp        0      0 ip1:19042      ip2:32811    ESTABLISHED
tcp        0      0 ip1:19042      ip2:32812    ESTABLISHED
tcp        0      0 ip1:19042      ip2:32814    ESTABLISHED

as can be seen there are:

• 13 connections to 19042
• 2 connections to 9042

I expected the control connection to be on 9042 and all the rest of the connections (actually sending traffic) to be on 19042

Maybe the first connection is not the control connection ?

I guess there is an explanation for this - yet am not sure what it is.

[piodul]

Although it's an old issue, I'm familiar with the implementation of the shard aware port support and I can answer what is going on here.

This is intended behavior. gocql maintains separate connection pools for each node. A pool first tries to establish a single connection to the non-shard-aware port. During handshake, the pool learns about the shard-aware port, and from now on directs new connections to the shard-aware port. The pool doesn't attempt to close the first connection and re-open it to the shard-aware port as it isn't really necessary. Until the first connection will be reopened due to other reasons, it will stay connected to non-shard-aware port.

In addition to this, the driver maintains a single control connection which isn't a part of any pool. I don't think the driver bothers to use the shard-aware port for it at all.

Assuming N nodes and S shards, you should have N + 1 non-shard-aware connections and (S - 1) * N shard-aware connections. I suppose here N = 1 and S = 14, so there should be 2 non-shard-aware connections and 13 shard-aware ones.