Announce Bus Port Failing.

[rubenhak]

Trying to run dockerized redis cluster. Ability to map ports looks like to be available since v.4. The mapping of client port works, but no the bus port. Here is my setup.

Redis Config:

port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
cluster-announce-ip 10.1.1.11
cluster-announce-port 10050
cluster-announce-bus-port 10060

Also starting nodes with --cluster-announce-ip 10.1.1.11 --cluster-announce-port 10050 --cluster-announce-bus-port 10060 arguments.

Cluster setup using ```
ruby /var/local/redis/redis-trib.rb create --verbose --replicas 0 52.91.138.97:10050 52.91.138.97:10051 54.197.133.192:10050

Creating cluster
Connecting to node 52.91.138.97:10050: OK
Connecting to node 52.91.138.97:10051: OK
Connecting to node 54.197.133.192:10050: OK
Performing hash slots allocation on 3 nodes...
Using 3 masters:
52.91.138.97:10050
54.197.133.192:10050
52.91.138.97:10051
M: ec46eca2a1f6b2261689a4ef13f1c774910101d0 52.91.138.97:10050
slots:0-5460 (5461 slots) master
M: 3992d249f5d569c43a08be2c81d649de3e8877db 52.91.138.97:10051
slots:10923-16383 (5461 slots) master
M: 1a6c68cb4c94be9f249496eebfd93dd82e395997 54.197.133.192:10050
slots:5461-10922 (5462 slots) master
Can I set the above configuration? (type 'yes' to accept): ANSWERED YES AUTOMATICALLY!!!>>> Nodes configuration updated
Assign a different config epoch to each node
Sending CLUSTER MEET messages to join the cluster
wait_cluster_join...
Waiting for the cluster to join

In the node logs I see:

Accepted 52.91.138.97:57330
configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH
Connecting with Node 5fae7a49333e4a68084ce2eba33e31097531c92e at 52.91.138.97:20050

The only way the **20050** could come into picture was the **cluster-announce-port**  + 10000, even though the bus port was overridden both in config file as well as command line arguments to **10060**.

Tried thus using redis:4.0 as well as redis:5.0-rc docker images. same behavior.

I'd appreciate fix or some workaround for this. Thanks.

[C-ra-ZY]

it's the same issue i met. can someone help?

[ip-sf]

I am seeing the exact same thing in 6.2.6 using Nomad (1.2.6) to deploy docker image redis:6.2

This is a pretty major and long lived feature to be broken this way/

Create Cluster (6 containers on 3 hosts)

root@alloc5:/data# echo ${redis_cluster[@]}
10.0.0.19:26975 10.0.0.225:29581 10.0.0.19:22796 10.0.0.96:21486 10.0.0.225:23366 10.0.0.19:25096

root@alloc5:/data# redis-cli --cluster-replicas 1 --cluster-yes --cluster create ${redis_cluster[@]}
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.0.0.225:23366 to 10.0.0.19:26975
Adding replica 10.0.0.19:25096 to 10.0.0.225:29581
Adding replica 10.0.0.19:22796 to 10.0.0.96:21486
M: ed335d72495213bc81405e0578c316aa382bab9c 10.0.0.19:26975
   slots:[0-5460] (5461 slots) master
M: 1b84f8df1713acfd276578d6ac4d42037cf5c5dd 10.0.0.225:29581
   slots:[5461-10922] (5462 slots) master
S: b115986cb15f529607d8d1ea86ca80f2aacc89f6 10.0.0.19:22796
   replicates 05ecc2e844232010d37ed03570fa8d45abc8bcce
M: 05ecc2e844232010d37ed03570fa8d45abc8bcce 10.0.0.96:21486
   slots:[10923-16383] (5461 slots) master
S: fe0739d9a0c74495a03c38446875c9f3f3540b94 10.0.0.225:23366
   replicates ed335d72495213bc81405e0578c316aa382bab9c
S: 6304abf13d4797e822ad3d56d6becb300a2ffc0d 10.0.0.19:25096
   replicates 1b84f8df1713acfd276578d6ac4d42037cf5c5dd
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.....................^C

Notice specifically this node (first in the list): 10.0.0.19:26975 (Port: 26975)

Logs

10.0.0.19:26975

1:M 29 Mar 2022 21:48:52.807 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH

All other containers

1:M 29 Mar 2022 21:48:52.809 # configEpoch set to 5 via CLUSTER SET-CONFIG-EPOCH
1:M 29 Mar 2022 21:48:52.846 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused
1:M 29 Mar 2022 21:48:52.947 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused
1:M 29 Mar 2022 21:48:53.049 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused
1:M 29 Mar 2022 21:48:53.149 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused
1:M 29 Mar 2022 21:48:53.251 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused
1:M 29 Mar 2022 21:48:53.352 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused
1:M 29 Mar 2022 21:48:53.453 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused
1:M 29 Mar 2022 21:48:53.555 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused
1:M 29 Mar 2022 21:48:53.656 - Connection with Node 9b06e2e9cffdeaef89e614b1dfd66cbd26af44e1 at 10.0.0.19:36975 failed: Connection refused

Notice the port being used to attempt a connection is 36975, or 10000 + cluster-announce-port

redis.conf

The same redis.conf is used on all nodes.

Nomad Template

loglevel   verbose
appendonly yes
port       {{ env "NOMAD_PORT_db" }}

cluster-enabled           yes
cluster-config-file       nodes.conf
cluster-node-timeout      5000
cluster-announce-ip       {{ env "attr.unique.network.ip-address" }}
cluster-announce-port     {{ env "NOMAD_HOST_PORT_db" }}
cluster-announce-bus-port {{ env "NOMAD_HOST_PORT_cluster" }}

10.0.0.19:26975 as rendered

loglevel   verbose
appendonly yes
port       6379

cluster-enabled           yes
cluster-config-file       nodes.conf
cluster-node-timeout      5000
cluster-announce-ip       10.0.0.19
cluster-announce-port     26975
cluster-announce-bus-port 26874

The other Node's rendered config (Epoch 5)

loglevel   verbose
appendonly yes
port       6379

cluster-enabled           yes
cluster-config-file       nodes.conf
cluster-node-timeout      5000
cluster-announce-ip       10.0.0.225
cluster-announce-port     23366
cluster-announce-bus-port 27484

Workaround

While it's not desirable, I intend to just hardset the cluster port that is assigned to just be {port} + 10000 to match the default, since it looks like cluster-announce-bus-port is completely ignored.