Redis/Sentinel Openshift Deployment under mulitple DCs to form a single cluster with an additional Sentinel running inside a Podman service.

[jhua04]

We have 3 Redis/Sentinels instances running in data centre A (DCA) in OpenShift and 3 Redis/Sentinel instances in data centre B (DCB) in OpenShift as well, and we use Skupper v2 to form this Redis/Sentinel instances as one single Redis cluster so the Sentinel nodes can monitor the Redis master and failover to any Redis instance in DCA or DCB in case if the master failed. In this case, we have

dca-redis-0
dca-redis-1
dca-redis-2
dcb-redis-0
dcb-redis-1
dcb-redis-2

all on Redis port 6379

And
dca-sentinel-0
dca-sentinel-1
dca-sentinel-2
dcb-sentinel-0
dcb-sentinel-1
dcb-sentinel-2

all on Sentinel port 26379

Once the connectors/Listeners with routing keys get created, the Redis cluster looks good and the manual failover master nodes between Redis instances are working as expected.

However we need another Sentinel outside of these two DCs so in case of one DC fail, the master still can be failover to another DC. In this case, we have another machine (just call it as DCC) which running a Sentinel instance inside the Podman for this purposes.

We now have the following issue:

1. DCC needs to talk to all the Redis/Sentinels in DCA and DCB, in this case we need 12 listeners. As listeners created need specific port number, we have to create each listener with a different port in this case
   localhost:6379 -> dca-redis-0:6379
   localhost:6380 -> dca-redis-1:6379
   localhost:6381 -> dca-redis-2:6379
   localhost:6382 -> dcb-redis-0:6379
   localhost:6383 -> dcb-redis-1:6379
   localhost:6384 -> dcb-redis-2:6379
   And
   localhost:26379 -> dca-sentinel-0:26379
   localhost:26380 -> dca-sentinel-1:26379
   localhost:26381 -> dca-sentinel-2:26379
   localhost:26382 -> dcb-sentinel-0:26379
   localhost:26383 -> dcb-sentinel-1:26379
   localhost:26384 -> dcb-sentinel-2:26379

However we have some difficulties to get the Sentinel in Podman working as expected in this case, The Sentinel running in Podman on DCC cannot resolve the hostname of the sentinel announced hostname on DCA and DCB, when we config the sentinel.conf on DCC

sentinel monitor mymaster localhost 6379 3
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

In this case, the sentinel on DCC Podman cannot really connect to Redis/Sentinel on DCA and DCB due to the sentinel.conf adding the announced host names and port numbers as dca-redis-{n}:6379 and dca-sentinel-{n}:26379.

The question is:

1. Is our DCC Skupper/Podman setup correctly in this case?
2. How to resolve sentinel announce issue
3. Is there a better way to do to meet our requirement instead of using DCC.

[fgiorgetti]

@jhua04 mapping these 12 hosts to 127.0.0.1 on /etc/hosts would possibly help you with name resolution, but not with the ports, as you're mapping a different port.

I am wondering if you could add IP aliases to your podman machine, so that each listener could bind port 6379/26379 on each specific IP address.

Example (not a persistent way of doing it, but just for evaluation):

ifconfig eth0:0 192.168.0.1
ifconfig eth0:1 192.168.0.2

Then you could define respective /etc/hosts entries, like:

192.168.0.1 dca-redis-0
192.168.0.2 dca-redis-1

If so you could create your listeners bound to the specific alias ips/hosts above, like:

skupper listener create dca-redis-0 6379 --host 192.168.0.1
skupper listener create dca-redis-1 6379 --host 192.168.0.2

[jhua04]

This looks a sound solution to get around both host/port issues. I did a test in homelab (not redis. but similar concepts). One k8s cluster and one vm running on podman. Two apps running on k8s, one called backend:8080 and one guestbook:80.

Once site linked, on k8s site:

$ skupper connector status
NAME            STATUS  ROUTING-KEY     SELECTOR                HOST    PORT    HAS MATCHING LISTENER   MESSAGE
backend         Ready   backend         app=backend                     8080    true                    OK
guestbook      Ready   guestbook       app=guestbook-ui                80      true                    OK

and on podman vm site:

sudo ifconfig ens18:0 192.168.0.1 netmask 255.255.255.0 up
sudo ifconfig ens18:1 192.168.0.2 netmask 255.255.255.0 up

And add the /etc/hosts

192.168.0.1 backend
192.168.0.2 guestbook

Then created the listeners

skupper listener create backend 8080  --host 192.168.0.1
skupper listener create guestbook 80 --routing-key guestbook --host 192.168.0.2 
skupper system reload

skupper listener status
NAME		STATUS	ROUTING-KEY	HOST		PORT	MATCHING-CONNECTOR	MESSAGE
backend		Ready	backend		192.168.0.1	8080	true			OK
guestbook	Ready	guestbook	192.168.0.2	80	true			OK

After all these done, running a pod inside podman with --net host command and do the curl in podman pod

curl -v http://backend:8080/api
curl -v http://guestbook/

Both of them working. In this case, the host name in podman has no issue been resolved. and also port should not be an issue as it match to individual different host name.

Looks to me it should be a valid solution so need to test on a more controlled with redis and redis sentinel

@fgiorgetti Thanks a lot for the suggestion.