Freno Raft consensus is not working with DNS Names in RaftBind, RaftNodes

[Ashok-Kalam]

Hi Team

I am working on deploying the Freno service in k8s using stateful sets with 3 replicas (freno-0, freno-1, freno-2).

I tried to use dns names in RaftBind, RaftNodes but raft consensus doesn’t seem to work. Following is the configuration I have used.

"ListenPort": 9777,
"RaftBind": "freno-1.freno-raft-service.n1stack.svc.cluster.local",
"RaftDataDir": "/var/lib/freno",
"DefaultRaftPort": 9888,
"RaftNodes": [
"freno-0.freno-raft-service. n1stack.svc.cluster.local",
"freno-1.freno-raft-service. n1stack.svc.cluster.local",
"freno-2.freno-raft-service. n1stack.svc.cluster.local"
],

Communication issue in Freno POD Logs:

2021/12/22 17:36:58 [WARN] raft: Clearing log suffix from 296904 to 296904 │
│ 2021/12/22 17:36:59 [WARN] raft: Rejecting vote request from 172.34.179.140:9888 since we have a leader: 172.16.213.139:9888 │
│ 2021/12/22 17:36:59 [WARN] raft: Rejecting vote request from 172.34.179.140:9888 since we have a leader: 172.16.213.139:9888 │
│ 2021/12/22 17:36:59 [DEBUG] raft-net: 172.34.154.61:9888 accepted connection from: 172.34.251.16:34644 │
│ 2021/12/22 17:36:59 [DEBUG] raft-net: 172.34.154.61:9888 accepted connection from: 172.34.251.16:34648 │
│ 2021/12/22 17:36:59 [INFO] raft: Duplicate RequestVote for same term: 332362 │
│ 2021/12/22 17:36:59 [WARN] raft: Duplicate RequestVote from candidate: 172.34.251.16:9888 │
│ 2021/12/22 17:36:59 [WARN] raft: Clearing log suffix from 296905 to 296905 │
│ 2021-12-22 17:37:00 DEBUG raft leader is 172.34.251.16:9888; state: Follower │
│ 2021/12/22 17:37:01 [DEBUG] raft-net: 172.34.154.61:9888 accepted connection from: 172.34.251.16:34856 │
│ 2021/12/22 17:37:01 [DEBUG] raft-net: 172.34.154.61:9888 accepted connection from: 172.34.251.16:34860 │
│ 2021/12/22 17:37:01 [INFO] raft: Duplicate RequestVote for same term: 332363 │
│ 2021/12/22 17:37:01 [WARN] raft: Duplicate RequestVote from candidate: 172.34.251.16:9888 │
│ 2021/12/22 17:37:01 [WARN] raft: Clearing log suffix from 296906 to 296906 │
│ 2021/12/22 17:37:02 [WARN] raft: Heartbeat timeout from "172.34.251.16:9888" reached, starting election │
│ 2021/12/22 17:37:02 [INFO] raft: Node at 172.34.154.61:9888 [Candidate] entering Candidate state │
│ 2021/12/22 17:37:02 [DEBUG] raft: Votes needed: 3 │
│ 2021/12/22 17:37:02 [INFO] raft: Duplicate RequestVote for same term: 332364 │
│ 2021/12/22 17:37:02 [WARN] raft: Duplicate RequestVote from candidate: 172.34.154.61:9888 │
│ 2021/12/22 17:37:02 [DEBUG] raft: Vote granted from 172.34.154.61:9888. Tally: 1 │
│ 2021/12/22 17:37:02 [WARN] raft: Remote peer freno-0.freno-raft-service.n1stack.svc.cluster.local:9888 does not have local node 172.16.215.220:9888 as a peer │
│ 2021/12/22 17:37:02 [DEBUG] raft: Vote granted from freno-0.freno-raft-service.n1stack.svc.cluster.local:9888. Tally: 2 │
│ 2021/12/22 17:37:02 [DEBUG] raft: Vote granted from freno-2.freno-raft-service.n1stack.svc.cluster.local:9888. Tally: 3 │
│ 2021/12/22 17:37:02 [INFO] raft: Election won. Tally: 3 │
│ 2021/12/22 17:37:02 [INFO] raft: Node at 172.34.154.61:9888 [Leader] entering Leader state

It is going in a loop and the leader is changing each and every minute.

If I try with the IPs approach it is working fine. Following is the working configuration.

"ListenPort": 9777,
"RaftBind": "172.34.179.140",
"RaftDataDir": "/var/lib/freno",
"DefaultRaftPort": 9888,
"RaftNodes": ["172.34.179.140", "172.34.251.16", "172.34.154.61"],

The raft consensus and freno service are working as expected with the above IP approach.

We would like to go with the DNS approach with stateful sets as Ips are ever-changing in k8s.
Could you please assist us in what we missing with the DNS approach because of which RAFT communication is not happening as expected?

Thanks in advance.

[Ashok-Kalam]

Hi Team

I am still awaiting inputs from you. Can you please check this and do the needful.

[dm-2]

Hi @Ashok-Kalam, we don't run freno in k8s currently so adding support for using DNS addresses for raft nodes isn't on our roadmap for freno. An option that may work for you is to run a custom sidecar service in k8s to handle this, as mentioned by @timvaillancourt here:

• How to change Raft node list #146 (comment)

[dm-2]

Closing as we can continue discussions in #146