Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Newly joined nodes don't receive all routes when using RR mode #676

Open
Feder1co5oave opened this issue Feb 21, 2019 · 0 comments
Open

Newly joined nodes don't receive all routes when using RR mode #676

Feder1co5oave opened this issue Feb 21, 2019 · 0 comments

Comments

@Feder1co5oave
Copy link
Contributor

@Feder1co5oave Feder1co5oave commented Feb 21, 2019

On an already existing kubernetes cluster with kube-router successfully running in route reflector mode (rr.server annotation on 3 master nodes, rr.client annotation on all the workers), my workflow for joining new worker nodes is like this (the cluster was created with kubeadm):

for each new node:

  • run kubeadm join on the node
  • kubectl annotate node kube-router.io/rr.client=23
  • restart kube-router on the new node to put them in RR mode

Turns out the nodes previously joined to the cluster will receive routes for all the new nodes, but the new nodes will only receive routes for the "old" nodes. Restarting kube-router on the route reflector nodes solved this issue.

My troubleshooting suggested that whenever a new node joins, the rr server kube-router daemons will peer with that node right away, even before the rr.client annotation gets added to it. Thus, the new peer is assumed to be in full mesh mode even if in fact it will be in RR mode and will only peer with RR servers. BGP route reflector allows for BGP daemons in the same AS to peer in either full-mesh or RR mode. Export policies are such that RR servers will reflect advertisements:

  • from RR peers to RR peers
  • from full-mesh peers to RR peers
  • from RR peers to full-mesh peers
  • but NOT from full-mesh peers to other full-mesh peers (they're supposed to peer with each other in full-mesh, duh)

So the new nodes, assumed to be forming full-mesh, don't get advertisements about other new nodes.
Restarting RR servers forces them to reload the node list and annotations, and correctly peer with the new nodes in RR mode.

To fix this, kube-router should watch for annotation changes on nodes, and update its internal information about which nodes are forming full-mesh, and which are joined to a RR cluster.
While waiting for a fix to be implemented, I suggest the workaround to restart RR servers be documented for newcomers!

Feder1co5oave added a commit to Feder1co5oave/kube-router that referenced this issue Feb 21, 2019
murali-reddy added a commit that referenced this issue Mar 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.