Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Only update peers if there was a significant change in nodes #234
This PR updates confd to only recompute BGP peers when node updates include meaninful changes compared to the cached values.
A little backstory for the motivation behind this PR.
We run some moderated size kubernetes clusters on our infrastructure and we were running Calico 3.0.5 without any problems. They run with Typha enable and using kubernetes as the data storage. Also, we disable node-to-node mesh and control the mesh ourselves by using the BGPPeers CRD with https://github.com/tsuru/remesher.
We decided to upgrade our development cluster to Calico 3.6.1. This cluster usually has around 63 nodes and we have around 2500 bgppeers entries:
After migrating to 3.6.1 we saw a huge increase in CPU usage on all calico-node pods. This high cpu usage was tracked to the
We went on to enable debug logging on the confd process and found out that it was constantly trying to regenerate the rendered templates even though there wasn't any changes to them. We also found out that regenerating the peers was triggered by node update events:
Node update events on kubernetes are tricky, they trigger roughly every 10 seconds for every node due to
This PR changes
@cezarsa Sounds like a good patch, thanks!
Couple of questions:
@fasaxc We're using the
Regarding the mesh configuration, one of the motivations for upgrading from 3.0 to 3.6 was being able to deploy a RR more easily using the in-cluster RR available since 3.3 I think. But we didn't want to do this in a single step and wanted to first migrate to 3.6 as we are today and after everything is working correctly update our network topology to include RRs.
referenced this pull request
Apr 5, 2019
Another update now, not a bugfix this time but a performance improvement over the last changeset.
First, here is an updated graph from our internal monitoring showing the performance improvements over time. The graphs on the PR description are no longer completely valid since they included a few bugs that were fixed since the original commit:
Now onto the reason behind this latest commit. When I started this PR I was under the incorrect assumption that every cache key must be associated with the latest revision, this caused confd to still try rendering all templates every time a node change happened (because the revision was incremented), even when we were skipping the bgp peers recompute step.
Upon understanding how this actually works I figured that we don't need to update the revision for a cache key unless it actually changes. This avoids re-rendering the templates needlessly.
caseydavenport left a comment
These changes look good to me - thanks for digging into this and providing a fix!
We'll target these for v3.8, but once they go through a bit of testing on master it might make sense to cherry-pick this to one or two earlier releases as well.