calico/node fails to start due to config race

[caseydavenport]

$ kubectl logs calico-node-jgp8b -n kube-system -c calico-node -p
Checking datastore connection
Datastore connection verified
ERROR: Unable to set global default configuration: resource already exists: GlobalBGPConfig(name=LogLevel)
Terminating
Calico node failed to start

Expected Behavior

calico/node should be resilient to the resource already existing, and carry on starting.

Current Behavior

calico/node fails and restarts successfully.

Possible Solution

Steps to Reproduce (for bugs)

Start a lot of calico/node instances at once, see some of them restart.

Context

This was on GCE, 1k nodes starting at once. Looks like only a few hit this issue and started up just fine afterwards.

[bcreane]

See #1038 for very useful (and quick to run) instructions for reproducing this problem with just a three node cluster.

[caseydavenport]

I noticed another race on my latest large scale test run:

$ k logs calico-node-r143m -n kube-system -c calico-node -p
Checking datastore connection
Datastore connection verified
ERROR: Unable to set node resource configuration: update conflict: 'Node(name=gke-casey-2k-test-default-pool-e6a5c621-479x)'
Terminating
time="2017-09-01T17:22:15Z" level=error msg="Failed to apply object: too many retries" Key=Node(name=gke-casey-2k-test-default-pool-e6a5c621-479x)
Calico node failed to start

I think this is Calico fighting with something else in this case, so it's probably a separate issue.

I'll raise another.

EDIT: https://github.com/projectcalico/libcalico-go/issues/505