Skip to content
Permalink
Browse files

capacity controller: listen to creation events on deployments (#111)

This is related to #77. When running shipper with resyncs disabled, we
noticed that some releases would hang, waiting for contender to achieve
capacity.

After investigating, we found that this was probably due to replication
lag on the application clusters. When operating over a freshly created
Deployment, we'd get a stream of the following errors:

	error syncing CapacityTarget "foo/bar" (will retry: true):
	expected exactly 1 deployment on cluster baz, namespace foo,
	with label "..." but 0 deployments exist

Usually, this resolves itself with a couple of retries, but when the
replication lag is severe enough, or different errors occur in sequence,
we'd drop the capacity target from the queue and never retry again.

By watching creation of Deployments, the CapacityTarget will re-join the
queue, and releases should no longer get stuck.
  • Loading branch information...
juliogreff authored and icanhazbroccoli committed Jul 8, 2019
1 parent 1ef3d3c commit 83d670eab46fbab36f1edf0e540c705170a06be7
Showing with 3 additions and 0 deletions.
  1. +3 −0 pkg/controller/capacity/deployment.go
@@ -104,6 +104,9 @@ func (c Controller) NewDeploymentResourceEventHandler(clusterName string) cache.
return ok
},
Handler: cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) {
c.enqueueDeployment(obj, clusterName)
},
UpdateFunc: func(old, new interface{}) {
c.enqueueDeployment(new, clusterName)
},

0 comments on commit 83d670e

Please sign in to comment.
You can’t perform that action at this time.