client-go workqueue example #44320

rmohr · 2017-04-11T09:27:27Z

Demonstrates how to compose a controller out of cache.Controller,
cache.Indexer and a workqueue.

k8s-reviewable · 2017-04-11T09:27:33Z

This change is

k8s-ci-robot · 2017-04-11T09:27:34Z

Hi @rmohr. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

rmohr · 2017-04-11T09:28:45Z

Follow up for kubernetes/client-go#65.

/cc @mbohlool @caesarxuchao

0xmichalis · 2017-04-11T09:40:16Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+	glog.Info("Starting Pod controller")
+
+	go c.informer.Run(stopCh)
+


You need to wait for your caches to sync before running any worker. See for example here

Since I only have one cache, I don't think so. The key is dirtied for every entry anyway, right?

But maybe it would make the example more complete. What do you think?

The number of caches does not matter. When a cluster restarts, controllers that don't wait for their caches to sync may end up doing all sorts of weird things.

Really, could you elaborate on that?

I think the informer has a well defined behaviour when you are using it's callback. So whenever something is added/updated/deleted in the underlying store, we dirty the key via the callbacks. This means that when we process the key in the controller, and fetch the item from the informer store, we get the object from this event or a newer version. Or do I miss something here?

But I totally agree, that when you have "secondary" caches, it is vital to sync.

You are right, I was thinking of a bug we had with a secondary cache. Yet, I think it still makes sense to have the check here for all caches.

0xmichalis · 2017-04-11T12:04:48Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+			// Tricky what to do in this situation. One thing we can do, is enqueueing it a few times to
+			// add some backoff delays on the invalid key. This way we avoid hotlooping
+			// on invalid keys.
+			if queue.NumRequeues(key) < 5 {


I can't see where the business logic of your controller exists. As a convention we use a separate function for it which surfaces an error at this level and then another function that is your error handling code which also facilitates requeues vs drops of objects. See #8 in https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md

Thx. Should be addressed now. @Kargakis could you have another look?

0xmichalis · 2017-04-11T12:55:33Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+	}
+
+	runtime.HandleError(err)
+	glog.Infof("Dropping deployment %q out of the queue: %v", key, err)


You are missing c.queue.Forget(key) here

0xmichalis · 2017-04-11T12:56:59Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+
+	// Wait for all involved caches to be synced, before processing items from the queue is started
+	if !cache.WaitForCacheSync(stopCh, c.informer.HasSynced) {
+		glog.Fatal("Timed out waiting for caches to sync")


instead of Fatal, use utilruntime.HandleError everywhere. You should also return if this path is accessed

0xmichalis · 2017-04-11T12:58:51Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+
+func (c *Controller) Run(threadiness int, stopCh chan struct{}) {
+	// Let the workers stop when we are done
+	defer c.queue.ShutDown()


Missing a defer utilruntime.HandleCrash()

0xmichalis · 2017-04-11T13:42:36Z

/lgtm
/release-note-none

caesarxuchao

A few nits. Thanks.

caesarxuchao · 2017-04-11T17:18:45Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+	// create the workqueue
+	queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter())
+
+	// Bind the workqueue to a cache with the help of an informer. This way we make sure than whenever the cache


s/than/that

caesarxuchao · 2017-04-11T17:20:18Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+	controller := NewController(queue, indexer, informer)
+
+	// We can now warm up the cache for initial synchronization
+	// Le's suppose that we knew about a pod mypod on our last run, so we add it to the cache


caesarxuchao · 2017-04-11T17:20:59Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+
+	// Bind the workqueue to a cache with the help of an informer. This way we make sure than whenever the cache
+	// is updated, the pod key is added to the workqueue. Note than when we finally process the item from the
+	// workqueue we might see a newer version of the Pod than the version which was responsible for triggering the update.


nit: could you wrap this line?

caesarxuchao · 2017-04-11T17:21:18Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+	flag.StringVar(&master, "master", "", "master url")
+	flag.Parse()
+
+	// Create the connection


nit: s/Create/creates

caesarxuchao · 2017-04-11T17:24:35Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+
+func (c *Controller) handleErr(err error, key interface{}) {
+	if err == nil {
+		c.queue.Forget(key)


Can you add a line of comment for Forget and later at AddRateLimited?

rmohr · 2017-04-12T08:28:00Z

@caesarxuchao comments are updated 👍

caesarxuchao · 2017-04-13T21:35:57Z

/lgtm
/approve

caesarxuchao · 2017-04-13T21:36:13Z

Thanks for the example, @rmohr !

caesarxuchao · 2017-04-14T00:15:36Z

need to run hack/make-rules/../../hack/verify-bazel.sh

caesarxuchao · 2017-04-19T21:22:38Z

@rmohr Could you run hack/update-bazel.sh and squash the commits?

rmohr · 2017-04-21T09:54:50Z

@caesarxuchao squashed and ran hack/update-bazel.sh

0xmichalis · 2017-04-21T10:05:27Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+	}
+	c.queue.Forget(key)
+	runtime.HandleError(err)
+	glog.Infof("Dropping deployment %q out of the queue: %v", key, err)


s/deployment/pod/

0xmichalis · 2017-04-21T10:05:46Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+	}
+
+	if c.queue.NumRequeues(key) < 5 {
+		glog.Infof("Error syncing controller %v: %v", key, err)


s/controller/pod/

0xmichalis · 2017-04-21T10:06:34Z

staging/src/k8s.io/client-go/examples/workqueue/main.go

+	return true
+}
+
+func (c *Controller) syncToStdout(key string) error {


Add a comment here that this function is responsible for handling the business logic of the controller

Demonstrates how to compose a controller out of cache.Controller, cache.Indexer and a workqueue.

0xmichalis · 2017-04-21T11:00:09Z

/lgtm

k8s-github-robot · 2017-04-21T11:00:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: caesarxuchao, kargakis, rmohr

Needs approval from an approver in each of these OWNERS Files:

~~staging/src/k8s.io/client-go/OWNERS~~ [caesarxuchao]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-04-21T13:58:25Z

Automatic merge from submit-queue

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 11, 2017

k8s-github-robot assigned derekwaynecarr and 0xmichalis Apr 11, 2017

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Apr 11, 2017

rmohr force-pushed the workqueue branch from 0cb02b7 to d6895c7 Compare April 11, 2017 09:32

0xmichalis reviewed Apr 11, 2017

View reviewed changes

rmohr force-pushed the workqueue branch from b6cb8e6 to f6e759f Compare April 11, 2017 12:52

0xmichalis reviewed Apr 11, 2017

View reviewed changes

rmohr force-pushed the workqueue branch from 4b89c38 to 976bbb0 Compare April 11, 2017 13:08

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed release-note-label-needed labels Apr 11, 2017

caesarxuchao reviewed Apr 11, 2017

View reviewed changes

caesarxuchao assigned caesarxuchao and unassigned derekwaynecarr Apr 11, 2017

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 12, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 13, 2017

caesarxuchao mentioned this pull request Apr 13, 2017

Add an Informer example in client-go #44446

Closed

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 13, 2017

caesarxuchao mentioned this pull request Apr 19, 2017

Improve the coverage of examples kubernetes/client-go#128

Closed

11 tasks

rmohr force-pushed the workqueue branch from 40d36c1 to dca6b16 Compare April 21, 2017 09:51

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 21, 2017

0xmichalis reviewed Apr 21, 2017

View reviewed changes

client-go workqueue example

b39b3ba

Demonstrates how to compose a controller out of cache.Controller, cache.Indexer and a workqueue.

rmohr force-pushed the workqueue branch from dca6b16 to b39b3ba Compare April 21, 2017 10:41

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 21, 2017

k8s-github-robot merged commit ef50807 into kubernetes:master Apr 21, 2017

		glog.Info("Starting Pod controller")

		go c.informer.Run(stopCh)

client-go workqueue example #44320

client-go workqueue example #44320

Conversation

rmohr commented Apr 11, 2017

k8s-reviewable commented Apr 11, 2017

k8s-ci-robot commented Apr 11, 2017

rmohr commented Apr 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xmichalis commented Apr 11, 2017

caesarxuchao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmohr commented Apr 12, 2017

caesarxuchao commented Apr 13, 2017

caesarxuchao commented Apr 13, 2017

caesarxuchao commented Apr 14, 2017

caesarxuchao commented Apr 19, 2017

rmohr commented Apr 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xmichalis commented Apr 21, 2017

k8s-github-robot commented Apr 21, 2017

k8s-github-robot commented Apr 21, 2017