etcd3/store: watcher implementation #23694

hongchaodeng · 2016-03-31T19:40:12Z

This PR does:

Provide a watcher that uses etcd v3 API to watch changes via etcd and process them based on existing logic of storage.Interface.Watch(), WatchList().
By using the watcher, very trivial to implement Watch() and WatchList() in etcd3 storage.Interface implementation.

xiang90 · 2016-03-31T19:42:14Z

pkg/storage/etcd3/etcdwatcher/watcher.go

+	go func() {
+		for _, kv := range getResp.Kvs {
+			wc.sendEvent(&event{
+				key:       string(kv.Key),


can we use parseEvent for this too?

wojtek-t · 2016-03-31T19:44:19Z

Let me take a look into it later - I have a full calendar while being in US, but hopefully will take a first pass on it later today (or in the worst case, tomorrow).

k8s-bot · 2016-03-31T20:12:02Z

GCE e2e build/test passed for commit 3e9950c0071a32683dfa7ba9193573a54b7787bf.

timothysc · 2016-03-31T20:25:43Z

Will look tomorrow.

k8s-bot · 2016-03-31T20:38:46Z

GCE e2e build/test passed for commit c58a63e28d91ef2d896d8fc04d3345b258edf7d4.

xiang90 · 2016-03-31T20:45:11Z

/cc @kubernetes/huawei

k8s-bot · 2016-04-01T05:53:08Z

GCE e2e build/test passed for commit 729bd2fc34770b023e9a8a674a4167e24215f53a.

hongchaodeng · 2016-04-01T17:06:27Z

Test failed because of data race. See etcd-io/etcd#4876

I will have to bump up godep again.

k8s-bot · 2016-04-01T17:28:26Z

GCE e2e build/test failed for commit 04882ca2e576c0c19f9d39d14752751a453b1bfe.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

k8s-bot · 2016-04-02T05:35:55Z

GCE e2e build/test passed for commit dadab986b46aa516c69a1257e5dc4927c60e3af5.

k8s-bot · 2016-04-02T05:38:42Z

GCE e2e build/test passed for commit fafda38e3c0e6b0978efa283deae394dc6fad9f4.

k8s-bot · 2016-04-02T07:41:20Z

GCE e2e build/test failed for commit 61fe10bcfe0fd89c9dcae93ddfa0c41568d2b44e.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

k8s-bot · 2016-04-02T21:14:51Z

GCE e2e build/test passed for commit acf2db98f008c83af71714b343525f97b5dadda0.

k8s-bot · 2016-04-03T04:00:02Z

GCE e2e build/test passed for commit c421d9632816dbbc999a80fc9c80662d9495736c.

timothysc · 2016-04-04T16:27:43Z

pkg/storage/etcd3/etcdwatcher/event.go

+		key:       string(etcdEvent.Kv.Key),
+		value:     etcdEvent.Kv.Value,
+		rev:       etcdEvent.Kv.ModRevision,
+		isDeleted: etcdEvent.Type == storagepb.DELETE || etcdEvent.Type == storagepb.EXPIRE,


I would almost think this abstraction could be in the client library.. same with the one below.

Can you clarify?

Ah, I see what you mean.

I'm considering adding it to etcd library too. I will try to coordinate with the etcd team on this.

k8s-bot · 2016-04-17T06:07:37Z

GCE e2e build/test passed for commit a69c15e0cf5fd7565a6648ce57ff07f09e897436.

Automatic merge from submit-queue Bump up etcd dependency to fix data race ref: #23694 What this PR does - Bumping up the godep of etcd to fix data race in etcd watcher. Without this change, watcher PR builds will fail in race detection. - Small changes to fix builds after upgrade

hongchaodeng · 2016-04-17T21:43:08Z

Rebased.
We need to get this in before the TTL PR #23995 and then update there with watcher.
@wojtek-t @timothysc PTAL

k8s-bot · 2016-04-17T22:15:00Z

GCE e2e build/test passed for commit 23571315d351d7a0d71d400d9eff0a2cad47f910.

k8s-bot · 2016-04-17T23:00:59Z

GCE e2e build/test passed for commit 78788cecd95bfa4a27777cece9245f903ffff1b0.

k8s-bot · 2016-04-18T00:24:05Z

GCE e2e build/test passed for commit b98ce67c528de4ea6c14043ba3be9eb4f8fa2440.

k8s-bot · 2016-04-18T01:01:22Z

GCE e2e build/test passed for commit 2dbd2d2685e65cbff26e280353ddc49f46c29a15.

hongchaodeng · 2016-04-18T03:14:06Z

@k8s-bot test node e2e github issue: #24396

wojtek-t · 2016-04-18T10:23:49Z

pkg/storage/etcd3/watcher.go

+	var errResult *watch.Event
+	select {
+	case err := <-wc.errChan:
+		errResult = parseError(err)


why not just wc.resultChan <- *parseError(err)

and get rid of errResult completely?

wojtek-t · 2016-04-18T11:15:28Z

Just some very minor nit. Other than that LGTM

hongchaodeng · 2016-04-18T13:43:03Z

@wojtek-t Thanks for the review. Addressed and squashed.

wojtek-t · 2016-04-18T13:52:31Z

LGTM - thanks!

k8s-bot · 2016-04-18T14:14:28Z

GCE e2e build/test passed for commit e18b4e6.

k8s-github-robot · 2016-04-18T14:48:31Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2016-04-18T15:22:40Z

GCE e2e build/test passed for commit e18b4e6.

k8s-github-robot · 2016-04-18T15:23:12Z

Automatic merge from submit-queue

googlebot added the cla: yes label Mar 31, 2016

hongchaodeng mentioned this pull request Mar 31, 2016

Upgrade to etcd v3 plan #22448

Closed

xiang90 reviewed Mar 31, 2016
View reviewed changes

wojtek-t self-assigned this Mar 31, 2016

k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 31, 2016

hongchaodeng force-pushed the w branch from 3e9950c to c58a63e Compare March 31, 2016 20:07

k8s-github-robot added the needs-ok-to-merge label Mar 31, 2016

k8s-github-robot added the release-note-label-needed label Mar 31, 2016

hongchaodeng force-pushed the w branch 2 times, most recently from d48e827 to 04882ca Compare April 1, 2016 16:41

k8s-github-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 2, 2016

hongchaodeng mentioned this pull request Apr 4, 2016

Bumping up etcd dep to fix data race #23808

Closed

timothysc reviewed Apr 4, 2016
View reviewed changes

hongchaodeng changed the title ~~etcd v3 watcher implementation~~ etcd3/store: watcher implementation Apr 17, 2016

hongchaodeng force-pushed the w branch from a69c15e to 2357131 Compare April 17, 2016 21:41

hongchaodeng force-pushed the w branch from 2357131 to 78788ce Compare April 17, 2016 22:22

hongchaodeng force-pushed the w branch from 78788ce to b98ce67 Compare April 17, 2016 23:48

hongchaodeng force-pushed the w branch from b98ce67 to 2dbd2d2 Compare April 18, 2016 00:27

wojtek-t reviewed Apr 18, 2016
View reviewed changes

etcd3/store: watcher implementation

e18b4e6

hongchaodeng force-pushed the w branch from 2dbd2d2 to e18b4e6 Compare April 18, 2016 13:42

wojtek-t added lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-merge and removed needs-ok-to-merge labels Apr 18, 2016

k8s-github-robot merged commit 616af68 into kubernetes:master Apr 18, 2016

hongchaodeng deleted the w branch April 18, 2016 22:23

jianhuiz mentioned this pull request Apr 22, 2016

type assertion failed on clientset #24656

Closed

hongchaodeng mentioned this pull request Jul 22, 2016

etcd v3 as storage backend for APIServer kubernetes/enhancements#44

Closed

30 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcd3/store: watcher implementation #23694

etcd3/store: watcher implementation #23694

hongchaodeng commented Mar 31, 2016 •

edited

xiang90 Mar 31, 2016

wojtek-t commented Mar 31, 2016

k8s-bot commented Mar 31, 2016

timothysc commented Mar 31, 2016

k8s-bot commented Mar 31, 2016

xiang90 commented Mar 31, 2016

k8s-bot commented Apr 1, 2016

hongchaodeng commented Apr 1, 2016

k8s-bot commented Apr 1, 2016

k8s-bot commented Apr 2, 2016

k8s-bot commented Apr 2, 2016

k8s-bot commented Apr 2, 2016

k8s-bot commented Apr 2, 2016

k8s-bot commented Apr 3, 2016

timothysc Apr 4, 2016

hongchaodeng Apr 6, 2016

hongchaodeng Apr 6, 2016

k8s-bot commented Apr 17, 2016

hongchaodeng commented Apr 17, 2016

k8s-bot commented Apr 17, 2016

k8s-bot commented Apr 17, 2016

k8s-bot commented Apr 18, 2016

k8s-bot commented Apr 18, 2016

hongchaodeng commented Apr 18, 2016

wojtek-t Apr 18, 2016

wojtek-t commented Apr 18, 2016

hongchaodeng commented Apr 18, 2016

wojtek-t commented Apr 18, 2016

k8s-bot commented Apr 18, 2016

k8s-github-robot commented Apr 18, 2016

k8s-bot commented Apr 18, 2016

k8s-github-robot commented Apr 18, 2016

etcd3/store: watcher implementation #23694

etcd3/store: watcher implementation #23694

Conversation

hongchaodeng commented Mar 31, 2016 • edited

xiang90 Mar 31, 2016

Choose a reason for hiding this comment

wojtek-t commented Mar 31, 2016

k8s-bot commented Mar 31, 2016

timothysc commented Mar 31, 2016

k8s-bot commented Mar 31, 2016

xiang90 commented Mar 31, 2016

k8s-bot commented Apr 1, 2016

hongchaodeng commented Apr 1, 2016

k8s-bot commented Apr 1, 2016

k8s-bot commented Apr 2, 2016

k8s-bot commented Apr 2, 2016

k8s-bot commented Apr 2, 2016

k8s-bot commented Apr 2, 2016

k8s-bot commented Apr 3, 2016

timothysc Apr 4, 2016

Choose a reason for hiding this comment

hongchaodeng Apr 6, 2016

Choose a reason for hiding this comment

hongchaodeng Apr 6, 2016

Choose a reason for hiding this comment

k8s-bot commented Apr 17, 2016

hongchaodeng commented Apr 17, 2016

k8s-bot commented Apr 17, 2016

k8s-bot commented Apr 17, 2016

k8s-bot commented Apr 18, 2016

k8s-bot commented Apr 18, 2016

hongchaodeng commented Apr 18, 2016

wojtek-t Apr 18, 2016

Choose a reason for hiding this comment

wojtek-t commented Apr 18, 2016

hongchaodeng commented Apr 18, 2016

wojtek-t commented Apr 18, 2016

k8s-bot commented Apr 18, 2016

k8s-github-robot commented Apr 18, 2016

k8s-bot commented Apr 18, 2016

k8s-github-robot commented Apr 18, 2016

hongchaodeng commented Mar 31, 2016 •

edited