*: Add progress notify request watch request #9869

jpbetz · 2018-06-20T00:32:37Z

Add RequestProgress function to Watcher interface to request a progress notify response in all watch channels.

That can be used to check if a watcher is "caught up" to a particular revision, which is useful when determining if a watch cache is stale by comparing the progress revision to the revision returned in a quorum read.

Note that this also changes how progress notify events are communicated. They are now broadcast to all watch channels.

Fixes #9855

Example etcdctl usage:

ETCDCTL_API=3 bin/etcdctl watch --interactive
watch a
progress
progress notify: 14
watch b
progress
progress notify: 14
progress notify: 14 # since progress notify is broadcast to both watch channels
# In a separate terminal: ETCDCTL_API=3 bin/etcdctl put c 2
progress
progress notify: 15
...

jpbetz · 2018-06-20T00:33:59Z

cc @wenjiaswe

gyuho · 2018-06-20T16:12:49Z

etcdctl/ctlv3/command/watch_command.go

+				continue
+			}
+			go printWatchCh(c, ch, execArgs)
+		case "progress":


Q. Does this work? Think watch streams are not shared in etcdctl? So, this progress request would be no-op?

Hm. I'm not familiar with that aspect of etcdctl. Here's a sample run I just did:

ETCDCTL_API=3 bin/etcdctl watch --interactive watch a progress progress notify: 14 watch b progress progress notify: 14

Is there a case you can think of that might exercise a problem with the separate watch streams?

@jpbetz Oh, actually you are right. This is for interactive watch command, so requests are sharing the stream. I was thinking of just regular watch command, and a separate progress command.

@gyuho I'll add more edge cases to the TestWatchRequestProgress clientv3/integration test that I added in this PR. I currently just does: (1) start a watch (2) put a k/v (3) verify the k/v is recieved over the watch (4) request progress (5) check that progress is received over the watch.

Corner cases might be:

No watches

More than one watch

Put a key not visible to the watch, ensure progress is incremented

If you think of any others, let me know.

I modified the watch stream publisher to send the progress notify responses on all watch channels, the result is etcdctl now does:

ETCDCTL_API=3 bin/etcdctl watch --interactive watch a progress progress notify: 14 watch b progress progress notify: 14 progress notify: 14

@jpbetz I tried those corner cases with this branch, and confirmed that it's covered. I can't think of any other, for now. Thanks!

gyuho · 2018-06-20T16:13:31Z

etcdserver/etcdserverpb/rpc.proto

@@ -367,6 +367,9 @@ message ResponseHeader {
  // member_id is the ID of the member which sent the response.
  uint64 member_id = 2;
  // revision is the key-value store revision when the request was applied.
+  // For watch progress responses, the header.revision indicates progress. All future events
+  // recieved in this stream are guarenteed to have a higher revision number than the


s/guarenteed/guaranteed/ :)

Fixed and re-ran genproto.

jpbetz · 2018-06-20T18:54:27Z

clientv3/watch.go

@@ -608,7 +662,28 @@ func (w *watchGrpcStream) dispatchEvent(pbresp *pb.WatchResponse) bool {
 		Canceled:        pbresp.Canceled,
 		cancelReason:    pbresp.CancelReason,
 	}
-	ws, ok := w.substreams[pbresp.WatchId]
+
+	if wr.IsProgressNotify() {


@gyuho Since progress notify is global, does it make sense to broadcast the watch response to all channel substreams? This will change the behavior of existing progress notifies as well, but makes it easier to ensure any watch caches being updated via a WatchChan are updated with the latest progress.

Yes, since Progress Notify request is sent with no specific watch ID, it makes sense to dispatch globally.

@jpbetz @gyuho

this changes the previous behavior of watcher. the notification is per watch creation not per stream.

codecov-io · 2018-06-22T18:47:37Z

Codecov Report

Merging #9869 into master will increase coverage by 0.07%.
The diff coverage is 56.79%.

@@            Coverage Diff             @@
##           master    #9869      +/-   ##
==========================================
+ Coverage   69.03%   69.11%   +0.07%     
==========================================
  Files         385      385              
  Lines       35703    35757      +54     
==========================================
+ Hits        24648    24713      +65     
+ Misses       9260     9238      -22     
- Partials     1795     1806      +11

Impacted Files	Coverage Δ
etcdctl/ctlv3/command/watch_command.go	`45% <0%> (-2.1%)`	⬇️
etcdserver/api/v3rpc/watch.go	`73.77% <0%> (-2.65%)`	⬇️
clientv3/watch.go	`92.56% <80.7%> (-1.61%)`	⬇️
pkg/transport/timeout_conn.go	`80% <0%> (-20%)`	⬇️
mvcc/watcher.go	`96.29% <0%> (-3.71%)`	⬇️
proxy/grpcproxy/watch.go	`89.44% <0%> (-3.11%)`	⬇️
lease/leasehttp/http.go	`56.61% <0%> (-2.95%)`	⬇️
etcdserver/api/v3election/election.go	`66.66% <0%> (-2.78%)`	⬇️
raft/progress.go	`94.17% <0%> (-1.95%)`	⬇️
... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acf75dd...fb75b83. Read the comment docs.

jpbetz · 2018-06-22T19:38:19Z

@gyuho @xiang90 This is ready for review.

gyuho

Just requesting some clarifications. With the code, the use case makes much more sense now.

LGTM.

I tried this branch, and works good.

gyuho · 2018-06-22T19:42:21Z

Documentation/dev-guide/interacting_v3.md

@@ -355,6 +355,23 @@ foo         # key
 bar_latest  # value of foo key after modification
 ```

+## Watch progress
+
+Applications may want to check the progress of a watch to determine how up-to-date the watch stream is. For example, if a watch is used to update a cache, it can be useful to know if the cache is stale compared to the revision from a qoroum read. 


s/qoroum/quorum/ :)

Can we also clarify this revision in watch response to progress notify request is revision from the local node that the watch stream is connected to? Since quorum read cannot be returned from the partitioned node, mention something like "stale compared to the revision fetched from a healthy cluster (e.g. revision in quorum read response)".

Ah, yes, I should be more explicit about that revision is returned. Fixing.

gyuho · 2018-06-22T19:43:44Z

clientv3/integration/watch_test.go

+						t.Fatalf("expected resp.IsProgressNotify() == true")
+					}
+					if resp.Header.Revision != 3 {
+						t.Fatalf("resp.Header.Revision expected 2, got %d", resp.Header.Revision)


s/resp.Header.Revision expected 2/resp.Header.Revision expected 3/?

Done. Thanks for the catch.

xiang90 · 2018-06-22T20:10:32Z

clientv3/watch.go

@@ -72,6 +72,9 @@ type Watcher interface {

 	// Close closes the watcher and cancels all watch requests.
 	Close() error
+
+	// RequestProgress requests a progress notify response be sent in all watch channels.
+	RequestProgress(ctx context.Context) error


Move this above Close method? Also rename this to just Progress?

Actually, after second thought RequestProgress is better.

Thanks @xiang90, I've moved above Close().

xiang90 · 2018-06-22T21:41:47Z

@jpbetz We still need to preserve the old behavior of progress notification.

gyuho · 2018-06-22T22:37:20Z

@xiang90 Good point.

Problem with current Protocol Buffer definition is we do not differentiate between regular progress notify and the one triggered via request progress. Also another problem is we do not expose watch response ID that can be used to pinpoint which stream to receive progress notify. So, it makes sense to broadcast the response for manually triggered progress request. But, it breaks the behavior of periodic progress notify response.

Maybe add an extra field to indicate that this response is requested by the request progress request?

jpbetz · 2018-06-27T21:25:05Z

@gyuho Looks like within dispatchEvent, we have access to the pb.WatchResponse struct, and can check the WatchId. This might be as simple as something like if pbresp.WatchId == 0 && wr.IsProgressNotify() { ? I'll add a test to verify we preserve the old behavior and try it out.

gyuho · 2018-06-27T21:57:17Z

@jpbetz

This might be as simple as something like if pbresp.WatchId == 0 && wr.IsProgressNotify() {

Sounds good. Adding extra check pbresp.WatchId == 0 before broadcasting would be simpler.

jpbetz · 2018-06-27T22:23:01Z

To preserve the previous behavior I've assigned -1 watch IDs to watch responses for RequestProgress. 0 did not work correctly since watch IDs are zero indexed (the 1st watch channel is assigned WatchId 0).

This makes it clear that the response is not associated with any particular watch id, which seems helpful in preventing from the watch response from being mis-interpreted. Since it's not obvious that it should be broadcast to all channels, I've added a comment explaining that.

gyuho · 2018-06-27T23:16:59Z

Can we fix the CI?

clientv3/watch.go.671: // watchIds are zero indexed, so RequestNotify watch responses are assigned watchId -1 to (spell: watchIds -> watch Ids?)

We just need to whitelist watchIds in .words file :)

jpbetz · 2018-06-27T23:25:12Z

Eek. Fixing! Looks like this should have failed during ./test when the fmt pass ran but for some reason it didn't. I'll have a look at my environment.

gyuho

lgtm thanks!

jpbetz added area/clientv3 WIP type/feature labels Jun 20, 2018

jpbetz added this to the etcd-v3.4 milestone Jun 20, 2018

jpbetz self-assigned this Jun 20, 2018

jpbetz requested review from gyuho and xiang90 June 20, 2018 00:33

gyuho reviewed Jun 20, 2018

View reviewed changes

jpbetz force-pushed the progress branch 2 times, most recently from 03a8503 to b82ede6 Compare June 20, 2018 17:07

gyuho mentioned this pull request Jun 20, 2018

Provide custom keepalive ping handler grpc/grpc-go#2166

Closed

jpbetz force-pushed the progress branch from b82ede6 to b96dc3c Compare June 20, 2018 18:52

jpbetz commented Jun 20, 2018

View reviewed changes

jpbetz force-pushed the progress branch 2 times, most recently from 0f2d760 to fbade37 Compare June 20, 2018 20:06

jpbetz changed the title ~~[WIP]*: Add progress notify request watch request~~ *: Add progress notify request watch request Jun 20, 2018

jpbetz force-pushed the progress branch 2 times, most recently from ea801b9 to fb75b83 Compare June 22, 2018 17:42

gyuho approved these changes Jun 22, 2018

View reviewed changes

gyuho removed the WIP label Jun 22, 2018

xiang90 reviewed Jun 22, 2018

View reviewed changes

jpbetz force-pushed the progress branch 3 times, most recently from ca942a3 to 1186f46 Compare June 22, 2018 20:56

gyuho approved these changes Jun 22, 2018

View reviewed changes

jpbetz force-pushed the progress branch from 1186f46 to a57127b Compare June 27, 2018 22:16

jpbetz force-pushed the progress branch from a57127b to 230aa6b Compare June 27, 2018 22:23

jpbetz force-pushed the progress branch from 230aa6b to 594baa9 Compare June 27, 2018 23:22

*: Add progress notify request watch request

4b51b6d

jpbetz force-pushed the progress branch from 594baa9 to 4b51b6d Compare June 27, 2018 23:46

gyuho approved these changes Jun 28, 2018

View reviewed changes

jpbetz merged commit 4df18a7 into etcd-io:master Jun 28, 2018

gyuho mentioned this pull request Jun 28, 2018

Add progress operation to watch API #9855

Closed

jpbetz mentioned this pull request Apr 30, 2019

leveraging etcd minor version features kubernetes/kubernetes#77272

Open

mikehuston mentioned this pull request Jun 10, 2021

Support for RequestProgress for Watch? etcd-io/jetcd#928

Closed

juliankung-db mentioned this pull request Jul 29, 2021

Add requestProgress function to Watch and Watcher interfaces etcd-io/jetcd#957

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: Add progress notify request watch request #9869

*: Add progress notify request watch request #9869

jpbetz commented Jun 20, 2018 •

edited

Loading

jpbetz commented Jun 20, 2018

gyuho Jun 20, 2018

jpbetz Jun 20, 2018

gyuho Jun 20, 2018

jpbetz Jun 20, 2018 •

edited

Loading

jpbetz Jun 20, 2018

gyuho Jun 22, 2018 •

edited

Loading

gyuho Jun 20, 2018

jpbetz Jun 20, 2018

jpbetz Jun 20, 2018 •

edited

Loading

gyuho Jun 20, 2018

xiang90 Jun 22, 2018

codecov-io commented Jun 22, 2018

jpbetz commented Jun 22, 2018

gyuho left a comment

gyuho Jun 22, 2018

gyuho Jun 22, 2018 •

edited

Loading

jpbetz Jun 22, 2018

gyuho Jun 22, 2018

jpbetz Jun 22, 2018

xiang90 Jun 22, 2018

xiang90 Jun 22, 2018

jpbetz Jun 22, 2018

xiang90 commented Jun 22, 2018

gyuho commented Jun 22, 2018 •

edited

Loading

jpbetz commented Jun 27, 2018

gyuho commented Jun 27, 2018

jpbetz commented Jun 27, 2018

gyuho commented Jun 27, 2018

jpbetz commented Jun 27, 2018

gyuho left a comment

*: Add progress notify request watch request #9869

*: Add progress notify request watch request #9869

Conversation

jpbetz commented Jun 20, 2018 • edited Loading

jpbetz commented Jun 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpbetz Jun 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gyuho Jun 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpbetz Jun 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jun 22, 2018

Codecov Report

jpbetz commented Jun 22, 2018

gyuho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gyuho Jun 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiang90 commented Jun 22, 2018

gyuho commented Jun 22, 2018 • edited Loading

jpbetz commented Jun 27, 2018

gyuho commented Jun 27, 2018

jpbetz commented Jun 27, 2018

gyuho commented Jun 27, 2018

jpbetz commented Jun 27, 2018

gyuho left a comment

Choose a reason for hiding this comment

jpbetz commented Jun 20, 2018 •

edited

Loading

jpbetz Jun 20, 2018 •

edited

Loading

gyuho Jun 22, 2018 •

edited

Loading

jpbetz Jun 20, 2018 •

edited

Loading

gyuho Jun 22, 2018 •

edited

Loading

gyuho commented Jun 22, 2018 •

edited

Loading