Add support for WatchProgressRequest

[brandond]

• Adds support for watch progress requests.
  If all channels are synced, sends the current revision using channel -1. This allows the watch client to determine the datastore's current revision on-demand in between progress report intervals.

  This matches the implementation from upstream etcd:

  • https://github.com/etcd-io/etcd/blob/v3.5.11/server/mvcc/watchable_store.go#L485-L487
  • https://github.com/etcd-io/etcd/blob/v3.5.11/server/mvcc/watchable_store.go#L500-L504
    For:
  • Kine does not handle WatchRequest_ProgressRequest message #267

• Make progress report notification interval configurable.
  Required by kubernetes as per Watch-List feature does not work as expected when ResourceVersion is not set. kubernetes/kubernetes#122805 (comment)

• Optimize CurrentRevision checks by sharing current revision marker from poll loop
  Progress notifications query the current revision frequently; this can be optimized by exposing the poll loop's current revision once the loop has been started.

[brandond]

@qwtsc @onprem are you able to test this?

[qwtsc]

@qwtsc @onprem are you able to test this?

good! let me have a try!

[qwtsc]

@brandond Bad news, it does not work in my e2e test environment.

[brandond]

Are you able to run Kine in debug mode, and share the results?

[qwtsc]

My e2e test doesn't work with real etcd as backend storage either. I am so confused now. @onprem can you share the settings of apiserver?

[qwtsc]

FYI, the related issue has been reported in k&k.

[brandond]

I commented on the Kubernetes issue you linked, but I think the behavior you're expecting was incorrect and is no longer observed by v1.27+

[brandond]

Hmm. If I add info-level logging on progress requests in kine, I see that they are CONSTANTLY being spammed by the apiserver. I also note that things seem to take way longer to start up when the featuregate is enabled. This feels pretty broken at the moment, and I don't think it is etcd or kine - I think the feature does not work properly.

[brandond]

@qwtsc @onprem are you able to test with the commit I just pushed? I am still seeing some weirdness from the apiserver, but I believe the implementation is correct.

[qwtsc]

@qwtsc @onprem are you able to test with the commit I just pushed? I am still seeing some weirdness from the apiserver, but I believe the implementation is correct.

Ok, Let's see if it works or not.

[qwtsc]

@brandond unfortunately, it doesn't work, but same test case works for etcd with special flag experimental-watch-progress-notify-interval open. Can you find out what happened in etcd when running with this "experimental" flag?

[brandond]

experimental-watch-progress-notify-interval is not a Boolean flag; it requires a value and overrides the default 10 minute watch progress interval. What are you setting it to?

Is your test case just the previously discussed curl request, or do you have something else that you can share?

[brandond]

Oh, I see from the upstream k/k issue that it must be set to 5 seconds instead of the default 10 minutes, for the Kubernetes feature to work. I didn't see that documented anywhere; let me take a look at making that configurable in kine.

[brandond]

@qwtsc please try with the most recent commit, using the new CLI flag --watch-progress-notify-interval=5s

[qwtsc]

experimental-watch-progress-notify-interval is not a Boolean flag; it requires a value and overrides the default 10 minute watch progress interval. What are you setting it to?

Is your test case just the previously discussed curl request, or do you have something else that you can share?

Something like curl, but it's written in golang.

try with the most recent commit, using the new CLI flag --watch-progress-notify-interval=5s

let's try. BTW, you're so hard working, I've noticed progress in this PR almost every six hours. Do you ever take time to rest?:-)

[brandond]

Do you ever take time to rest? :-)

We have had a weeklong ice storm and my kids are out of school, so I don't have much else to do lol

[qwtsc]

Cheers, now It works smoothly. Thanks for your impressive contribution. @brandond