Add jitter in ProgressNotify

[cw9]

We use ProgressNotify in most of our etcd watchers and notice that the etcd servers have a spike of etcd_network_client_grpc_sent_bytes_total every exact 10 minutes. We think it's because we created a lot of watchers at the same time so every 10 minutes the servers need to send out progress update for every watcher. Is it true? If so, is it possible to add some jittering for the progress notify on the server side to avoid this kind of network spike?

I vaguely recall that the progress notification is supposed to be approximately every 10 minutes in the past, was it changed to exact every 10 minutes recently?

[xiang90]

@cw9

i agree we should add jitter to it as we add jitter to the internal lease revoke.

would you have time to try to add jitter. or @gyuho or I might give it a try soon.

[gyuho]

Yeah jittering was added to lease revoke, not to watch progress notify.
Should be straightforward to do the same for watch.

[cw9]

If it's straightforward to you guys, it's probably easier for you to make the change

[gyuho]

@cw9 Sure, we will address this in next release cycle 3.4.

[cw9]

Awesome, thanks!