Description
What would you like to be added?
Currently progress notifications are only sent if all watches are synchronized within stream. This is because progress notification request is sent and received for the whole stream.
If there is any slow watcher within stream this can totally sarve progress notifications. With #20229 watchers opened on future revision will also starve progress notification. This means that any setup with more than 1 watcher per stream doesn't make sense.
Note this doesn't affect Kubernetes as it setups only 1 watcher per stream.
Proposal: Allow sending individual progress notification responses if we cannot send response for the whole stream.
To mitigate issue with one watcher preventing whole stream receiving response, I propose to etcd fallback to sending individual responses. Cases of handling individual progress notification within stream.
- watcher is unsynchronized: Still don't send progress notification as this watcher should get events during next synchronization.
- watcher is synchronized and rev >= startRev: Send individual progress notification.
- watcher is synchronized and rev < startRev: We might send a progress notification with rev=startRev, however this would not be beneficial to client as no progress was made.
This change can potentially hugely increase number of progress notification response. Before implementing this change we should benchmark to better understand performance impact. If there is 1`000 watchers within a stream, before we would send at max 1 response if there was no unsynchronized member. With this change we might send up to 999 responses.
Alternatives:
- Allow clients to request progress on single watch instead of stream.
Why is this needed?
Makes progress notification more predictable.