Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fragment Watch Response in server #9294

Closed
gyuho opened this issue Feb 7, 2018 · 0 comments
Closed

Fragment Watch Response in server #9294

gyuho opened this issue Feb 7, 2018 · 0 comments

Comments

@gyuho
Copy link
Contributor

gyuho commented Feb 7, 2018

From #8188.

I saw at a scaled run where where are several (~2M) KV pairs associated with the same lease then upon lease expiry grpc-proxy (there are several issues using grpc proxy more on that later) / etcd will send around ~2M DEL watchresponse notifications, which will close the channel (with canceled not being set) and the Err being set to : error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (140103271 vs. 4194304)"

This also blocks #7624 if implemented using a keyspace-wide watch. The watch messages will need a fragmentation flag (probably want semantics like limit and more from Range) to indicate the events for a single revision are split over multiple messages.

Thinking about this issue more, limit semantics with some max n events would be something different from what's needed for the fix. Probably Limit would just drop more than n events and set More, but never deliver those extra events.

The fix here would involve a fragment boolean flag when creating the watch that will enable splitting up the large revisions (indicated with fragment set on the watch response when fragmented). This will also need client-side work to piece the results together.

Initial work in progress #9291.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

1 participant