grpcproxy: efficiently handle millions of unique watches

[heyitsanthony]

The current watch proxy will scale for many clients watching one key / one key range by coalescing similar watch streams. However, there are cases where it may be a single client per key with millions of clients. The current grpcproxy would help a little by using a single grpcstream, but it will still open many substreams on an etcd server. This wastes server resources and makes disconnects expensive since the proxy must (serially) reregister all the watches.

Two approaches to scaling:

• Extend the coalescing code to merge similar nearby key intervals, either being exact about the intervals or by throwing away unneeded data. (probably too complicated to be useful)
• Open a full keyspace watch and stream the data into an mvcc backend. Watches would be handled with the same code that handles watches on the etcd server.

The backend watcher proxy won't replace the current stream coalescing proxy; the two should be able to work together or independently.

[mangoslicer]

@heyitsanthony

Does the current stream coalescing model work like the following?

            +-------------+
            |     mvcc    |
            +------+------+
                   ^ watch key A 
                   |
            +-------------+
            | etcd server |
            +------+------+
                   ^ watch key A (s-watcher)
                   |
           +-------+-----+
           | gRPC proxy  |  -------+
           |             |         |
           ++-----+------+         |watch key A (c-watcher)
watch key A ^     ^ watch key A    |
(c-watcher) |     | (c-watcher)    |
    +-------+-+  ++--------+  +----+----+
    |  client |  |  client |  |  client |
    |         |  |         |  |         |
    +---------+  +---------+  +---------+

Open a full keyspace watch and stream the data into an mvcc backend. Watches would be handled with the same code that handles watches on the etcd server.

Would what your proposing work like this, assuming that the clients want to watch millions of keys which happen to be mostly unique? If that's the case, are we essentially removing load from the etcd server by transmitting all events to the grpc proxy without discriminating depending on whether any client actually wanted to receiving updates on that specific key. Then allowing the grpc proxy to handle the discrimination part?

            +-------------+
            |     mvcc    |
            +------+------+
            |______^______| Watch all keys?
                   |
            +-------------+
            | etcd server |
            +------+------+
                   ^ watch key A-C (s-watcher)
                   |
           +-------+-----+
           | gRPC proxy  |  -------+
           |             |         |
           ++-----+------+         |watch key C (c-watcher)
watch key A ^     ^ watch key B    |
(c-watcher) |     | (c-watcher)    |
    +-------+-+  ++--------+  +----+----+
    |  client |  |  client |  |  client |
    |         |  |         |  |         |
    +---------+  +---------+  +---------+

[heyitsanthony]

Yes. The proxy would stream events from the server into its own mvcc backend to service watches. The diagram should have an mvcc hanging off the gRPC proxy, though.

[xiang90]

moving to 3.4

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.