observability: allow "event sink" services to update configuration in-band

[junr03]

In #10889 @htuch suggested allowing in-band config updates for the metrics service stats sink. This is a pretty powerful idea. This issue details some of the complexities of implementing this, and the tradeoffs involved.

1. The current service definition is not bi-di streaming. It is only streaming on the request side. So we can't support in-band config updates for the life of the stream. Although only allowing one response would make the protocol easier, as it would obviate difficulty 2 below. As pointed out by Harvey in this comment.
2. If we want to support continuous, in-band config changes that affect the shape of the data (counters as absolute values vs. deltas) we need a way for the message envoy sends to encode the "version" of config that was used to generate the data. This is the base case of what I was talking about above where the data sent prior to config might not be in the shape the service expects. In general, we have to have a way for the service to know if the data was generated with the version of the config (and hence the shape of data) the service expects. I believe this wasn't a problem with API definitions for health check & outlier detection event services #10407 and LRS, because there the in-band updates are just filters, not changing the shape of the data itself.
3. I'd like to have the logic of in-band config changes implemented in a base class that "event reporting" services can use, as I think that logic can be abstracted. I created an issue about this (event emission: implement generic event reporting gRPC client #10966)

[mattklein123]

In general, we have to have a way for the service to know if the data was generated with the version of the config (and hence the shape of data) the service expects

One thought exercise would be whether this actually matters. I'm guessing for most/all config changes probably not. Meaning, the configuration updates could likely be eventually consistent. If that is the case it makes the problem a lot simpler.

[htuch]

Yeah, either eventually consistent or in the common case, they only actually happen on the first message from the server.