internal/{grpc,contour}: EDS always returns a value

[davecheney]

Fixes #1091
Updates #420
Updates #439

Problems with Contour not returning a value to Envoy when asked specifically have occurred several times in the past. When they have occurred, because I didn't understand the underlying issue, they were fixed in a piecemeal fashion. This PR addresses the root cause of the issue and hopefully we'll never have to talk about this again.

What caused the issue?

Envoy can ask Contour two different questions; the first is "please tell me the contents of the xDS table", the second is "please tell me the contents of these specific named xDS entries". The former is performed when a xDS Discovery Request supplies no resource_name hints. The latter uses resource_name hints to say "I want to know about these specific resources".

Naively one might think that the second case is a subset of the first, send me what you know about these specific resources is the union of the contents of the xDS cache and the names supplied. The first case is even simpler, it's the union of the cache with itself. However it's not so simple.

The first question, "tell me everything you have it the cache" is simple; everything in the cache is turned into grpc and sent over the wire to Envoy. The non intersecting set of things which are not in the cache are not sent to Envoy because that makes NO SENSE. However in the second case, for some resource types, Envoy will ask for a resource by name and will not be happy if the resource is not provided. This was the cause of #439, and #1091.

In the latter case a Service had been scaled to zero. The service was present in the api server so a valid ingress document pointed to it. Thus Contour created a Cluster record and sent that to Envoy, which in turn asked EDS for the members of the cluster we just told it existed. However, because the service was scaled to zero, the endpoint object was blank, there were no valid endpoints, so nothing was entered into the EDS cache on Contour's side. Here's where things get really upsetting.

xDS does not provide any way for a server to reject a request. Well, you can close the connection, but that's not going to help, Envoy will just reconnect and ask again, and all the while, it will be blocked on startup. So, when Envoy asks for an EDS record we must provide it, even if we don't have any information about it. This is actually a more general problem than EDS, but EDS is more susceptible to this problem because of the background described in #1091 (comment).

How does this PR fix the issue?

This PR does two things. First, it recognises that the two questions Envoy are different--the DiscoveryRequest with resource_name hint is not a subset of tell me everything in the cache--this this PR removes the Values(...string) method on grpc.Resource and replaces it with Contents() and Query([]string). The former clearly returns the contents of the cache, the latter takes a set of resources to query.

Secondly, on a per resource basis, the Query implementation will return blank results for records it does not hold in its cache when safe to do so. It turns out that for CDS, LDS, and SDS, there is no safe way to construct a blank record, so we cannot provide one to Envoy if it asks for one. For EDS and RDS, there is a way to construct a blank record given the query name provided, which we do, and this satisfies Envoy, unblocking the startup process.

In addition to tests at the internal/contour and internal/e2e level I have also tested it in my cluster repeatedly scaling a service to zero, restarting Envoy, and validating that prior to this change Envoy would block on startup when a Service with zero active pods was referenced via an Ingress record. Post this change, Envoy starts immediately.

[stevesloka]

/lgtm

Fantastic work @davecheney! 🎉

[davecheney]

Thanks for the prompt review.