On-demand DNS resolution

[howardjohn]

Title: On-demand DNS resolution

Description:
Currently, DNS clusters (LOGICAL and STRICT), constantly resolve DNS in the background.

We see fairly common reports of DNS servers being overloaded due to having a large explosion of these clusters across many Envoy workloads sharing the same DNS server (typically this is kube-dns).

Often, these workloads are either infrequently or never actually sending requests to these clusters. However, due to the operational complexity involved with having fine grained configuration, the cluster is still present on an excessive number of Envoy instances. Even with perfect configuration, we may have a service we call once per hour but need to resolve repeatedly.

We use the respect_ttl field, so sometimes this can be fixed with a larger TTL. However, this is not always configurable, and even when it is high can still lead to thundering herd problems.

It would be ideal if we could support on-demand DNS resolution. This could be implemented in a few ways:

1. (preferred) on first request to a cluster, do DNS resolution and set timer to re-resolve (based on TTL). Once timer is ready, re-resolve only if there was any requests during this time. This means that as long as 1/QPS > TTL, we are blocked for DNS only on the first requests. If we have infrequent requests, we only resolve once per request.

2. On the first request to a cluster, start DNS resolution like normal.

If we have full on-demand CDS this would look a lot like (2). However, I expect that will take a long time, and even when we have it this is still likely useful

[daixiang0]

Could you share more details about the case that never actually send requests, I do not hear it before. Do you mean health check?

This means that as long as 1/QPS > TTL, we are blocked for DNS only on the first requests. If we have infrequent requests, we only resolve once per request.

Is there a cache inconsistency issue when 1/QPS > TTL if I understand correctly?

[howardjohn]

I mean a user creates a config that results in a envoy cluster for googeapis.com being sent to a bunch of Envoy instances in the mesh. However, only 1/2 of their applications actually call googleapis.com, so we have a bunch of excess DNS requests.

Even in the 50% of instances we do depend on googleapis.com, we may call googleapis.com infrequently.

Is there a cache inconsistency issue when 1/QPS > TTL if I understand correctly?

No, this just means we would need to perform a synchronous DNS request

[mattklein123]

Could you use dynamic forward proxy for this? Dynamic forward proxy is effectively on demand DNS. :)

[howardjohn]

Interesting, hadn't thought of that before but does seem close to what we want. One concern would be it only works for HTTP (I think?), and we would likely need cluster-per-hostname for a lot of cases to have more control. The dynamic proxy could be used in some cases though, I think

[mattklein123]

L4 has a variant called the SNI forward proxy which works the same way. I would take a look at that also. Almost certainly we can make one of these work how you want it to.

[howardjohn]

We also support just raw TCP without TLS. An example in Istio API:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-database
spec:
  addresses:
  - 1.2.3.4
  ports:
  - number: 1234
  resolution: DNS
  endpoints:
  - address: some-sql-instance.aws.com

What this sets up in Envoy terms is a Listener on 1.2.3.4:1234 which forwards to a STRICT_DNS cluster resolving some-sql-instance.aws.com.

That may just be a minor edge case though. Let me play around with the HTTP/SNI forms and see if there are any gaps

[mattklein123]

If you really needed it we could add an option to the "SNI" filter to just hard code a target if none is supplied or not TLS, etc.

[ggreenway]

It may be generally useful to have a way to delay service discovery until a cluster is used (even if it's EDS, not DNS). I think it's similar to why we have a separate healthcheck interval for clusters that have never been used.

[lambdai]

Update: correct the linked issue
I used to explore delaying EDS request until the upstream connection is desired. A huge compleixity there, and the gain is marginal on top of ondemand CDS.

@ggreenway what's your opinion on my proposal #20873?

I'd like to build a notfication/callback on endpoint ready in the Cluster. The initial goal was to resolve the race condition at currrent ondemand cds, and my hunch is that the potential delayed discovery is another use case.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.