Envoy does not respect DNS TTL when no max_connection_duration is configured

[style95]

Title: Envoy does not respect DNS TTL when no max_connection_duration is configured

Description:
My envoy acts as a proxy to a certain DNS endpoint.

• envoy ---(DNS)---> servers(dynamic IP change)

When the max_connection_duration is configured, it is working as expected but if no duration is configured, the envoy does not respect the DNS TTL. So even if I removed some IPs from the DNS records, removed servers keep receiving requests.

I am not sure if this is expected behavior. I expected DNS ttl is always respected with this change.

I am using this version: envoy version: e546bf5fc89b063bb911dc717c9beb26efa27a9f/1.25.4-dev/Clean/RELEASE/BoringSSL

Please let me know if any further information is needed.

Config:

  name: outbound|80||MY_SERVER
  outlierDetection:
    baseEjectionTime: 1s
    consecutive5xx: 4294967295
    enforcingConsecutive5xx: 100
    enforcingSuccessRate: 0
    interval: 1s
  respectDnsTtl: true
  type: STRICT_DNS
  typedExtensionProtocolOptions:
    envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
      '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
      commonHttpProtocolOptions:
        idleTimeout: 300s
        maxConnectionDuration: 300s  ###### this one is necessary
      explicitHttpConfig:
        httpProtocolOptions: {}

[style95]

ok, I noticed this only happens when the active health check is configured and I've just found this page.

https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/service_discovery#on-eventually-consistent-service-discovery

Host absent / health check OK:
Envoy will route to the target host. This is very important since the design assumes that the discovery service can fail at any time. If a host continues to pass health check even after becoming absent from the discovery data, Envoy will still route. Although it would be impossible to add new hosts in this scenario, existing hosts will continue to operate normally. When the discovery service is operating normally again the data will eventually re-converge.

So a DNS server initially responds with 4 IPs and if I removed one of the IPs from the DNS records, the envoy will keep sending requests to the removed IP as long as health checks succeed?

[style95]

I confirmed that the removed host is not removed from the endpoint list with active health checks.
But I am still curious: when I update the max_connection_duration setting, the endpoint is suddenly removed.
Could anyone share what's going on under the hood?

Addition:
It seems not only max_connection_duration, but also changing connection settings like below triggers the deletion of endpoints.
But still couldn't understand the correlation between them.

  trafficPolicy:
    connectionPool:
      http:
        idleTimeout: 300s
      tcp:
        connectTimeout: 11s
        tcpKeepalive:
          interval: 74s
          time: 600s

[style95]

@alyssawilk Could you take a look at this?
Since you made a DNS commit here, I mentioned you.
Please bear with me if you are not in charge of this module.