Unexpected behaviour with SDS / CDS APIs

[timperrett]

Hi gang,

I think there may be a few separate issues with the CDS API.

Overly frequent calling of SDS/CDS

The period timer in which Envoy is calling the CDS endpoint seems to be off by an order of magnitude. Given the following config:

"cluster_manager": {
    "sds": {
      "cluster": {
        "name": "sds",
        "connect_timeout_ms": 250,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://localhost:4000"
          }
        ]
      },
      "refresh_delay_ms": 30000
    },
    "cds": {
      "cluster": {
        "name": "cds",
        "connect_timeout_ms": 250,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://localhost:4000"
          }
        ]
      },
      "refresh_delay_ms": 30000
    },
    "clusters": [ ]
  }

This should be 30 seconds between calls, but in the log (I have a local testing harness now to test the CDS/SDS stuff and our custom backend providing the contract Envoy needs):

2017/02/06 03:49:48 GET	/v1/clusters/service1/172.17.0.2	988.023µs
2017/02/06 03:49:49 GET	/v1/registration/consul	812.934µs
2017/02/06 03:50:23 GET	/v1/registration/consul	1.580317ms
2017/02/06 03:50:46 GET	/v1/clusters/service1/172.17.0.2	1.218456ms
2017/02/06 03:51:00 GET	/v1/registration/consul	1.185353ms
2017/02/06 03:51:32 GET	/v1/clusters/service1/172.17.0.2	1.422009ms
2017/02/06 03:51:52 GET	/v1/registration/consul	1.386849ms

Envoy appears to be calling these APIs rather frequently - perhaps something is miss configured?

Appending not replacing clusters internally?

2. Perhaps i'm missing reading the output, but when looking locally at the Envoy (asking for /clusters, i'm not sure if the internal state of SDS/CDS is accurate as there are many duplicates. I've posted the output here: https://gist.github.com/timperrett/773c9c707b54077533ecf7144058cba1 - this to me at least looks as though the same cluster is in memory multiple times. My anticipation would be that there was only a single backend entry.

Thanks for your time. Happy to answer any questions on Gitter if this was not sufficient information.

[mattklein123]

@timperrett regarding (1), unless I'm reading it incorrectly, that looks correct to me. The APIs are being called anywhere from 30-60s apart (expected w/ jitter).

regarding (2), that seems incorrect. Let's talk about it in Gitter.

[timperrett]

For anyone reading this after the fact:

1. This issue can be explained by the fact that the jitter added to to the SDS and CDS calls is 0-basetime where (in my example) basetime is 30 seconds. The reason two of the calls appear close together is that the random jitter for that iteration was randomly decided to be near-zero.

2. Turns out that this part is a legitimate issue, and rears its head when running inside a docker container with bridged networking (note that --net=host does not have this issue), and that it is caused by Envoy not currently de-duplicating the hosts. See dns: de-dup returned entries #433 for more information on that.