Very odd domain resolution w/ green dns

[mike-code]

I've noticed that since I've switched to greendns, the application is unable to resolve some of the domains, but after ~30 minutes, it suddenly can (sic!). The eventlet is running in a flask application.

I've added a new A domain record (foo.bar.acme.dev) to the NS (cloudflare) and started the app in a docker container on GKE (GCP kubernetes). This domain however cannot be resolved if eventlet.monkey_patch() is applied.

Right after the container started, I logged into it and ran tcpdump to monitor the DNS queries (port 53) and I found something quite strange. The flash app w/ eventlet is the only application in this container which queries for this particular domain.

First, the default k8s /etc/resolv.conf

search my-app.svc.cluster.local svc.cluster.local cluster.local europe-central2-b.c.g-project.internal c.g-project.internal google.internal
nameserver 10.186.32.10
options ndots:5

As you can see there are 6 domains to search and a ndots option. Note that foo.bar.acme.dev is not an internal domain, so the proper DNS resolve query (FQDN) would be foo.bar.acme.dev..

Now what I've observed on the port 53 traffic is that the only DNS queries are

foo.bar.acme.dev.europe-central2-b.c.g-project.internal.
foo.bar.acme.dev.c.g-project.internal.
foo.bar.acme.dev.google.internal.

As you can see these are only 3 out of 6 search domains, but more importantly, there is no foo.bar.acme.dev. (the one I'm interested in).

After around 12-13 minutes of container being up and running, I can see that the tcpdump now logs queries for

foo.bar.acme.dev.europe-central2-b.c.g-project.internal.
foo.bar.acme.dev.c.g-project.internal.
foo.bar.acme.dev.google.internal.
foo.bar.acme.dev.my-app.svc.cluster.local.
foo.bar.acme.dev.svc.cluster.local.
foo.bar.acme.dev.cluster.local.

which are 6 out of 6 search domains, but still no foo.bar.acme.dev. FQDN.

Only after ~30 minutes of the container being up I can see that it tried all 7 (6 search domains + 1 without any domain (just .) and was finally able to resolve the domain.

If I try to resolve foo.bar.acme.dev. explicitly (FQDN) then they'll be no traffic on port 53 at all, but after 30 minutes it will start querying it.

Any ideas?