coredns 1.0.6

[miekg]

There some important fixes (esp. in proxy); kick of new release.

[johnbelamaric]

I think we need decisions on a couple things, and as to whether or not we hold the release for them. These are:

• reload/global plugins - see think about global plugins #1452 and plugin/reload: enable proper coredns instance shutdown #1507
• upstream behavior - see plugin/kubernetes: Add upstream @self and loop count #1484 and kubernetes: Upstream to self deployment#55 and plugin/kubernetes: Document Stub Domains configuration #1427

I would like to see these finalized for the release, and then we can use 1.0.6 in the k8s 1.10 release for the beta of CoreDNS as default. That means we need to act pretty quickly on this.

Reload
I think it's important to have this capability to simplify changes to the k8s config map - which means making a decision and changing it if necessary. So, options are:

1. Define the concept of global plugins and make it one of those.
2. Leave it as is, but error on syntax if it's defined more than once.
3. Make it a CLI flag.

Opinions? I am OK with any of these. It's nice to have the plugin structure but I did originally also thing about a CLI flag. But are we just kicking the can down the road in not dealing with global plugins?

Upstream Behavior
Here, I think we need something like in #1484, otherwise I think behavior will change for the CNAME lookups compared to kube-dns (I think that's true?). It's also a more general issue, because right now CNAME lookups only work if the zone of the CNAME is handled by the same plugin, or some external server.

[miekg]

upstream and the proxy fix are most critical,, the rest is less important IMO

[chrisohaver]

... behavior will change for the CNAME lookups compared to kube-dns (I think that's true?)

Thats my understanding. I will verify.

[chrisohaver]

Thats my understanding. I will verify.

I was mistaken about kube-dns behavior. kube-dns does not honor stub-domains when looking up CNAMEs.

I also found an open issue, kubernetes/dns#131 that describes the same behavior.

IMO, this is a bug, but since it's a pre-existing behavior, there is no urgency to fix it.

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] coredns 1.0.6..." ]

> Thats my understanding. I will verify. I was mistaken about kube-dns behavior. kube-dns does not honor stub-domains when looking up CNAMEs. I also found an open issue, kubernetes/dns#131 that describes the same behavior. IMO, this is a bug, but since it's a pre-existing behavior, there is no urgency to fix it.

John made some good points, that (finally) made me understand the issue. I think it is good to fix, but if the immediate time pressure is off we could do just 1.0.6 (I mean, releases are cheap) /Miek

…

-- Miek Gieben

[miekg]

also seeing all the bugs issues with reload and other plugins we should disable the reload plugin.

[johnbelamaric]

That's fine, but all those issues still apply with signal based reloads. But having the plugin encourages its use a lot more.

[miekg]

Think all things that should be in are in, and should be could to go for a 1.0.6?
I could easily kick one off today/tomorrow?

[chrisohaver]

I think we need to sort out #1532 first...

[johnbelamaric]

Yes, very soon. @fturib just filed one more issue he found, it looks like someone made a backwards-incompatible change in K8s API so we are going to work around it. #1532 - @chrisohaver should have it done today or tomorrow. It's a pain to test because the issue is only when running against a k8s built off of k8s master not in any released version.

[miekg]

yeah, saw it. Wish we could easily release and update the CoreDNS in k8s, separate from their release cadence

[caffeinejolt]

Not sure if this will be fixed in proxy or not - but for what its worth... I am running coredns to proxy requests arriving on an ipv6 address/interface to an older tinydns authoritative server that I have not bothered to apply this patch since it does not apply cleanly and I was 1) lazy and 2) wanted to play with coredns. I am running coredns in a chroot as non-root and allow it to bind to port 53 ala setcap cap_net_bind_service=+ep coredns Here is the config:

. {
bind [IPv6 address]
proxy . [IPv4 address]
log
}

With that background out of the way - here's the problem I am seeing... When I fire it up it works perfect - for about 5 minutes or so and then it gets nothing but SERVFAIL responses. The "fix" is to restart coredns - then it works fine again - for another 5 minutes or so and then back to all SERVFAIL

[johnbelamaric]

Can you add errors to the config and see if anything shows up in the log?

[caffeinejolt]

Sure - I now have this:

. {
bind [IPv6 address]
proxy . [IPv4 address]
errors
log
}

Will have something shortly. Not sure if it matters or not, but again - for a full background, coredns gets somewhere between 20-200 queries per second on this setup.

[caffeinejolt]

Results are in: unreachable backend: no upstream host

However, I can dig the upstream host (which runs on the same machine) while it is issuing that error. Restarting coredns fixes. Will just add a cron to kill coredns once a minute for now.

[johnbelamaric]

This is 1.0.5 or master? This may already be fixed in master, there are upstream health check problems in 1.0.5

[caffeinejolt]

1.0.5 - guess i will just update when 1.0.6 gets released and see what happens

[caffeinejolt]

is there any way to disable upstream health checks?

[chrisohaver]

max_fails 0 i think...

max_fails is the number of failures within fail_timeout that are needed before considering a backend to be down. If 0, the backend will never be marked as down. Default is 1.

[caffeinejolt]

Ok - had chance to play with this a bit more and figured out what is going on. Tinydns by default...

tinydns is quite possibly the simplest possible full-function authoritative nameserver. It serves authoritative data only; any query that cannot be answered from its disk database is simply not answered. The disk database has provisions for telling the tinydns that it has comprehensive knowledge of a specific domain; only if that is set (indicated by the presence of a "." record in the data file) does tinydns return NXDOMAIN, the official statement that the requested domain does not exist.

So without the "." in the data file, tinydns never replies to requests for which it has no data - which coredns of treats as a timeout. When I add the "." tinydns replies immediately with NXDOMAIN so coredns does not get a timeout.

My guess is that coredns marks the upstream as down and stops trying once it gets too many timeouts? However, most of the queries sent upstream do not timeout even when I do not have the "." in the data file. Therefore, I concur there are upstream health check problems in 1.0.5.

Now I guess the question is if this problem is present in master - I will download and build master and try it out to see and let you know.

[caffeinejolt]

Scratch that plan...

go build github.com/coredns/coredns
go/src/github.com/coredns/coredns/core/dnsserver/register.go:33:13: cannot use newContext (type func() caddy.Context) as type func(*caddy.Instance) caddy.Context in field value

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] coredns 1.0.6..." ]

Scratch that plan... go build github.com/coredns/coredns # github.com/coredns/coredns/core/dnsserver go/src/github.com/coredns/coredns/core/dnsserver/register.go:33:13: cannot use newContext (type func() caddy.Context) as type func(*caddy.Instance) caddy.Context in field value

See #1540 Use 'make' or manual check out Caddy v0.10.10. Also, as I've stated in other issues, I'm not happy with the health checking in proxy. Can you try the forward plugin? (still need to compile your self I think). Happy to give you a binary though

[caffeinejolt]

Thanks for the offer on the binary. I was able to get things working fine by setting max_fails to 0 - I also ended up leaving tinydns responding to unknown hosts as nxdomain since I think this is a better default in any case - doesn't leave resolvers hanging. Overall I really like coredns - a great tool!

[fturib]

@miekg : do you have an idea when the 1.0.6 will be released ? The code freeze for K8s is this Friday (2/26). We would need this new versions with fixes for integration in v1.10 as BETA.
Let me know. Thanks!

[miekg]

yeah, we can release. Want to get #1543 in as well, but that should be OK to do today and then kick off a release.

[miekg]

and released

[fturib]

Thanks !!!