-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consul panic after repeated DNS recursion failure #284
Comments
Interesting. Looks like a DNS decoding issue. Is the upstream address you were resolving sensitive? It would be useful to build a test case around it. (Or if you know the type/format of records that should be resolving) |
I haven't looked at the network captures, but there may be some sensitive data in there. Rather than post it publicly, I can forward it to your email. It might be that there's something odd about the DNS records, but I feel Consul should log an error and not panic in that case. Do you think it would make any sense to invert the recursion relationship? i.e. recurse .consul names to the Consul server? I feel this approach loses some of the efficiency & power of the local Consul client but I'm curious as to what others think. |
Feel free to email that information to me if it contains anything sensitive. I agree, it should never panic. This is a bug in the DNS library we use. I will try to work upstream to get it fixed. You really don't loose anything inverting the relationship with Consul. If anything, you get some perks of using a caching DNS server in front of Consul, and just forwarding the .consul TLD |
From that codepath the only thing I can see is that off would be larger then lenmsg. Not likely, but might be possible. Would be nice to have some binary data for a test case. |
Working under the assumption
obviously against the latest git of go dns. |
@payoffstan Can you confirm with your test case that this patch works? I don't have a test case to reproduce this. |
Thanks @miekg and @armon. We don't have a test case per se but I can patch a dev server and see how it holds up. We finally encountered another panic recently but I haven't gotten the scrubbed packet capture from my ops guy yet. I'll email that once I have it so maybe we can construct a test case for the future. |
[ Quoting notifications@github.com in "Re: [consul] Consul panic after rep..." ]
Yes, some binary data would help here. I think this check should also be /Miek Miek Gieben |
I had the same problem with dd90d7942ba6f768acc803b3fba190d2c8a3559d panic: runtime error: index out of range goroutine 976 [running]: I've updated to the latest, I'll report back if the problem persists. |
@krolaw If you have a reproducible case, or a test input that would be great! |
Even a tcpdump would suffice.
|
Closing due to inactivity. Please ping us if this comes up again! |
This commit makes a couple of changes: * Uses master branch for the hashicorp/consul/sdk dependency. * Fix server-acl-init federation tests. Now that consul always enforces ACLs, we have to make sure that we have a valid token when calling the API. Prior to that we called secondary's agent/join API with a token that it had no way of verifying. * Standardize on retry.Run timeouts in lifecycle-sidecar tests. Sometimes tests were using too short of a timeout compared to other tests. I switched it to just use the default timer. * Fix lifecycle-sidecar tests that were leaving running consul agents behind. * Bumps Consul version in CI
Consul hits a runtime.panic after repeated i/o timeouts on dns recursion. This is on 0.3.1. Here's what the log looks like:
panic: runtime error: slice bounds out of range
goroutine 488917 [running]:
runtime.panic(0x9c21a0, 0xf7d40f)
/opt/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/miekg/dns.unpackStructValue(0x9d8360, 0xc2084f8440, 0x0, 0x196, 0xc208285000, 0x2e3, 0x1000, 0x1b, 0xc200000000, 0x0, ...)
/opt/gopath/src/github.com/miekg/dns/msg.go:1170 +0x5df4
github.com/miekg/dns.UnpackStruct(0xa1b5a0, 0xc2084f8440, 0xc208285000, 0x2e3, 0x1000, 0x11, 0x17, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/msg.go:1254 +0xc8
github.com/miekg/dns.UnpackRR(0xc208285000, 0x2e3, 0x1000, 0x17, 0x7f12c44e3770, 0xc2084f8440, 0x11, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/msg.go:1331 +0x214
github.com/miekg/dns.(_Msg).Unpack(0xc2080f03f0, 0xc208285000, 0x2e3, 0x1000, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/msg.go:1546 +0x504
github.com/miekg/dns.(_Conn).ReadMsg(0xc2080aee10, 0xc2080f02d0, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:176 +0x1b6
github.com/miekg/dns.(_Client).exchange(0xc2080aedc0, 0xc2080f02d0, 0xc2080ed000, 0x10, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:152 +0x388
github.com/miekg/dns.(_Client).Exchange(0xc2080aedc0, 0xc2080f02d0, 0xc2080ed000, 0x10, 0xa45c10, 0x6, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:91 +0x88
github.com/hashicorp/consul/command/agent.(_DNSServer).handleRecurse(0xc2081109b0, 0x7f12cb00a4a0, 0xc208ed1bc0, 0xc2080f02d0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/dns.go:571 +0x222
github.com/hashicorp/consul/command/agent._DNSServer.(github.com/hashicorp/consul/command/agent.handleRecurse)·fm(0x7f12cb00a4a0, 0xc208ed1bc0, 0xc2
080f02d0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/dns.go:79 +0x44
github.com/miekg/dns.HandlerFunc.ServeDNS(0xc2080ed010, 0x7f12cb00a4a0, 0xc208ed1bc0, 0xc2080f02d0)
/opt/gopath/src/github.com/miekg/dns/server.go:79 +0x40
github.com/miekg/dns.(_ServeMux).ServeDNS(0xc2080ecf70, 0x7f12cb00a4a0, 0xc208ed1bc0, 0xc2080f02d0)
/opt/gopath/src/github.com/miekg/dns/server.go:176 +0xc9
github.com/miekg/dns.(_Server).serve(0xc20813a9c0, 0x7f12cb008ca0, 0xc208b84f30, 0x7f12cb009320, 0xc2080ecf70, 0xc20822a000, 0x1c, 0xffff, 0xc2080b2018, 0xc208e9a620, ...)
/opt/gopath/src/github.com/miekg/dns/server.go:330 +0x510
created by github.com/miekg/dns.(*Server).serveUDP
/opt/gopath/src/github.com/miekg/dns/server.go:299 +0x1fb
The text was updated successfully, but these errors were encountered: