-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DNS resolution failed: dns server error: 3 name error #13011
Comments
@chobits Could you take a look? |
after downgrade the Kong to 3.4.2, it is very rare but it did happen still. |
It seems that kong attempted many queries of If kong reportes this error sporadically, it means your local dns accidentally replied NXDOMAIN for all the queries |
Yea, you can increase |
3.6.1 is pretty often and we increased dns_stale_ttl but no mitigate that. and downgrade 3.4.2 is much better. |
If you could easily reproduce this problem, it's not hard to debug. And you need to follow the queried chain provided by the error log to check if you could get the DNS result from your local DNS. We could tell you how to debug, while we could not debug for you if you cannot provide a reproduce step for us.
For these chain, kong tried all the DNS protocol type number: 33 - SRV |
And for this You could provide the output of |
Thank you for helping. we tested the dig on our cluster.
Got Answer:
|
I tested the domain:
Do you think it could be the cause? how can we fix that? |
I did not know what is happening on the coreDNS, the logs show as follow:
|
feel that it's related to k8s/dns configuration, but it's beyond my understanding 😢 From kong's output, it seems
|
If you are sure that you could use A type for
|
Thank you again! |
If you remove SRV from dns_order, kong will not try SRV. Kong tries to query all the The query sequence of these |
Is there an existing issue for this?
Kong version (
$ kong version
)3.6.1
Current Behavior
Kong 3.6.1 is running in a K8S cluster and there is a plugin call a backend endpoint: https://service-name/v1/session
which is used to verify the bear token.
Requests fail randomly with the following log and it is happening sporadically(5% possibility reject as 503 error):
2024/05/09 17:52:28 [error] 43#0: *112534 [lua] init.lua:371: execute(): DNS resolution failed: dns server error: 3 name error. Tried: ["(short)service-name:(na) - cache-hit/stale","service-name.default.svc.cluster.local:33 - cache-hit/stale/scheduled/dereferencing SRV","(short)6233306365613731.service-name.default.svc.cluster.local:(na) - cache-hit/stale","6233306365613731.service-name.default.svc.cluster.local:1 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.default.svc.cluster.local:1 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.svc.cluster.local:1 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.cluster.local:1 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.ec2.internal:1 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local:33 - cache-hit/stale/scheduled/recursion detected","6233306365613731.service-name.default.svc.cluster.local.default.svc.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.svc.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.ec2.internal:33 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local:5 - cache-hit/stale/scheduled/dns client error: 101 empty record received","6233306365613731.service-name.default.svc.cluster.local.default.svc.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.svc.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","6233306365613731.service-name.default.svc.cluster.local.ec2.internal:5 - cache-hit/stale/scheduled/dns server error: 3 name error"], client: 100.118.0.0, server: kong, request: "GET /v1/check HTTP/1.1", host: "xxxx.com", referrer: "http:/xxxx/", request_id: "6e2b4652d6ac6713802c4a4fe87b0b53"
Expected Behavior
Requests should go through Kong as expected
Steps To Reproduce
Anything else?
No response
The text was updated successfully, but these errors were encountered: