Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnsdist packet cache is not working if all ECS enabled backend servers are down #6098

Closed
M4t7e opened this issue Dec 15, 2017 · 6 comments
Closed
Labels
Milestone

Comments

@M4t7e
Copy link

@M4t7e M4t7e commented Dec 15, 2017

  • Program: dnsdist
  • Issue type: Bug report

Short description

I wanted to try the setStaleCacheEntriesTTL parameter and found a problem with the packet cache in dnsdist for ECS enabled backend servers. Also for valid and non stale entries, the caching behaves not as expected and is not working if all backend servers are down.

Environment

  • Operating system: SUSE Linux Enterprise Server 12 SP2 (x86_64)
  • Software version: dnsdist 1.2.0

Steps to reproduce

  1. Configure dnsdist like in the following example configuration:
setECSSourcePrefixV4(32)
setECSSourcePrefixV6(128)

newServer({address="127.0.0.1", pool={"test"}, useClientSubnet=true})

pc_test = newPacketCache(100000, 86400, 0, 60, 60, true)
getPool("test"):setCache(pc_test)

setStaleCacheEntriesTTL(86400)
addAction(AllRule(), PoolAction("test"))
  1. Query dnsdist for a DNS record to fill the packet cache and shutdown the backend server.
  2. Query dnsdist again for the same record and this will lead to a cache miss.

Expected behaviour

dnsdist should respond from the packet cache if all backend servers (with "useClientSubnet=true" set) are down. Also the setStaleCacheEntriesTTL should work for that.

Actual behaviour

If all backend servers (with "useClientSubnet=true" set) are down, dnsdist can't respond directly from the packet cache. It leads to a high rate of cache misses. If "useClientSubnet" is set to "false", it is working.

Usecase

If all backend servers are down, dnsdist should respond from the packet cache as an emergency mechanism. --> Better to answer with stale entries than not to answer and the service behind is down.

Description

When using "useClientSubnet=true" for backend servers, the dnsdist packet cache is working. As far as I know by hashing the whole query including the ECS. But if all backend servers are down, dnsdist can't respond from the cache. I assume it's because the ECS is not honored for the query hashing anymore. I can see 100% cache misses in this scenario.

@rgacogne
Copy link
Member

@rgacogne rgacogne commented Dec 15, 2017

Are you sending the queries in 2/ and 3/ from the exact same host? I'm asking because when ECS is enabled, the packet cache will not reply with the same answer for different ECS values since it can't know whether the server used that information to generate the answer.

@M4t7e
Copy link
Author

@M4t7e M4t7e commented Dec 15, 2017

Yes, I'm sending the queries from the exact same host with the same source IP. Btw, if I already add the ECS in the query it is working. Just if dnsdist has to add it for the backend servers it is not working. I assume it's because the useClientSubnet parameter belongs to the "newServer" configuration and if there is no backend server available, the cache lookup is done without adding the ECS information.

@rgacogne
Copy link
Member

@rgacogne rgacogne commented Dec 15, 2017

Ah, you are right, we only add the ECS information if the backend needs it, so if there is no backend the cache lookup will be done without it. I'm not sure how we should handle that case, perhaps by adding an option to specify whether the ECS information should be added if we have no backend?

@M4t7e
Copy link
Author

@M4t7e M4t7e commented Dec 15, 2017

I'm not sure how we should handle that case, perhaps by adding an option to specify whether the ECS information should be added if we have no backend?

This could be a good solution. I think that option then belongs to the packet cache?

@M4t7e
Copy link
Author

@M4t7e M4t7e commented Apr 16, 2018

@rgacogne: I tested the new feature from #6400 in dnsdist 1.3 and it's working like a charm!
Many thanks for that 😃

@rgacogne
Copy link
Member

@rgacogne rgacogne commented Apr 16, 2018

Cool, many thanks for the feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.