Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnsdist: 1.5.1 response to "dnsdist -e 'grepq("")' | wc -l" always "4" after some time #11488

Closed
joanna159 opened this issue Mar 30, 2022 · 6 comments · Fixed by #11543
Closed

Comments

@joanna159
Copy link

  • Program: dnsdist
  • Issue type: Bug report

Short description

somewhere after 120946 requests the answer to
"dnsdist -e 'grepq("")' | wc -l"
is always "4"

Environment

Steps to reproduce

dnsdist -> unbound -> bind
I configured my testdomains an a bind.
wunder.de
zeit.de

  1. I changed these configuration defaults in dnsdist.conf
    setRingBuffersSize(800000)
    setConsoleOutputMaxMsgSize(20000000)

  2. I did a lot of dns requests.

  3. I asked regulary "dnsdist -e 'grepq("")' | wc -l"

  4. Everything was normal up to 67998 requests but after 101943 requests the answer was always "4"

  5. "dnsdist -e 'grepq("")' shows
    /DNS/log/dnsdist/dnsdist_err.log
    /DNS/log/dnsdist/dnsdist_warn.log
    /DNS/log/dnsdist/dnsdist_info.log
    Connection closed by server.

  6. dnsdist console shows: Got an exception in client connection from 127.0.0.1:38070: failed in writen2: Broken pipe

Expected behaviour

Since the result of topQueries(0)
is 147999
I would expect 2*147999 as result of "dnsdist -e 'grepq("")' | wc -l"

Actual behaviour

the result of "dnsdist -e 'grepq("")' | wc -l" is always "4"

Other information

After grepping for every result fails
"dnsdist -e 'grepq("wunder.de")' | wc -l"
"dnsdist -e 'grepq("wunder.de")'"
still worked.
Now after "topQueries (0)" = 164393 that is also failing.

This still works:
"dnsdist -e 'grepq("zeit.de")' | wc -l"
33362

topQueries(3)
1 wunder.de. 108162 65.7%
2 version.bind. 39706 24.1%
3 zeit.de. 16679 10.1%
4. Rest 0 0.0%


linux-kpqb:~ # dnsdist -e 'grepq("")' | wc -l
4
linux-kpqb:~ # dnsdist -e 'grepq("wunder.de.")' | wc -l
4
linux-kpqb:~ # dnsdist -e 'grepq("zeit.de.")' | wc -l
33362

linux-kpqb:~ # dnsdist -e 'grepq("wunder.de.")'
/DNS/log/dnsdist/dnsdist_err.log
/DNS/log/dnsdist/dnsdist_warn.log
/DNS/log/dnsdist/dnsdist_info.log
Connection closed by the server.

linux-kpqb:~ # dnsdist -e 'grepq("zeit.de.")' | less
/DNS/log/dnsdist/dnsdist_err.log
/DNS/log/dnsdist/dnsdist_warn.log
/DNS/log/dnsdist/dnsdist_info.log
Time Client Server ID Name Type Lat. TC RD AA Rcode
-18682.3 127.0.0.1:33007 44639 zeit.de. A RD Question
-18682.3 127.0.0.1:33007 10.0.2.15:42 44639 zeit.de. A 0.3 RD AA No Error. 1 answers
[...]

@rgacogne rgacogne added this to the dnsdist-1.8.0 milestone Mar 30, 2022
@joanna159 joanna159 changed the title dnsdist: 1.5.1 response to "dnsdist -e 'grepq("")' | wc -l" always "4" after changing dnsdist: 1.5.1 response to "dnsdist -e 'grepq("")' | wc -l" always "4" after some time Mar 30, 2022
@joanna159
Copy link
Author

Next fail.
linux-kpqb:~ # dnsdist -e 'grepq("zeit.de.")' | wc -l
83240
linux-kpqb:~ # dnsdist -e 'grepq("zeit.de.")' | wc -l
84452
linux-kpqb:~ # dnsdist -e 'grepq("zeit.de.")' | wc -l
86563
linux-kpqb:~ # dnsdist -e 'grepq("zeit.de.")' | wc -l
95695
linux-kpqb:~ # dnsdist -e 'grepq("zeit.de.")' | wc -l
4
linux-kpqb:~ #

topQueries(0)
1 Rest 222128 100%

topQueries(3)
[...]
2 zeit.de. 74260 33.4%

@rgacogne
Copy link
Member

rgacogne commented Apr 1, 2022

Would you be able to test if raising the maximum response size again (setConsoleOutputMaxMsgSize(30000000) for example) helps?

@joanna159
Copy link
Author

I did set setConsoleOutputMaxMsgSize(30000000)
Did not change: setRingBuffersSize(800000)

Now the error occurs later.


linux-kpqb:~ # dnsdist -e 'grepq("")' | wc -l
155352

topQueries(0)
1 Rest 77786 100.0%

topQueries(3)
1 wunder.de. 77786 100.0%


linux-kpqb:/DNS/named # dnsdist -e 'grepq("")' | wc -l
171698
linux-kpqb:/DNS/named # dnsdist -e 'grepq("")' | wc -l
4
linux-kpqb:/DNS/named # dnsdist -e 'grepq("wunder.de")' | wc -l
165312
linux-kpqb:/DNS/named # dnsdist -e 'grepq("zeit.de")' | wc -l
74892

topQueries(0)
1 Rest 120098 100.0%

topQueries(3)
1 wunder.de. 82654 68.8%
2 zeit.de. 37444 31.2%
3 Rest 0 0.0%

@rgacogne
Copy link
Member

rgacogne commented Apr 5, 2022

Thanks! So I think you are reaching the maximum size of a message indeed. If you have enough memory you can keep raising that limit (there is a hard limit at 2^32-1, so 4294967295) until it is high enough for the number of entries you want to inspect. A better option would be to implement some kind of paging in grep("") but it's complicated by the fact that the content of the buffer is always changing.
So I guess a better question would be, what are you trying to do? :) Perhaps we can implement the kind of logic you need directly into grepq so we do not need to return all that data.

@joanna159
Copy link
Author

In this case I wanted to test if I get what I configured.
And how much CPU and memory usage rises.

I still can not see how are these related to the result.
setConsoleOutputMaxMsgSize(30000000)
setRingBuffersSize(800000)

120098 Queries * 2 = 240195 (and still smaller than both configured values)
I expected the cache to give me 800000 results.

@rgacogne
Copy link
Member

rgacogne commented Apr 5, 2022

setConsoleOutputMaxMsgSize

This is how many bytes the output of a single command executed over the console can be.

setRingBuffersSize(800000)

This is the maximum number of entries (queries + responses) that will be kept in the ring buffers.

I expected the cache to give me 800000 results.

Did dnsdist process 400k queries or more? If so I'd expect 800k lines indeed, provided that setConsoleOutputMaxMsgSize is set to a large enough value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants