no-packet-error

[giganteous]

• Program: Recursor
• Issue type: Bug report

Short description

I'm seeing 'no-packet-error' incremented on most received packets. Old chats on irc (around april 2015) indicate that this is not an error, but a bug in the statistics.

Environment

• Operating system: debian linux (verified on 3.16.36/jessie and 4.9.40/stretch)
• Software version: 4.0.5 (pdns-recursor_4.0.5-1pdns.stretch_amd64.deb)
• Software source: repo.powerdns.com

Steps to reproduce

1. Run powerdns recursor, on any port
2. Ask it a single (valid) question; it should answer correctly
3. See that the no-packet-error is incremented

Expected behaviour

The no-packet-error should not increment on valid questions, or it should be renamed to something that doesn't make our eyebrows raise.

Actual behaviour

The no-packet-error is incrementing far too fast, almost as often as it gets questions

Other information

I dug some chats up from irc, dating from april 2015:

15:25        m4n: guys, am seeing too many no-packet-errors. how do I see what packets/requests they are?
15:25        m4n: no-packet-error 498270914
15:25        m4n: questions       624901828
15:25        m4n: that seem to be too much
15:25     Habbie: it means a socket reported 'i have a packet for you!' and then there was none
15:26     Habbie: are you running with pdns-distributes-queries?
15:26        m4n: strange. any pointers on how to troubleshoot that further?
15:26        m4n: yes
15:26     Habbie: hmm
15:27     Habbie: no, nothing immediately comes to mind
15:27        m4n: should I try turning that off to see if that helps?
15:27     Habbie: actually i was hoping you had it off so i could explain it from that ;)
15:27        m4n: :)
15:27     Habbie: if this is something you see consistently over time, sure, try turning that knob
15:28        m4n: yep consistent and the percentage compared to total questions is what worries me
15:28     Habbie: yes, agreed
15:30        m4n: isnt there a reload option?
15:30     Habbie: only for very specific settings
15:30     Habbie: not for this one, certainly :(
15:31        m4n: ok
15:32        m4n: also, SIGTERM to the recursor daemon will take it as "quit" or "quit-nicely"?
15:32  falkowich: Now I tried the same thing (on the test same server) with a bind9 (cache/forward). And the friggin iPad worked :)
15:33     Habbie: m4n, not sure
15:42        m4n: wow, after the restart with the setting turned off, it is even more
15:42        m4n: no-packet-error 810510
15:42        m4n: questions       768137
15:43     Habbie: i see
15:43     Habbie: do you have any external monitoring (pcap based, preferably) that can notice unanswered questions?
15:45        m4n: well, if questions are unanswered, the requests will go to the second nameserver in the resolv.conf, right?
15:45        m4n: i can check the metrics of that
15:45     Habbie: should be
15:47        m4n: the second server is getting very less queries
15:48        m4n: avg is .05 qps
15:48        m4n: am looking at "questions" metric
15:48     Habbie: ok
15:49     Habbie: that suggests there is no real problem
15:49     Habbie: just spurious wakeups of the recursor process
15:49     Habbie: there's some reason, of course, and it should be fixed
15:50        m4n: would be good to publish what are the metrics to watch out for
15:50        qvr: https://github.com/PowerDNS/pdns/blob/881dd6ba7112e556737c49e2c02c3089ef5a9530/pdns/pdns_recursor.cc#L1028
15:50        m4n: basically, to set alerts on
15:51        qvr: wouldnt that counter be incremented for every question?
15:51        qvr: because it wont exit the for() loop unless recmsg returns an error and that would usually be EAGAIN
15:55     Habbie: qvr, huh, yes, maybe
15:56        qvr: https://github.com/PowerDNS/pdns/commit/bf3b0cec366c090af000b066267b6f6bbb3a512a
15:57     Habbie: well spotted
15:58     Habbie: m4n, wanna file a ticket?
15:59        m4n: Habbie: i could have, but i was thinking it was something wrong with my setup
16:00     Habbie: well, qvr just found out it's a bug
16:00     Habbie: and the bug is we increase that number too much
16:00     Habbie: no harm
16:01        m4n: well, what I meant to say is, I didn't uncover the bug :)
16:01     Habbie: you're the one suffering from it :)
16:03        m4n: for me, that was an alarm to revisit my setup to see if anything was wrong

I'm suffering from it, and couldn't find any open or resolved issues.

Thanks for writing PowerDNS recursor 🥇