Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnsdist: Double-check we only increment the outstanding counter once #8113

Merged
merged 1 commit into from Jul 23, 2019

Conversation

@rgacogne
Copy link
Member

commented Jul 22, 2019

Short description

At least one user of 1.4.0-beta1 has reported that the outstanding counter is not always properly decremented, or is incremented twice. I have not been able to reproduce it so far but let's make sure we can only increment it once per TCP connection.

Checklist

I have:

  • read the CONTRIBUTING.md document
  • compiled this code
  • tested this code
  • included documentation (including possible behaviour changes)
  • documented the code
  • added or modified regression test(s)
  • added or modified unit test(s)

@rgacogne rgacogne added this to the dnsdist-1.4.0 milestone Jul 22, 2019

@rgacogne rgacogne merged commit 34f3230 into PowerDNS:master Jul 23, 2019

3 of 5 checks passed

LGTM analysis: JavaScript No code changes detected
Details
LGTM analysis: Python No code changes detected
Details
LGTM analysis: C/C++ No new or fixed alerts
Details
ci/circleci: test-dnsdist-regression Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@rgacogne rgacogne deleted the rgacogne:ddist-tcp-outstanding branch Jul 23, 2019

@johnhtodd

This comment has been minimized.

Copy link

commented Aug 9, 2019

Update on this: we still see the problem occurring even after this change.

Here is an example from one of our systems that is showing this behavior. Pool2 is not decrementing the outstanding counters correctly. The cause for the failure to decrement is unknown. We do not have reproduce-able trigger circumstances, but do see it happening often and is causes build-up that renders the outstanding policy ineffective.

#   Name                 Address                       State     Qps    Qlim Ord Wt    Queries   Drops Drate   Lat Outstanding Pools
0   pool1-1              [::1]:22000                      up     0.0     200 100 100     106824    1465   0.0  95.2           0 ECS
1   pool1-2              [::1]:22000                      up     0.0     200 100 100     105157    1386   0.0 198.1           0 ECS
2   pool2-1              [::1]:22001                      up     0.0     200 100 100    8631534  277646   0.0 791.4         178 DP
3   pool2-2              [::1]:22001                      up     0.0     200 100 100    9479246  284874   0.0 778.3         210 DP
4   pool2-3              [::1]:22001                      up     0.0     200 100 100    8438413  289073   0.0 873.3         206 DP
5   pool2-4              [::1]:22001                      up     0.0     200 100 100    5540786  276649   0.0 876.9         218 DP
6   pool3-1              [::1]:22002                      up     0.0     200 100 100   14124029  231884   0.0 221.0          14 DP
7   pool3-2              [::1]:22002                      up     0.0     200 100 100   13324160  229408   0.0 215.8          18 DP
8   pool3-3              [::1]:22002                      up     0.0     200 100 100   13429779  221635   0.0 366.4           5 DP
9   pool3-4              [::1]:22002                      up     0.0     200 100 100   13751142  226417   0.0 254.5           3 DP
All                                                              0.0                  86931070 2040437
@johnhtodd

This comment has been minimized.

Copy link

commented Aug 9, 2019

Will open new ticket on this.

@rgacogne

This comment has been minimized.

Copy link
Member Author

commented Aug 9, 2019

Thanks for the feedback, this PR was mostly a sanity check but I am actually more hopeful about #8154 fixing the issue. Then again, I couldn't reproduce it so it might not..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.