Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Periodic waves of EOF every 60 seconds. #69

Open
stevej opened this issue Jun 28, 2017 · 3 comments
Open

Periodic waves of EOF every 60 seconds. #69

stevej opened this issue Jun 28, 2017 · 3 comments

Comments

@stevej
Copy link

stevej commented Jun 28, 2017

testing linkerd-tcp with the stock example.yml and slow_cooker at 20k qps is resulting in large waves of EOFs.

Get http://proxy-test-4d:7474: EOF                                                                                                                              [96/25466]
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
2017-06-28T18:53:27Z  19829/0/181 20000  99% 10s   0 [  2   4   5   18 ]   18      0
...
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: read tcp 10.240.0.4:47703->10.240.0.21:7474: read: connection reset by peer
2017-06-28T18:54:27Z  19819/0/181 20000  99% 10s   0 [  2   4   6   14 ]   14      0
...
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
2017-06-28T18:55:27Z  19817/0/182 20000  99% 10s   0 [  1   4   5  206 ]  206      0
...
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
2017-06-28T18:56:27Z  19828/0/172 20000  99% 10s   0 [  2   4   6    9 ]    9      0
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
2017-06-28T18:57:27Z  19839/0/161 20000  99% 10s   0 [  3   5   7    9 ]    9      0

Interestingly, they seem to come in groups of 170-200 at a time about every 60 seconds.
@stevej
Copy link
Author

stevej commented Jun 29, 2017

High concurrency even at low qps seems to cause this

stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -qps 1 -concurrency 1000 http://proxy-test-4d:7474
# sending 1000 GET req/s with concurrency=1000 to http://proxy-test-4d:7474 ...
#                      good/b/f t   goal%   min [p50 p95 p99  p999]  max bhash change
2017-06-29T20:23:24Z   9000/0/0 10000  90% 10s   0 [ 17 105 257  291 ]  291      0 +
2017-06-29T20:23:34Z  10000/0/0 10000 100% 10s   0 [  7  15  16   22 ]   22      0
2017-06-29T20:23:44Z  10051/0/0 10000 100% 10s   0 [ 18  26  26   32 ]   32      0
2017-06-29T20:23:54Z  10272/0/0 10000 100% 10s   1 [ 11  25  26   30 ]   30      0
2017-06-29T20:24:04Z   9772/0/0 10000  97% 10s   0 [ 14  25  26   29 ]   29      0
2017-06-29T20:24:14Z  10004/0/0 10000 100% 10s   1 [ 16  25  26   30 ]   30      0
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
...
2017-06-29T20:24:24Z   9729/0/561 10000  97% 10s   1 [ 12  43 209  220 ]  220      0

again at the 1 minute mark.

@wmorgan
Copy link
Member

wmorgan commented Jun 29, 2017

egg timer bug

@stevej
Copy link
Author

stevej commented Jul 6, 2017

I haven't dug into tcpdump yet but I'll note this doesn't happen when the backends and test client are all on localhost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wmorgan @stevej