Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: StatusNoContent closes HTTP/1.0 keep-alive connections #15647

Closed
Dave-White-Rakuten opened this issue May 11, 2016 · 9 comments

Comments

Projects
None yet
3 participants
@Dave-White-Rakuten
Copy link

commented May 11, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?

1.5

  1. What operating system and processor architecture are you using (go env)?

linux / osx

  1. What did you do?
    If possible, provide a recipe for reproducing the error.
    A complete runnable program is good.
    A link on play.golang.org is best.

Load test http server with requests using keep-alive, return 204 (no content) codes. For example:
ab -n 1000000 -c 100 "http://yourserver/andendpoint"

  1. What did you expect to see?

Low number of connections in use as they should be reused after each completed request..

  1. What did you see instead?

Ran out of connections quickly. You can watch the count like so:
sudo netstat -tnop | grep ":81" | wc -l

I have a "fix". Just set content-length to 0 in the header. The thing is that a 204 has no content so specifying this value seems implicit.

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 11, 2016

This seems like a bug in ab (which is notoriously buggy and non-compliant HTTP-wise), not in Go.

Please find a better HTTP load testing tool.

@bradfitz bradfitz closed this May 11, 2016

@Dave-White-Rakuten

This comment has been minimized.

Copy link
Author

commented May 11, 2016

I can’t also cause this to happen with a real browser. It’s just a lot more tedious.

Dave White
Senior Software Engineer
RAKUTEN marketing

6985 S Union Park Center, Suite 300 | Salt Lake City | UT | 84047
David.White@rakuten.com | rakutenmarketing.com

Rakuten Marketing

AFFILIATE NETWORK • ATTRIBUTION • DISPLAY • SEARCH
–––––––––––––––––––––––––––––––––––––––––––––––––––

!SAVE THE DATE!
SYMPOSIUM SAO PAULO SEPTEMBER 1
SYMPOSIUM SYDNEY SEPTEMBER 24
–––––––––––––––––––––––––––––––––––––––––––––––––––

Confidential Information (C) Copyright 2015 Rakuten Marketing, LLC. All rights reserved.
This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and CONFIDENTIAL.
If you are not the intended recipient, dissemination of this communication is prohibited. If you have received this communication in error,
please erase all copies of the message and its attachments and notify us immediately by replying to sender.

From: Brad Fitzpatrick
Reply-To: golang/go
Date: Wednesday, May 11, 2016 at 11:48 AM
To: golang/go
Cc: David White, Author
Subject: Re: [golang/go] StatusNoContent closing connections (#15647)

This seems like a bug in ab (which is notoriously buggy and non-compliant HTTP-wise), not in Go.

Please find a better HTTP load testing tool.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/15647#issuecomment-218535763

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 11, 2016

I just tested this by hand and don't see a problem.

I think you're counting the wrong thing if you're counting connections in TIME_WAIT.

@Dave-White-Rakuten

This comment has been minimized.

Copy link
Author

commented May 11, 2016

I was running out of connections because they were stuck in time_wait after being closed (which shouldn’t happen with keep-alive). The issue was noticed by our systems guys. A few months ago we had about a huge drop in performance on production servers after upgrading to a new version of our Go-based server. The connections are dropped between the server and haproxy; not by the browser (although haproxy could be eliminated to simplify the testing environment). Connection counts between haproxy and the server are usually very small and static on our other servers.

The breakage ended up being a design change to return significantly more 204 codes, which was never suspect. I just spent over a week tackling this problem. Setting content-length to 0 fixed the performance in test environments, but has not been pushed into production yet.

In my test environment, using ab for the load, within a few seconds, netstat would be reporting about 64k. Then we’d start seeing errors and performance drop. If I continually refresh from Chrome, I can raise that number as well, just not as high because how many times can you refresh one browser within 60 seconds?

After my “fix”, I was able to throw that ab test at out server, 6x in separate processes, and the netstat command would always report 600 just as I would expect.

Anyway, I just figured I’d let you guys know so that someone else doesn’t spend another week on the same issue.

Thanks
dw

Dave White
Senior Software Engineer
RAKUTEN marketing

6985 S Union Park Center, Suite 300 | Salt Lake City | UT | 84047
David.White@rakuten.com | rakutenmarketing.com

Rakuten Marketing

AFFILIATE NETWORK • ATTRIBUTION • DISPLAY • SEARCH
–––––––––––––––––––––––––––––––––––––––––––––––––––

!SAVE THE DATE!
SYMPOSIUM SAO PAULO SEPTEMBER 1
SYMPOSIUM SYDNEY SEPTEMBER 24
–––––––––––––––––––––––––––––––––––––––––––––––––––

Confidential Information (C) Copyright 2015 Rakuten Marketing, LLC. All rights reserved.
This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and CONFIDENTIAL.
If you are not the intended recipient, dissemination of this communication is prohibited. If you have received this communication in error,
please erase all copies of the message and its attachments and notify us immediately by replying to sender.

From: Brad Fitzpatrick
Reply-To: golang/go
Date: Wednesday, May 11, 2016 at 1:25 PM
To: golang/go
Cc: David White, Author
Subject: Re: [golang/go] StatusNoContent closing connections (#15647)

I just tested this by hand and don't see a problem.

I think you're counting the wrong thing if you're counting connections in TIME_WAIT.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/15647#issuecomment-218562866

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 11, 2016

I think you're confusing the relationship between TIME_WAIT and HTTP keep-alives.

What exactly do think Go is doing wrong? I see connections in TIME_WAIT regardless of HTTP keep-alive-vs-not and regardless of 204-vs-200.

@Dave-White-Rakuten

This comment has been minimized.

Copy link
Author

commented May 11, 2016

There’s no confusion. One is being used to observe the effects of the other.

That TIME_WAIT count can be used to view the number of connections that were recently used and are now in a wait state (where they will generally remain for about 60 seconds before they can be reused).

Keep-alive is a request to reuse an established connection for subsequent requests. You will see very few TIME_WAIT states under heavy load by few connections if those connections are being properly reused as requested.

Under load, if connections are forcibly closed after each response, you will see a rapid rise in the TIME_WAIT count. Soon there will be no connections available to Accept().

If you send a request using keep-alive, which receives a 204 response, the connection will be closed UNLESS a content-length is specified. This behavior makes no sense because by definition a 204 response will not contain any content.

Dave White
Senior Software Engineer
RAKUTEN marketing

6985 S Union Park Center, Suite 300 | Salt Lake City | UT | 84047
David.White@rakuten.com | rakutenmarketing.com

Rakuten Marketing

AFFILIATE NETWORK • ATTRIBUTION • DISPLAY • SEARCH
–––––––––––––––––––––––––––––––––––––––––––––––––––

!SAVE THE DATE!
SYMPOSIUM SAO PAULO SEPTEMBER 1
SYMPOSIUM SYDNEY SEPTEMBER 24
–––––––––––––––––––––––––––––––––––––––––––––––––––

Confidential Information (C) Copyright 2015 Rakuten Marketing, LLC. All rights reserved.
This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and CONFIDENTIAL.
If you are not the intended recipient, dissemination of this communication is prohibited. If you have received this communication in error,
please erase all copies of the message and its attachments and notify us immediately by replying to sender.

From: Brad Fitzpatrick
Reply-To: golang/go
Date: Wednesday, May 11, 2016 at 2:02 PM
To: golang/go
Cc: David White, Author
Subject: Re: [golang/go] StatusNoContent closing connections (#15647)

I think you're confusing the relationship between TIME_WAIT and HTTP keep-alives.

What exactly do think Go is doing wrong? I see connections in TIME_WAIT regardless of HTTP keep-alive-vs-not and regardless of 204-vs-200.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/15647#issuecomment-218572734

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 11, 2016

Okay, I see it now.

It only affects HTTP/1.0 requests with "Connection: keep-alive" requests. HTTP/1.1 requests are fine (which are implicitly keep-alive), and it's also fine (even with HTTP/1.1+keep-alive) if the server sets a Content-Length of "0" (which is then scrubbed in the response).

I'm not sure how you're able to reproduce this with a browser, though, since I know of no browsers which still send HTTP/1.0 requests.

@bradfitz bradfitz reopened this May 11, 2016

@bradfitz bradfitz changed the title StatusNoContent closing connections net/http: StatusNoContent closes HTTP/1.0 keep-alive connections May 11, 2016

@bradfitz bradfitz self-assigned this May 11, 2016

@bradfitz bradfitz added this to the Go1.7Maybe milestone May 11, 2016

@Dave-White-Rakuten

This comment has been minimized.

Copy link
Author

commented May 11, 2016

Awesome. Thanks for sticking with me.

Thanks
dw

Dave White
Senior Software Engineer
RAKUTEN marketing

6985 S Union Park Center, Suite 300 | Salt Lake City | UT | 84047
David.White@rakuten.com | rakutenmarketing.com

Rakuten Marketing

AFFILIATE NETWORK • ATTRIBUTION • DISPLAY • SEARCH
–––––––––––––––––––––––––––––––––––––––––––––––––––

!SAVE THE DATE!
SYMPOSIUM SAO PAULO SEPTEMBER 1
SYMPOSIUM SYDNEY SEPTEMBER 24
–––––––––––––––––––––––––––––––––––––––––––––––––––

Confidential Information (C) Copyright 2015 Rakuten Marketing, LLC. All rights reserved.
This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and CONFIDENTIAL.
If you are not the intended recipient, dissemination of this communication is prohibited. If you have received this communication in error,
please erase all copies of the message and its attachments and notify us immediately by replying to sender.

From: Brad Fitzpatrick
Reply-To: golang/go
Date: Wednesday, May 11, 2016 at 3:23 PM
To: golang/go
Cc: David White, Author
Subject: Re: [golang/go] StatusNoContent closing connections (#15647)

Okay, I see it now.

It only affects HTTP/1.0 requests with "Connection: keep-alive" requests. HTTP/1.1 requests are fine (which are implicitly keep-alive), and it's also fine (even with HTTP/1.1+keep-alive) if the server sets a Content-Length of "0" (which is then scrubbed in the response).

I'm not sure how you're able to reproduce this with a browser, though, since I know of no browsers which still send HTTP/1.0 requests.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/15647#issuecomment-218594163

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 11, 2016

Sent https://golang.org/cl/23061

Sorry for the initial skepticism. The overwhelming majority of bug reports or email threads about either ab or TIME_WAIT are bogus.

@gopherbot gopherbot closed this in eb9062b May 11, 2016

@golang golang locked and limited conversation to collaborators May 11, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.