net/http: bundled x/net/http2.responseWriter hangs forever #42534
Comments
|
Possible related issue #23559 |
|
cc @fraenkel |
|
@dtelyukh We are going to need something that can reproduce the issue. It would also help to enable http2 debug and a thread dump when it hangs. |
|
Your title says the bundled version of http2 has this issue. Are you implying that if you use the latest x/net/http2 you dont? |
It's not easy to prepare code, which can reproduce this problem with 100% guarantee, but I think I could try.
h2_bundle.go used by third-party code. I didn't try to use x/net/http2 directly. Do you mean that I should do that? |
|
Don't worry. I am going to need something that can reproduce this issue. Notice the stream for the PUSH is 1511 but the above is the first time I see that Framer. And the rest are in the 300s. I don't exactly see how this happened. |
|
I truncated log-file after each successful request until it was hanged. Maybe that is why the log-file was broken. I attach here other log-file, which was made when I caught problem from the first time. This log-file was never truncated. |
|
@fraenkel, we prepared test application for problem reproducing. My apologies for so complicated app. We cannot extract some small piece of code, because we don't know where is the problem exactly.
To have more chances to catch the problem it should to remove proxy cache:
To patch or debug server:
|
|
@dtelyukh I did find a way to cause the hang locally, from my machine it would never happen. |
|
Never mind, I got it working again.... |
|
So one thing I did verify is that using the latest golang/x/net/http2 code does not cause the hang I see with my simple testcase. |
|
@fraenkel, what can I help? |
|
You can see the one line change I made to caddyserver with a |
|
@fraenkel, this new version is never hangs for us. And also we noticed that app become faster. |
|
Full page load time (with all resources) is 2% less than with old http/2, and median absolute deviation is 1% less too. So it's both faster, and shows more stable performance. |
|
@fraenkel, should I close this ticket? |
|
yes, given there is a solution and this should be fixed in 1.16 although one should verify that is true. |
* implement default values for header directive closes #3804 * remove `set_default` header op and rely on "require" handler instead This has the following advantages over the previous attempt: - It does not introduce a new operation for headers, but rather nicely extends over an existing feature in the header handler. - It removes the need to specify the header as "deferred" because it is already implicitely deferred by the use of the require handler. This should be less confusing to the user. * add integration test for header directive in caddyfile * bubble up errors when parsing caddyfile header directive * don't export unnecessarily and don't canonicalize headers unnecessarily * fix response headers not passed in blocks * caddyfile: fix clash when using default header in block Each header is now set in a separate handler so that it doesn't clash with other headers set/added/deleted in the same block. * caddyhttp: New idle_timeout default of 5m * reverseproxy: fix random hangs on http/2 requests with server push (#3875) see golang/go#42534 * Refactor and cleanup with improvements * More specific link Co-authored-by: Matthew Holt <mholt@users.noreply.github.com> Co-authored-by: Денис Телюх <telyukh.denis@gmail.com>
|
@fraenkel, this bug is still exist. But we found more clear way to reproduce it. An issue in Caddy's repository: caddyserver/caddy#3896 How to reproduceIt depends on the proxied website and caddy config, and some random factors, thus it occurs with different frequency on different hardware. The steps are:
|
|
I reproduced a hang but its using the bundle http2 stack. |
|
Write me, please, to fix it should bundle a recent version of http2-library? Or does this library itself need to be fixed? |
|
Looks like with this hack it never hangs |
|
The solution is to use explicitly x/net/http2. |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
So, let's begin
High-Level Problem Description
I'm creating reverse proxy with Caddy Server and my own plugin. I use http/2 and Server Push. Sometimes requests hang forever. Here is screenshot from Chrome DevTools:
Low-Level Problem Description
So, I started to debug this situation. I found that my code execution stuck at
(http.responseWriter).Write(), which is an instance of http2responseWriter.With help of
pprofI found that lockup happens in two functions: http2serverConn.writeHeaders and http2serverConn.writeDataFromHandler - endless waiting of data fromdonechannel.Here is an illustration from
pprof:Next I built
gofrom source with adding some debug messages and start to dive deeper.I found a problem with frames are sent to output. At this line
Nframes were pushed: https://github.com/golang/go/blob/go1.15.4/src/net/http/h2_bundle.go#L4692. Afterpush-functionscheduleFrameWrite-function is called. I watched into it and found that it often exit here: https://github.com/golang/go/blob/go1.15.4/src/net/http/h2_bundle.go#L4817. And onlyM(M<N) frames were popped from queue here: https://github.com/golang/go/blob/go1.15.4/src/net/http/h2_bundle.go#L4837Pushed Frames
Popped Frames
What did you expect to see?
No lockups.
What did you see instead?
Random lockups.
The text was updated successfully, but these errors were encountered: