Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bundle collision when using the same proxy for two remotes #3951

Closed
TvdW opened this issue May 27, 2019 · 8 comments

Comments

Projects
None yet
2 participants
@TvdW
Copy link

commented May 27, 2019

I did this

https://gist.github.com/TvdW/bea189b3926993fff584471457350d1d

* Issue another request to this URL: 'http://tvdw.eu/'
* Found bundle for host tvdw.eu: 0xc5c8d0 [can multiplex]
* Multiplexed connection found!
* Found connection 8, with 1 requests on it
* Re-using existing connection! (#8) with proxy my-proxy-host
* Send failure: Bad file descriptor
* Failed sending HTTP request
* Connection died, retrying a fresh connect
* Issue another request to this URL: 'http://tvdw.eu/'
* Found bundle for host tvdw.eu: 0xc5c8d0 [can multiplex]
* Multiplexed connection found!
* Found connection 8, with 1 requests on it
* Re-using existing connection! (#8) with proxy my-proxy-host
* Send failure: Bad file descriptor
* Failed sending HTTP request
* Connection died, retrying a fresh connect

"Bad file descriptor" after a few requests. Seems to happen when mixing HTTP/1.1 and HTTP/2 in the same multi.

It looks like the fd gets closed while a request is still waiting for a response. A new connection may be opened, reusing the same fd, making it a valid file descriptor once again. curl may then issue a new request on the reused fd, but when reading the response could become available to the original request. Since the connection is not closed afterwards, the second request never finishes and times out.

(To be clear, that's not speculation: I've actually seen several responses arrive for the wrong request, and a lot of timeouts.)

Seems to be a new bug in 7.65.0, 7.64.1 did not have it.

I expected the following

All requests should happily execute and the program should finish. There should be no responses arriving for the wrong request either.

curl/libcurl version

curl 7.65.0 (x86_64-pc-linux-gnu) libcurl/7.65.0 OpenSSL/1.0.2k-fips zlib/1.2.7 brotli/1.0.1 c-ares/1.15.0 nghttp2/1.38.0
Release-Date: 2019-05-22
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets

operating system

CentOS 7 with custom-built curl

@bagder

This comment has been minimized.

Copy link
Member

commented May 28, 2019

Possibly related to #3952 and I think #3953 might fix or at least change this.

@bagder bagder added the HTTP/2 label May 28, 2019

@TvdW

This comment has been minimized.

Copy link
Author

commented May 28, 2019

#3953 doesn't seem to change this.

I filed #3951 and #3952 separately because 3951 is new in 7.65.0, and 3952 was already in 7.64.1. I suspect they are different bugs.

bagder added a commit that referenced this issue May 28, 2019

conncache: make "bundles" per host name when doing proxy tunnels
Only HTTP proxy use where multiple host names can be used over the same
connection should use the proxy host name for bundles.

Reported-by: Tom van der Woerdt
Fixes #3951
@bagder

This comment has been minimized.

Copy link
Member

commented May 28, 2019

I didn't manage to reproduce your symptoms exactly, but I found this issue that I think could be one that causes these problems for you. Let's start out trying this at least. See #3955

@TvdW

This comment has been minimized.

Copy link
Author

commented May 28, 2019

Hm, I don't think #3955 will fix this, or at least not the issue I started out with, as it didn't involve a proxy at all... (edit: github issue has a proxy listed, but the bug I originally started debugging was with connections to localhost, no proxy, but hard to replicate outside of my specific environment)

Will try the patch when I'm near a compiler, and report back.

@bagder

This comment has been minimized.

Copy link
Member

commented May 28, 2019

No, if you make it happen without proxy, #3955 will not change anything for you...

@TvdW

This comment has been minimized.

Copy link
Author

commented May 28, 2019

Beautiful.

#3955 resolves the problem as reproducible using the code in this issue. However, it turns out that during test minimization yesterday, I hit a different bug and then started minimizing towards that instead. 😆 🎉 I reported a different bug than I was experiencing.

So let's try this again... here's a gist trying to reproduce the issue I was originally having. https://gist.github.com/TvdW/f4815f8080d9dc30daa5d74b1f878dae

There's a nginx server running on 127.0.0.1, listening on port 80 for http/1.1, and on port 81 for http/2 (no TLS on either). I guess a simple nginx config to reproduce would be along the lines of

server {
listen *:80;
root /var/empty;
location / { }
}
server {
listen *:81 http2;
root /var/empty;
location / { }
}

Full log: https://gist.github.com/TvdW/37f7e01bf7b3b8763557d0a1c204379f

@bagder

This comment has been minimized.

Copy link
Member

commented May 28, 2019

Ah, ok. Would you mind submitting this as a new issue as then we can consider #3955 a fix of this and I can work on this new one separately?

@TvdW TvdW changed the title multi: requests may get response for request issued in parallel (or timeout, or errors) bundle collision when using the same proxy for two remotes May 28, 2019

@TvdW

This comment has been minimized.

Copy link
Author

commented May 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.