curl 7.77.0 proxy tunnel CONNECT busy loop waiting for reply

[zloi-user]

curl loads the cpu, with many threads this is especially noticeable

on version 7.76.1 the problem is not observed

similar problem #5532

[bagder]

Please give us a lot more details. What protocol, what components, what platform and most importantly: how can we reproduce. We cannot act on this issue until we get data.

[zloi-user]

curl --proxy 'http://185.169.198.98:3128' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://208.115.109.250:443' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://165.22.81.30:42663' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://157.230.103.189:40151' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://150.136.178.43:80' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://181.3.93.87:10809' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://34.230.61.238:80' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://54.166.131.99:80' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://51.81.82.175:2003' 'https://www.google.com/favicon.ico' -o - >>/dev/null &
curl --proxy 'http://167.172.180.46:39898' 'https://www.google.com/favicon.ico' -o - >>/dev/null &

the problem manifests itself when using HTTP proxy
(Linux Gentoo, kernel 5.10.27 x86_64 AMD)

[jay]

We need curl version curl -V. Is there one particular site proxy causing the problem?

[zloi-user]

curl -V
curl 7.77.0 (x86_64-pc-linux-gnu) libcurl/7.77.0 OpenSSL/1.1.1k zlib/1.2.11 nghttp2/1.43.0
Release-Date: 2021-05-26
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps tftp
Features: HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM SSL TLS-SRP UnixSockets

the problem is not in the site, but when using an http proxy

[jay]

Ok. That's what i meant, is there one particular proxy causing the problem? This is going to be hard to debug unless we can isolate it to a single process.

[zloi-user]

Ok. That's what i meant, is there one particular proxy causing the problem? This is going to be hard to debug unless we can isolate it to a single process.

curl --proxy 'http://185.169.198.98:3128' 'https://www.google.com/favicon.ico' -o - >>/dev/null

[jay]

I can't reproduce this in Windows or Ubuntu.

[zloi-user]

I can't reproduce this in Windows or Ubuntu.

this is a public proxy, it does not work now,
if you take others, you will get a load on the cpu

curl --proxy 'http://167.172.110.165:9090' 'https://www.google.com/favicon.ico' -o - >>/dev/null

[jay]

curl --proxy 'http://167.172.110.165:9090' 'https://www.google.com/favicon.ico' -o - >>/dev/null

I get connection refused and no load

[zloi-user]

I get connection refused and no load

for clarity, run in a loop

while true;do curl --proxy 'http://157.230.103.91:37749' 'https://www.google.com/favicon.ico' -o - >>/dev/null; done

[jay]

bisected to 51c0ebc http: deal with partial CONNECT sends. repro done by not sending a reply to CONNECT, eg

socat tcp-listen:8000,reuseaddr,fork -
curl -v --proxy http://localhost:8000 https://foo

curl/lib/http.h

Lines 184 to 188 in 6b951a6

	enum {
	HTTPSEND_NADA, /* init */
	HTTPSEND_REQUEST, /* sending a request */
	HTTPSEND_BODY /* sending body */
	} sending;

curl/lib/http_proxy.c

Lines 141 to 155 in 6b951a6

	/* when we've sent a CONNECT to a proxy, we should rather either wait for the
	socket to become readable to be able to get the response headers or if
	we're still sending the request, wait for write. */
	int Curl_connect_getsock(struct connectdata *conn)
	{
	struct HTTP *http;
	DEBUGASSERT(conn);
	DEBUGASSERT(conn->connect_state);
	http = &conn->connect_state->http_proxy;
	if(http->sending)
	return GETSOCK_WRITESOCK(0);
	return GETSOCK_READSOCK(0);
	}

the commit changed the logic by having multi check if http->sending is !=0 during CONNECT then assume we're still writing and wait for the socket to be writable instead of readable, whereas before it would just check if it was readable. i assume this was necessary to implement partial CONNECT send. however in the CONNECT init phase (TUNNEL_INIT) Curl_buffer_send is called which sends out the actual CONNECT to the server and then sets http->sending to HTTPSEND_BODY immediately afterward. so it just busy loops checking for write when it should actually be checking for read. @bagder

[zloi-user]

the problem is not solved, curl still loads the cpu

on version 7.76.1, the problem is not observed, all versions after loading the cpu 100%

while true;do curl --proxy 'http://157.230.103.91:37749' 'https://www.google.com/favicon.ico' -o - >>/dev/null; done