crypto/tls: TLS handshake timeout

[moeen]

What version of Go are you using (go version)?

$ go version
go version go1.15.7 linux/amd64

Does this issue reproduce with the latest release?

I have tested this with Go 1.13+.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/moein/.cache/go-build"
GOENV="/home/moein/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/moein/go/pkg/mod"
GONOPROXY="="
GONOSUMDB="="
GOOS="linux"
GOPATH="/home/moein/go"
GOPRIVATE="="
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib64/go/1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib64/go/1.15/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build173751188=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.15.7 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.15.7
uname -sr: Linux 5.10.12-1-default
LSB Version:	n/a
Distributor ID:	openSUSE
Description:	openSUSE Tumbleweed
Release:	20210210
Codename:	n/a
/lib64/libc.so.6: GNU C Library (GNU libc) release release version 2.32 (git 0a8262a1b2).

What did you do?

I have a http.Client like this:

client := &http.Client{
	Timeout: 20 * time.Second,
	Transport: &http.Transport{
		TLSHandshakeTimeout: 10 * time.Second,
	},
}

And I send a HTTP request with:
res, err := client.Get(url)

What did you expect to see?

It should return a 2xx response without any error.

What did you see instead?

I get net/http: TLS handshake timeout.

Same URL works perfectly with curl.
I have also tried with InsecureSkipVerify: true but it didn't change anything.

[seankhliao]

This isn;t enough info to investigate or reproduce the error

[moeen]

The URL I'm trying to access is: https://api.kavenegar.com/v1/0/utils/getdate.json

curl -v output

*   Trying 172.67.72.69:443...
* Connected to api.kavenegar.com (172.67.72.69) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Aug 10 00:00:00 2020 GMT
*  expire date: Aug 10 12:00:00 2021 GMT
*  subjectAltName: host "api.kavenegar.com" matched cert's "*.kavenegar.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5581b9761e00)
> GET /v1/0/utils/getdate.json HTTP/2
> Host: api.kavenegar.com
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Mon, 15 Feb 2021 07:58:59 GMT
< content-type: application/json;charset=utf-8
< set-cookie: __cfduid=db8c442833a0950a337432cf944ee00ba1613375939; expires=Wed, 17-Mar-21 07:58:59 GMT; path=/; domain=.kavenegar.com; HttpOnly; SameSite=Lax
< cache-control: no-store, no-cache, must-revalidate,no-cache
< pragma: no-cache
< access-control-allow-origin: *
< access-control-allow-headers: Content-Type
< access-control-allow-method: POST, GET, OPTIONS
< cf-cache-status: DYNAMIC
< cf-request-id: 08464c3cad000016f27ea35000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=T3UJaKwL9fgmEatX7OjGfvEyD6LmEjrQ2YouKs5Fo9UkENrdOsqt6oYvdayG9ZcIb2U%2Bqv9zeGX0FNmlkxoRp5zh77DAUxb5uyqYNl2c6%2FxBCw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 621d7ca77a5916f2-FRA
< alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400
< 
* Connection #0 to host api.kavenegar.com left intact
{"return":{"status":200,"message":"تایید شد"},"entries":{"datetime":"27/11/1399 11:28:59 ق.ظ","year":2021,"month":2,"day":15,"hour":11,"minute":28,"second":59,"unixtime":1613388539}}

[davecheney]

Can you tcp dump the connection made from your go code?

[davecheney]

Also, does the timeout occur when you use the default http transport?

[moeen]

Also, does the timeout occur when you use the default http transport?

Yes. It does not change.

[moeen]

Also It only occurs with some Internet Providers.

[moeen]

Can you tcp dump the connection made from your go code?

sudo tcpdump -i wlp5s0 port 443 -w capture
capture.zip

[davecheney]

This pcap is talking to a different host, 104.26.10.226, than the curl conversation your provided before. Are you sure they’re the same?

[moeen]

This pcap is talking to a different host, 104.26.10.226, than the curl conversation your provided before. Are you sure they’re the same?

It might be related to Cloudflare. Now curl is also using the same IP:

curl -v output

*   Trying 104.26.10.226:443...
*   Trying 2606:4700:20::ac43:4845:443...
* Immediate connect fail for 2606:4700:20::ac43:4845: Network is unreachable
*   Trying 2606:4700:20::681a:ae2:443...
* Immediate connect fail for 2606:4700:20::681a:ae2: Network is unreachable
*   Trying 2606:4700:20::681a:be2:443...
* Immediate connect fail for 2606:4700:20::681a:be2: Network is unreachable
* Connected to api.kavenegar.com (104.26.10.226) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Aug 10 00:00:00 2020 GMT
*  expire date: Aug 10 12:00:00 2021 GMT
*  subjectAltName: host "api.kavenegar.com" matched cert's "*.kavenegar.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e5489ede00)
> GET /v1/0/utils/getdate.json HTTP/2
> Host: api.kavenegar.com
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Mon, 15 Feb 2021 08:25:14 GMT
< content-type: application/json;charset=utf-8
< set-cookie: __cfduid=d3edb3684e8b20f35337dcfb489fef6e31613377514; expires=Wed, 17-Mar-21 08:25:14 GMT; path=/; domain=.kavenegar.com; HttpOnly; SameSite=Lax
< cache-control: no-store, no-cache, must-revalidate,no-cache
< pragma: no-cache
< access-control-allow-origin: *
< access-control-allow-headers: Content-Type
< access-control-allow-method: POST, GET, OPTIONS
< cf-cache-status: DYNAMIC
< cf-request-id: 084664449c0000325c3f002000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"max_age":604800,"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=KAOxy2XKu%2BvOPcY1ccX8rgoFbGKZnBKrO9udwfz5rwHc7QUn7gaUCgPnmJ%2FGiKqe0nRKbd4CQsOGxUf2sxIQEmWUkZqayhiOJKsLVh1IvOetBw%3D%3D"}],"group":"cf-nel"}
< nel: {"max_age":604800,"report_to":"cf-nel"}
< server: cloudflare
< cf-ray: 621da31a9e56325c-FRA
< alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400
< 
* Connection #0 to host api.kavenegar.com left intact
{"return":{"status":200,"message":"تایید شد"},"entries":{"datetime":"15/02/2021 11:55:06 ق.ظ","year":2021,"month":2,"day":15,"hour":11,"minute":55,"second":6,"unixtime":1613390106}}

[davecheney]

(~) % tcpdump -r ~/Downloads/capture
reading from file /Users/davecheney/Downloads/capture, link-type EN10MB (Ethernet)
19:16:05.022681 IP 172.20.10.4.53224 > 104.26.10.226.https: Flags [S], seq 3912068042, win 64240, options [mss 1460,sackOK,TS val 2814302166 ecr 0,nop,wscale 7], length 0
19:16:05.208290 IP 104.26.10.226.https > 172.20.10.4.53224: Flags [S.], seq 3429290968, ack 3912068043, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
19:16:05.208406 IP 172.20.10.4.53224 > 104.26.10.226.https: Flags [.], ack 1, win 502, length 0
19:16:05.209020 IP 172.20.10.4.53224 > 104.26.10.226.https: Flags [P.], seq 1:288, ack 1, win 502, length 287
19:16:05.277965 IP 104.26.10.226.https > 172.20.10.4.53224: Flags [.], ack 288, win 64, length 0
19:16:15.209236 IP 172.20.10.4.53224 > 104.26.10.226.https: Flags [F.], seq 288, ack 1, win 502, length 0
19:16:15.346076 IP 104.26.10.226.https > 172.20.10.4.53224: Flags [.], ack 289, win 64, length 0

Looks like the start of a TLS session to me. Is someone MITMing your connection?

[moeen]

We have a high internet censorship in Iran. It might be related to that.

[davecheney]

It's quite likely. Is there anything that you think can be done in Go to address this problem (I cannot think of anything offhand), or should we close this issue?

[moeen]

It's quite likely. Is there anything that you think can be done in Go to address this problem (I cannot think of anything offhand), or should we close this issue?

I think it might be a crypto/tls issue since browser and cURL are working fine.