net/http: Request to certain server hangs, works fine with curl

[wjdp]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/will/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/will/go-build795220856=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

package main

import (
	"fmt"
	"net/http"
	"time"
)

func main() {
	client := &http.Client{
		Timeout: 30 * time.Second,
	}
	resp, err := client.Get("https://www.southampton.ac.uk/")
	if err != nil {
		panic(err)
	}
	fmt.Println(resp)
}

What did you expect to see?

A response within a second or two

What did you see instead?

Timeout after 30 seconds, will happily hang for longer if timeout extended.

The following is the output of curl on the same machine / network. This timeout issue with this server has also been observed on a Travis test runner.

~ ❯ curl -v https://www.southampton.ac.uk/
*   Trying 152.78.118.52...
* Connected to www.southampton.ac.uk (152.78.118.52) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 697 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: www.southampton.ac.uk (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: OU=Domain Control Validated,CN=www.southampton.ac.uk
* 	 start date: Tue, 13 Jan 2015 00:00:00 GMT
* 	 expire date: Fri, 12 Jan 2018 23:59:59 GMT
* 	 issuer: C=NL,ST=Noord-Holland,L=Amsterdam,O=TERENA,CN=TERENA SSL CA 2
* 	 compression: NULL
* ALPN, server did not agree to a protocol
> GET / HTTP/1.1
> Host: www.southampton.ac.uk
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 03 Aug 2017 21:38:44 GMT
< Content-Type: text/html;charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: JSESSIONID=9E437837C159A9276F8DA64CFF182DE7; Path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Referrer-Policy: unsafe-url
< Content-Security-Policy-Report-Only: default-src 'self' 'unsafe-inline' *.atbar.org *.issuu.com ddfbm2kt5ml6l.cloudfront.net cdn.soton.ac.uk cdn.southampton.ac.uk corporate.wordpress.soton.ac.uk blog.soton.ac.uk; script-src 'self' 'unsafe-inline' 'unsafe-eval' code.jquery.com www.googleadservices.com www.googletagmanager.com *.google-analytics.com connect.facebook.net static.ads-twitter.com j.flxpxl.com hj.flxpxl.com d2hlpp31teaww3.cloudfront.net d1d7fjtb6d4i2m.cloudfront.net analytics.twitter.com ib.adnxs.com www.intelliworkschat.com access.ecs.soton.ac.uk *.atbar.org www.youtube.com s.ytimg.com *.issuu.com ddfbm2kt5ml6l.cloudfront.net cdn.soton.ac.uk cdn.southampton.ac.uk stats.wp.com soton.disqus.com stats.wp.com corporate.wordpress.soton.ac.uk blog.soton.ac.uk s0.wp.com use.typekit.net connect.facebook.com platform.twitter.com; font-src 'self' data: ddfbm2kt5ml6l.cloudfront.net cdn.soton.ac.uk cdn.southampton.ac.uk use.typekit.net; img-src data: 'self' www.southampton.ac.uk cdn.soton.ac.uk cdn.southampton.ac.uk t.co mpp2.vindicosuite.com *.google-analytics.com *.facebook.com pixel.adsafeprotected.com *.doubleclick.net www.google.com www.google.co.uk *.atbar.org *.issuu.com image.isu.pub www.intelliworkschat.com pixel.wp.com blog.soton.ac.uk platform.twitter.com pbs.twimg.com syndication.twitter.com p.typekit.net d1d7fjtb6d4i2m.cloudfront.net; frame-src 'self' www.google.com www.google.co.uk www.youtube.com *.issuu.com ddfbm2kt5ml6l.cloudfront.net cdn.soton.ac.uk cdn.southampton.ac.uk; child-src 'self' www.google.com www.google.co.uk www.youtube.com *.issuu.com ddfbm2kt5ml6l.cloudfront.net cdn.soton.ac.uk cdn.southampton.ac.uk;
< Set-Cookie: BIGipServer~CorpWeb~pool-corpweb-prod-80=2678630592.20480.0000; path=/; Httponly; Secure
< 
<!DOCTYPE html>

[bradfitz]

This is a dup of #20979

The server negotiates "h2" and has "BIGipServer" in its response headers.

I can verify with GODEBUG=http2debug that it does the immediate GOAWAY thing. Closing this bug and moving to #20979

[wjdp]

Apologies for the duplicate and thanks for the quick response 😄

[odeke-em]

Oops, nvm @bradfitz beat me to it. @wjdp if you want to use http1, please use

package main

import (
	"crypto/tls"
	"flag"
	"fmt"
	"net/http"
	"time"
)

func main() {
	var http1 bool
	flag.BoolVar(&http1, "http1", false, "use http1 to invoke the server")
	flag.Parse()

	tr := &http.Transport{}
	if http1 {
		tr.TLSClientConfig = &tls.Config{
			NextProtos: []string{"h1"},
		}
	}

	client := &http.Client{
		Timeout:   30 * time.Second,
		Transport: tr,
	}
	resp, err := client.Get("https://www.southampton.ac.uk/")
	if err != nil {
		panic(err)
	}
	fmt.Println(resp)
}

and invoke it with flag http1 ie go run main.go --http1 and that should return your result ASAP.

[wjdp]

Thanks @odeke-em, provided snippet does the trick!

[HaraldNordgren]

I had a problem with a hanging host as well. On the surface very similar to this case although setting the protocol version did not work.

For me it was necessary in the TLS config to use

	DynamicRecordSizingDisabled: true

I posted a more fleshed out comment about it in: #20979 (comment) for debugging similar issues.

[bradfitz]

@HaraldNordgren, please file a separate crypto/tls bug for your case. It sounds like a crypto/tls issue instead of a net/http issue.