net, net/http: Deadline and Timeout Fail to Close Hung Connections.

[JustinTimperio]

What version of Go are you using (go version)?

go 1.16 
AND 
go 1.17

Does this issue reproduce with the latest release?

Yes, this has been tested on 1.17

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/justin/.cache/go-build"
GOENV="/home/justin/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/justin/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/justin/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/justin/deadlinetest/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build833553530=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Context:

I am part of a research project called "Project-Lernaean". We are working on mapping and surveying the internet for both developers and security researchers. We have been using Colly quite successfully in our scans but recently have run into a fairly obscure issue. In a variety of cases, we run into IPs that deadlock something during a page request.

Test Cases:

Two test repositories have been created to show the bug in action. One uses Colly and the other is pure net/http and net.
https://github.com/dherbst/deadlinetest (Thanks to the work of https://github.com/dherbst)
https://github.com/Project-Lernaean/CollyTimeoutError

In both cases, Deadline and Timeout setting within http.Transport fail to terminate these connections and hang indefinitely.

Transport Settings:

transport := &http.Transport{
	IdleConnTimeout:       5 * time.Second,
	MaxIdleConns:          100,
	ExpectContinueTimeout: 10 * time.Second,
	DialContext: (&net.Dialer{
		Timeout:  1 * time.Second,
		Deadline: time.Now().Add(20 * time.Second),
	}).DialContext,
}

What did you expect to see?

I would expect that Deadline would be the absolute max time a connection can take before it is forcefully terminated.

What did you see instead?

The connection never fails and remains "open" indefinitely.

[cherrymui]

I think the Timeout and Deadline on DialContext are for dialing (creating) the connection. Once the connection is created, they are not responsible for closing it. But I'm not very familiar with how this works.

cc @neild @bradfitz

[kbolino]

I think @cherrymui is correct, but that raises the question of how to set a timeout/deadline for the request properly.

The obvious answer seems to be to add a context.Context with a deadline set to the request (e.g. via http.NewRequestWithContext or adding to an existing request with its WithContext method). However, this doesn't seem to have the desired effect in all scenarios. In particular, if the client is hung in a call to Read or Write on the underlying connection, it won't (can't?) be interrupted even if it exceeds the deadline.

What I'm describing may be a separate issue, however.

[neild]

net.Dialer.Timeout and net.Dialer.Deadline are indeed for the controlling the deadline of the dial operation only.

Setting a request deadline with http.NewRequestWithContext should work in all circumstances, including when reads or writes to the underlying connection are slow. If that doesn't work, we have a bug, but that does not seem to be the case here.