Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net, net/http: Deadline and Timeout Fail to Close Hung Connections. #48125

Open
JustinTimperio opened this issue Sep 1, 2021 · 1 comment
Open

Comments

@JustinTimperio
Copy link

@JustinTimperio JustinTimperio commented Sep 1, 2021

What version of Go are you using (go version)?

go 1.16 
AND 
go 1.17

Does this issue reproduce with the latest release?

Yes, this has been tested on 1.17

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/justin/.cache/go-build"
GOENV="/home/justin/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/justin/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/justin/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/justin/deadlinetest/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build833553530=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Context:

I am part of a research project called "Project-Lernaean". We are working on mapping and surveying the internet for both developers and security researchers. We have been using Colly quite successfully in our scans but recently have run into a fairly obscure issue. In a variety of cases, we run into IPs that deadlock something during a page request.

Test Cases:

Two test repositories have been created to show the bug in action. One uses Colly and the other is pure net/http and net.
https://github.com/dherbst/deadlinetest (Thanks to the work of https://github.com/dherbst)
https://github.com/Project-Lernaean/CollyTimeoutError

In both cases, Deadline and Timeout setting within http.Transport fail to terminate these connections and hang indefinitely.

Transport Settings:

transport := &http.Transport{
	IdleConnTimeout:       5 * time.Second,
	MaxIdleConns:          100,
	ExpectContinueTimeout: 10 * time.Second,
	DialContext: (&net.Dialer{
		Timeout:  1 * time.Second,
		Deadline: time.Now().Add(20 * time.Second),
	}).DialContext,
}

What did you expect to see?

I would expect that Deadline would be the absolute max time a connection can take before it is forcefully terminated.

What did you see instead?

The connection never fails and remains "open" indefinitely.

@cherrymui cherrymui added this to the Backlog milestone Sep 2, 2021
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Sep 2, 2021

I think the Timeout and Deadline on DialContext are for dialing (creating) the connection. Once the connection is created, they are not responsible for closing it. But I'm not very familiar with how this works.

cc @neild @bradfitz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants