Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Hijack deadlock if system clock is 1970 #19747

Closed
eidmanna opened this issue Mar 28, 2017 · 8 comments

Comments

Projects
None yet
4 participants
@eidmanna
Copy link

commented Mar 28, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8 windows/amd64
with crosscompile GOOS: "linux", GOARCH: "arm"

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=E:\gopath
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

with crosscompile for
GOOS: "linux",
GOARCH: "arm"

--> for raspberry pi with no internal clock

What did you do?

Start the programm and call --> :8080/test

package main

import (
"log"
"net/http"

_ "net/http/pprof"
)

func main() {
http.HandleFunc("/test", testHandler)
panic(http.ListenAndServe(":8080", nil))
}

func testHandler(w http.ResponseWriter, r *http.Request) {
h := w.(http.Hijacker)
log.Println("calling hijack")
c, _, err := h.Hijack()
log.Println("hijack returned err=", err)
c.Close()
}

What did you expect to see?

No deadlock

What did you see instead?

Deadlock in call to
c, _, err := h.Hijack()

The reason for this deadlock is, that the clock is set to
1970/01/01
With newer date eg 2017/01/01 no deadlock.

For details, discussion and heapdump see:
gorilla/websocket#232

@garyburd agreed, that this is probably an net/http issue

@bradfitz bradfitz changed the title Deadlock in net/http Hijack with very old date (1970/1/1) net/http: Hijack deadlock if system clock is 1970 Mar 28, 2017

@bradfitz

This comment has been minimized.

Copy link
Member

commented Mar 28, 2017

Yes, the net and net/http packages implement some connection timeout logic by changing the timeout to times way in the past. But apparently on your machine aLongTimeAgo is actually in the future and not in the past.

We could move aLongTimeAgo even earlier, perhaps to 1 second past the start of unix time.

But does your clock at least advance ever, even though it boots at 1970-01-01?

Which Raspberry Pi is this?

@bradfitz bradfitz added this to the Go1.8.1 milestone Mar 28, 2017

@eidmanna

This comment has been minimized.

Copy link
Author

commented Mar 28, 2017

I have a Raspberry Pi 3 Model B.

The time during my test was:
1970/01/01 01:13:59 (output of time by log.Println)

1 hour 13 min after boot.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Mar 28, 2017

I see. And you don't want to use ntp to fix the clock?

This is an HTTP server, so by definition it at least has some network available. Or maybe it's just localhost or local LAN?

And you're fine with the Server's "Date" response header being wrong? And Last-Modified and If-Modified-Since and such not working quite right?

I guess the question is how much we make an effort to do the right thing for busted clocks. I suppose Raspberry Pis are common, so we should make some effort.

@eidmanna

This comment has been minimized.

Copy link
Author

commented Mar 28, 2017

Now it is easy for me to fix it.

But it was not easy to figure out, that an old timestamp was the reason for a broken websocket.

My raspberry pi has no internet connection, only local lan. Without any changes I have this old timestamp automatically. Raspberry pi has no internal battery clock.

For my IoT project I do not need a correct time.

BTW: In the standard raspberry pi image ntp is installed and setup by default.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Mar 28, 2017

@rsc, cool to adjust aLongTimeAgo to Unix(1, 0)?

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2017

@bradfitz OK

@bradfitz bradfitz added the NeedsFix label Mar 29, 2017

@bradfitz bradfitz self-assigned this Mar 29, 2017

@bradfitz bradfitz removed the NeedsDecision label Mar 29, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Mar 29, 2017

CL https://golang.org/cl/38785 mentions this issue.

@gopherbot gopherbot closed this in 6983b9a Mar 29, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Mar 29, 2017

CL https://golang.org/cl/38786 mentions this issue.

@bradfitz bradfitz modified the milestones: Go1.9, Go1.8.1 Mar 29, 2017

gopherbot pushed a commit that referenced this issue Mar 29, 2017

[release-branch.go1.8] net, net/http: adjust time-in-past constant ev…
…en earlier

The aLongTimeAgo time value in net and net/http is used to cancel
in-flight read and writes. It was set to time.Unix(233431200, 0)
which seemed like far enough in the past.

But Raspberry Pis, lacking a real time clock, had to spoil the fun and
boot in 1970 at the Unix epoch time, breaking assumptions in net and
net/http.

So change aLongTimeAgo to time.Unix(1, 0), which seems like the
earliest safe value. I don't trust subsecond values on all operating
systems, and I don't trust the Unix zero time. The Raspberry Pis do
advance their clock at least. And the reported problem was that Hijack
on a ResponseWriter hung forever, waiting for the connection read
operation to finish. So now, even if kernel + userspace boots in under
a second (unlikely), the Hijack will just have to wait for up to a
second.

Updates #19747
Fixes #19771 (backport to Go 1.8.x)

Change-Id: Id59430de2e7b5b5117d4903a788863e9d344e53a
Reviewed-on: https://go-review.googlesource.com/38785
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
(cherry picked from commit e83fc2e44336423dab94bfe74fad4c4e6a4703b3)
Reviewed-on: https://go-review.googlesource.com/38786
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

lparth added a commit to lparth/go that referenced this issue Apr 13, 2017

net, net/http: adjust time-in-past constant even earlier
The aLongTimeAgo time value in net and net/http is used to cancel
in-flight read and writes. It was set to time.Unix(233431200, 0)
which seemed like far enough in the past.

But Raspberry Pis, lacking a real time clock, had to spoil the fun and
boot in 1970 at the Unix epoch time, breaking assumptions in net and
net/http.

So change aLongTimeAgo to time.Unix(1, 0), which seems like the
earliest safe value. I don't trust subsecond values on all operating
systems, and I don't trust the Unix zero time. The Raspberry Pis do
advance their clock at least. And the reported problem was that Hijack
on a ResponseWriter hung forever, waiting for the connection read
operation to finish. So now, even if kernel + userspace boots in under
a second (unlikely), the Hijack will just have to wait for up to a
second.

Fixes golang#19747

Change-Id: Id59430de2e7b5b5117d4903a788863e9d344e53a
Reviewed-on: https://go-review.googlesource.com/38785
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

@golang golang locked and limited conversation to collaborators Mar 29, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.