net/http: race detector not working

[win-t]

What version of Go are you using (go version)?

$ go version
go version go1.18 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/win/.cache/go-build"
GOENV="/home/win/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/win/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/win/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2412875146=/tmp/go-build -gno-record-gcc-switches"

What did you do?

func main() {
	counter := 0
	panic(http.ListenAndServe(":8080", http.HandlerFunc(
		func(w http.ResponseWriter, r *http.Request) {
			counter++
			fmt.Fprintln(w, counter)
		},
	)))
}

and run it with go run -race main.go, then curl it twice

What did you expect to see?

race detector detects the race condition

What did you see instead?

race detector doesn't detect the race condition

I believe the go race detector will catch it even though I run curl sequentially because in the following code, the race detected do detect it correctly

func main() {
	counter := 0

	go func() {
		counter++
		fmt.Println(counter)
	}()

	time.Sleep(1 * time.Second)

	go func() {
		counter++
		fmt.Println(counter)
	}()

	time.Sleep(1 * time.Second)
}

[dmitshur]

I'm not sure what guarantees the race detector makes about what it's definitely able to detect. Maybe not reporting a race when the curls were done sequentially is a feature, in that there was not actually a race (but could be if the requests arrived at the same time). Or maybe detecting such "potential races" could be a feature request.

It seems the following program also doesn't report a race (which likely doesn't happen if you wait enough seconds to be sure the first goroutine completed before providing stdin):

package main

import "fmt"

func main() {
	counter := 0

	go func() {
		counter++
		fmt.Println(counter)
	}()

	fmt.Scanln(new(string))

	go func() {
		counter++
		fmt.Println(counter)
	}()

	// Output (with -race):
	// 1
	// a
	// 2
}

CC @ianlancetaylor, @neild.

[ianlancetaylor]

The race detector does make an effort to detect potential races. Usually when the race detector fails to report a race it is because there is an unexpected synchroniation/serialization in a library.

The later example above will indeed report a race if run as ./x < /dev/null or as echo hi | ./x. Otherwise I think it's being effectively synchronized by the runtime I/O code. That may also be what is happening in the earlier example, I'm not sure.

[ianlancetaylor]

CC @dvyukov

[dvyukov]

All real races should be eventually detectable. In this case you play role of synchronization since you run curl sequentially ;)
If you write a stress test where the handler is invoked concurrently, the race should be detected eventually.

As Ian pointed, the false negative happens due to induced io synchronization:

src$ egrep "race\.(Acquire|Release)" syscall/*.go
syscall/syscall_unix.go:			race.Acquire(unsafe.Pointer(&ioSync))
syscall/syscall_unix.go:		race.ReleaseMerge(unsafe.Pointer(&ioSync))
syscall/syscall_unix.go:			race.Acquire(unsafe.Pointer(&ioSync))
syscall/syscall_unix.go:		race.ReleaseMerge(unsafe.Pointer(&ioSync))
syscall/syscall_unix.go:		race.ReleaseMerge(unsafe.Pointer(&ioSync))
syscall/syscall_windows.go:		race.Acquire(unsafe.Pointer(&ioSync))
syscall/syscall_windows.go:		race.ReleaseMerge(unsafe.Pointer(&ioSync))

Removing that will lead to false positives and we prefer to err on side of false negatives.

It may be possible to restructure http package to run the handler in unsynchronized goroutines. But not sure if it will work b/c the goroutine will still need to accept/read/receive before the handler and send after the handler.
Otherwise my vote for #WAI

[ianlancetaylor]

Thanks, closing as there is nothing we can reasonably do here.

[win-t]

thanks for clarification