runtime: `-race` leaks memory on Go >= 1.19

[walles]

What version of Go are you using (go version)?

root@4fa353104d14:/go# go version
go version go1.21.1 linux/arm64
root@4fa353104d14:/go#

Inside of docker run -it --rm golang:1.21.

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

root@4fa353104d14:/go# go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1393083128=/tmp/go-build -gno-record-gcc-switches'
root@4fa353104d14:/go#

What did you do?

• Save this program as main.go.
• With Go 1.19 or later (1.18 does not leak, or at least a lot less)...
• go run -race main.go (without -race there is no leak)

Also, just having the HTTP server part is enough to leak, but I wanted a repro that is standalone so the client code and the memory reporting code are there for that.

package main

import (
	"log"
	"os"
	"os/exec"
	"strconv"
	"strings"
	"time"

	"net/http"

	_ "net/http/pprof"
)

func main() {
	printMemUsage()

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {})
	server := http.Server{
		Addr:        ":5019",
		ReadTimeout: 5 * time.Second,
	}

	go func() {
		log.Println("Starting service", "http://localhost:5019/")
		if err := server.ListenAndServe(); err != nil { // blocks
			log.Fatalln("Error while running service:", err)
		}
	}()

	// Give the server time to start
	time.Sleep(2 * time.Second)

	log.Println("Now flooding the server with requests...")
	nextMemLog := time.Now()
	for {
		// Trigger the handler function
		response, err := http.Get("http://localhost:5019/")
		if err != nil {
			log.Fatalln("Error while making request:", err)
		}
		if response.StatusCode != http.StatusOK {
			log.Fatalln("Unexpected status code:", response.StatusCode)
		}
		if response.Body != nil {
			response.Body.Close()
		}

		if time.Now().After(nextMemLog) {
			nextMemLog = time.Now().Add(5 * time.Second)
			printMemUsage()
		}
	}
}

func printMemUsage() {
	out, err := exec.Command("ps", "-o", "rss", "-p", strconv.Itoa(os.Getpid())).Output()
	if err != nil {
		log.Fatalln("Error while running ps command, is it installed?", err)
	}

	// Split output into "RSS" and "1234"
	rss := strings.Fields(string(out))

	log.Println("Memory usage:", rss[1], "kB")
}

What did you expect to see?

Stable memory usage

What did you see instead?

2023/09/27 12:54:27 RssAnon:	   14172 kB
2023/09/27 12:54:27 Starting service http://localhost:5019/
2023/09/27 12:54:29 Now flooding the server with requests...
2023/09/27 12:54:29 RssAnon:	   15724 kB
2023/09/27 12:54:34 RssAnon:	   54000 kB
2023/09/27 12:54:39 RssAnon:	   63400 kB
2023/09/27 12:54:44 RssAnon:	   72044 kB
2023/09/27 12:54:49 RssAnon:	   80796 kB
2023/09/27 12:54:54 RssAnon:	   89384 kB
2023/09/27 12:54:59 RssAnon:	   99744 kB
2023/09/27 12:55:04 RssAnon:	  108400 kB
2023/09/27 12:55:09 RssAnon:	  116744 kB
2023/09/27 12:55:14 RssAnon:	  124632 kB
2023/09/27 12:55:19 RssAnon:	  132332 kB
[...]

Notes

• Go 1.19 got a new Thread Sanitizer version, this might be related: https://tip.golang.org/doc/go1.19#runtime/race
• Unsure whether this is the same as runtime: defer/recover support in race detector #26813.
  • This one is only Go 1.19 and above but the other issue was reported for Go 1.10.
  • There's a comment saying that the repro case in the other ticket doesn't work any more with Go 1.21, but this repro does work with Go 1.21.

[thanm]

@golang/compiler @golang/runtime

[thanm]

Thanks for the report.

@walles you write that this happens "On Linux (not on macOS)..." -- does that mean you've tried it on a mac and the leak doesn't happen, or just that the program is linux-specific?

[walles]

@thanm I have tried it on a mac and the leak didn't happen.

It leaks on macOS as well, sorry about that.

I updated the repro case at the start of this ticket to use ps to measure its own RAM usage, so now the repro case can be run on both macOS and Linux.

[mknyszek]

In triage, we're a bit unsure as to what the cause could be. It seems like bisecting the toolchain might be the best path forward to try and identify the root cause.

[mknyszek]

@walles Would you be willing to do the bisection, since you have a reproducer readily available in your environment?

[walles]

@mknyszek What happened when you tried the repro?

Did go run -race main.go reproduce the leak for you?