runtime: `-race` leaks memory on Go >= 1.19 #63276

walles · 2023-09-28T06:18:22Z

What version of Go are you using (`go version`)?

root@4fa353104d14:/go# go version
go version go1.21.1 linux/arm64
root@4fa353104d14:/go#

Inside of docker run -it --rm golang:1.21.

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

root@4fa353104d14:/go# go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1393083128=/tmp/go-build -gno-record-gcc-switches'
root@4fa353104d14:/go#

What did you do?

Save this program as main.go.
With Go 1.19 or later (1.18 does not leak, or at least a lot less)...
go run -race main.go (without -race there is no leak)

Also, just having the HTTP server part is enough to leak, but I wanted a repro that is standalone so the client code and the memory reporting code are there for that.

package main

import (
	"log"
	"os"
	"os/exec"
	"strconv"
	"strings"
	"time"

	"net/http"

	_ "net/http/pprof"
)

func main() {
	printMemUsage()

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {})
	server := http.Server{
		Addr:        ":5019",
		ReadTimeout: 5 * time.Second,
	}

	go func() {
		log.Println("Starting service", "http://localhost:5019/")
		if err := server.ListenAndServe(); err != nil { // blocks
			log.Fatalln("Error while running service:", err)
		}
	}()

	// Give the server time to start
	time.Sleep(2 * time.Second)

	log.Println("Now flooding the server with requests...")
	nextMemLog := time.Now()
	for {
		// Trigger the handler function
		response, err := http.Get("http://localhost:5019/")
		if err != nil {
			log.Fatalln("Error while making request:", err)
		}
		if response.StatusCode != http.StatusOK {
			log.Fatalln("Unexpected status code:", response.StatusCode)
		}
		if response.Body != nil {
			response.Body.Close()
		}

		if time.Now().After(nextMemLog) {
			nextMemLog = time.Now().Add(5 * time.Second)
			printMemUsage()
		}
	}
}

func printMemUsage() {
	out, err := exec.Command("ps", "-o", "rss", "-p", strconv.Itoa(os.Getpid())).Output()
	if err != nil {
		log.Fatalln("Error while running ps command, is it installed?", err)
	}

	// Split output into "RSS" and "1234"
	rss := strings.Fields(string(out))

	log.Println("Memory usage:", rss[1], "kB")
}

What did you expect to see?

Stable memory usage

What did you see instead?

2023/09/27 12:54:27 RssAnon:	   14172 kB
2023/09/27 12:54:27 Starting service http://localhost:5019/
2023/09/27 12:54:29 Now flooding the server with requests...
2023/09/27 12:54:29 RssAnon:	   15724 kB
2023/09/27 12:54:34 RssAnon:	   54000 kB
2023/09/27 12:54:39 RssAnon:	   63400 kB
2023/09/27 12:54:44 RssAnon:	   72044 kB
2023/09/27 12:54:49 RssAnon:	   80796 kB
2023/09/27 12:54:54 RssAnon:	   89384 kB
2023/09/27 12:54:59 RssAnon:	   99744 kB
2023/09/27 12:55:04 RssAnon:	  108400 kB
2023/09/27 12:55:09 RssAnon:	  116744 kB
2023/09/27 12:55:14 RssAnon:	  124632 kB
2023/09/27 12:55:19 RssAnon:	  132332 kB
[...]

Notes

Go 1.19 got a new Thread Sanitizer version, this might be related: https://tip.golang.org/doc/go1.19#runtime/race
Unsure whether this is the same as runtime: defer/recover support in race detector #26813.
- This one is only Go 1.19 and above but the other issue was reported for Go 1.10.
- There's a comment saying that the repro case in the other ticket doesn't work any more with Go 1.21, but this repro does work with Go 1.21.

The text was updated successfully, but these errors were encountered:

thanm · 2023-09-28T15:47:46Z

@golang/compiler @golang/runtime

thanm · 2023-09-28T15:48:33Z

Thanks for the report.

@walles you write that this happens "On Linux (not on macOS)..." -- does that mean you've tried it on a mac and the leak doesn't happen, or just that the program is linux-specific?

walles · 2023-09-28T16:45:37Z

~~@thanm I have tried it on a mac and the leak didn't happen.~~

It leaks on macOS as well, sorry about that.

I updated the repro case at the start of this ticket to use ps to measure its own RAM usage, so now the repro case can be run on both macOS and Linux.

mknyszek · 2023-10-04T19:17:04Z

In triage, we're a bit unsure as to what the cause could be. It seems like bisecting the toolchain might be the best path forward to try and identify the root cause.

mknyszek · 2023-10-04T19:17:40Z

@walles Would you be willing to do the bisection, since you have a reproducer readily available in your environment?

walles · 2023-10-04T19:41:25Z

@mknyszek What happened when you tried the repro?

Did go run -race main.go reproduce the leak for you?

aclements · 2023-10-10T18:58:24Z

@walles , we might not get a chance to bisect this for a while. If you have a chance to bisect it, that might help us a lot with moving this issue forward more quickly.

walles · 2023-10-10T19:59:16Z

Do you think you could at least validate the repro case now?

I have only run it myself, so if at least one other person could verify that it shows a leak that would be super!

nakulbajaj · 2023-10-11T23:25:23Z

Can confirm I am seeing a leak with the same script on macOS

GOARCH='arm64'
GOOS='darwin'

Output

~/open-source » go run -race test.go
2023/10/11 16:19:36 Memory usage: 24416 kB
2023/10/11 16:19:36 Starting service http://localhost:5019/
2023/10/11 16:19:38 Now flooding the server with requests...
2023/10/11 16:19:38 Memory usage: 28624 kB
2023/10/11 16:19:43 Memory usage: 76256 kB
2023/10/11 16:19:48 Memory usage: 85456 kB
2023/10/11 16:19:53 Memory usage: 97936 kB
2023/10/11 16:19:58 Memory usage: 107552 kB
2023/10/11 16:20:03 Memory usage: 116432 kB
2023/10/11 16:20:08 Memory usage: 124512 kB
2023/10/11 16:20:13 Memory usage: 134416 kB
2023/10/11 16:20:18 Memory usage: 136544 kB
2023/10/11 16:20:23 Memory usage: 150528 kB
2023/10/11 16:20:28 Memory usage: 160464 kB
2023/10/11 16:20:33 Memory usage: 168704 kB
2023/10/11 16:20:38 Memory usage: 177744 kB
2023/10/11 16:20:43 Memory usage: 186848 kB
2023/10/11 16:20:48 Memory usage: 195184 kB
2023/10/11 16:20:53 Memory usage: 205136 kB

cuonglm · 2023-10-12T10:57:40Z

git bisect points to https://go-review.googlesource.com/c/go/+/333529

thanm · 2023-10-12T13:11:22Z

Thank you @cuonglm for the bisect. Suspect CL seems plausible to me. I will dig into it a bit more.

jgbooks · 2023-12-04T08:29:34Z

I didn't add -race. When I ran it for 2 hours, I noticed that the memory was growing slowly.
@walles @thanm

go version go1.21.0 linux/amd64

cat /etc/*release:
Red Hat Enterprise Linux Server release 6.1 (Santiago)

uname -a :
Linux bogon 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

What did you do?

go build main.go

What did you see instead?

After 2 hours :

walles changed the title ~~affected/package: -race leaks memory on Go >= 1.19~~ -race leaks memory on Go >= 1.19 Sep 28, 2023

seankhliao changed the title ~~-race leaks memory on Go >= 1.19~~ runtime: -race leaks memory on Go >= 1.19 Sep 28, 2023

bcmills added RaceDetector compiler/runtime Issues related to the Go compiler and/or runtime. labels Sep 28, 2023

thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 28, 2023

mknyszek added this to the Backlog milestone Oct 4, 2023

mknyszek assigned thanm Oct 11, 2023

runtime: `-race` leaks memory on Go >= 1.19 #63276

runtime: `-race` leaks memory on Go >= 1.19 #63276

walles commented Sep 28, 2023 •

edited

thanm commented Sep 28, 2023

thanm commented Sep 28, 2023

walles commented Sep 28, 2023 •

edited

mknyszek commented Oct 4, 2023

mknyszek commented Oct 4, 2023

walles commented Oct 4, 2023 •

edited

aclements commented Oct 10, 2023

walles commented Oct 10, 2023

nakulbajaj commented Oct 11, 2023

cuonglm commented Oct 12, 2023

thanm commented Oct 12, 2023

jgbooks commented Dec 4, 2023

runtime: -race leaks memory on Go >= 1.19 #63276

runtime: -race leaks memory on Go >= 1.19 #63276

Comments

walles commented Sep 28, 2023 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Notes

thanm commented Sep 28, 2023

thanm commented Sep 28, 2023

walles commented Sep 28, 2023 • edited

mknyszek commented Oct 4, 2023

mknyszek commented Oct 4, 2023

walles commented Oct 4, 2023 • edited

aclements commented Oct 10, 2023

walles commented Oct 10, 2023

nakulbajaj commented Oct 11, 2023

cuonglm commented Oct 12, 2023

thanm commented Oct 12, 2023

jgbooks commented Dec 4, 2023

What did you do?

What did you see instead?

runtime: `-race` leaks memory on Go >= 1.19 #63276

runtime: `-race` leaks memory on Go >= 1.19 #63276

walles commented Sep 28, 2023 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

walles commented Sep 28, 2023 •

edited

walles commented Oct 4, 2023 •

edited