Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: memory of finished goroutines not freed #65864

Open
ohhfeng opened this issue Feb 22, 2024 · 4 comments
Open

runtime: memory of finished goroutines not freed #65864

ohhfeng opened this issue Feb 22, 2024 · 4 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@ohhfeng
Copy link

ohhfeng commented Feb 22, 2024

Go version

go version go1.19 windows/amd64

Output of go env in your module/workspace:

Does not depend env

What did you do?

I have observed that after creating a large number of Goroutines, the memory does not seem to be released back to the system. Specifically, I suspect there might be an issue with the sched.gFree.noStack in proc.go, where the Goroutine structures are not being properly cleared, leading to potential memory leak.

What did you see happen?

The memory used by the finished Goroutines does not appear to be released, even when the Goroutines are no longer in use. This suggests a potential memory leak in the runtime's Goroutine management.

What did you expect to see?

I expected that after Goroutines finish their execution and are no longer needed, the memory they occupied would be returned to the system.

@septemhill
Copy link

I have observed that after creating a large number of Goroutines, the memory does not seem to be released back to the system.

In short period of time ? If so, it might related to #9869 ?

@seankhliao
Copy link
Member

1.19 isn't a supported version anymore (currently 1.22 and 1.21).
how are you measuring memory?
do you have a reproducer?

@seankhliao seankhliao changed the title Potential Memory Leak in Runtime with Goroutines not being Freed runtime: memory of finished goroutines not freed Feb 22, 2024
@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 22, 2024
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Feb 22, 2024
@ohhfeng
Copy link
Author

ohhfeng commented Feb 24, 2024

package main

import (
	"fmt"
	"net/http"
	_ "net/http/pprof"
	"runtime"
	"sync"
	"time"
)

func printMemStats() {
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Println("`````````````````````````````````````````````")
	fmt.Println("Memory Allocation:")
	fmt.Println("Alloc:", m.Alloc)
	fmt.Println("TotalAlloc:", m.TotalAlloc)
	fmt.Println("HeapAlloc:", m.HeapAlloc)
	fmt.Println("HeapSys:", m.HeapSys)
	fmt.Println("HeapIdle:", m.HeapIdle)
	fmt.Println("HeapInuse:", m.HeapInuse)
	fmt.Println("HeapReleased:", m.HeapReleased)
	fmt.Println("HeapObjects:", m.HeapObjects)
	fmt.Println("runtime.NumGoroutine():", runtime.NumGoroutine())
}

func main() {
	go func() {
		fmt.Println(http.ListenAndServe("localhost:6060", nil))
	}()

	num := 1000000
	printMemStats()
	wg := sync.WaitGroup{}
	wg.Add(num)
	for i := 0; i < num; i++ {
		go func() {
			defer wg.Done()
			time.Sleep(time.Second)
		}()
	}
	wg.Wait()
	printMemStats()
	time.Sleep(time.Second * 5)
	runtime.GC()
	for {
		time.Sleep(time.Second * 10)
		printMemStats()
	}
}


The memory usage after creating one million goroutines is as follows.
image

The memory usage after a period of time is as follows.And then it never goes down again.
image

I think that the goroutines in sched.gFree.noStack have not been properly removed.

@randall77
Copy link
Contributor

It is true that we never free a g structure once allocated. They get put in the allgs list and live there forevermore.
Freeing the gs could be done, I suppose, but would make unsynchronized access to allgs trickier because it would need to handle the shrinking case. And maybe there are other parts of the runtime that assume gs are never freed.

Which is to say, maybe we could do this, but it isn't trivial.

Is there a situation where this is actually causing a problem? Most servers, at least in steady state, would reuse these gs effectively.

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Mar 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Status: No status
Development

No branches or pull requests

5 participants