-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: a Windows application launched via Steam sometimes freezes #71242
Comments
This was originally reported at hajimehoshi/ebiten#3181 by @corfe83.
|
I could reproduce this even with a console application (without package main
import (
"log"
"runtime"
"runtime/debug"
"time"
)
func main() {
go func() {
for {
_ = make([]byte, 256*1024)
time.Sleep(time.Millisecond)
}
}()
for {
time.Sleep(time.Second)
var gcStats debug.GCStats
debug.ReadGCStats(&gcStats)
log.Printf("LastGC: %s, NumGC: %d, PauseTotal: %s", gcStats.LastGC, gcStats.NumGC, gcStats.PauseTotal)
}
}
A freeze occurs between 00:19:21 and 00:19:37 |
CC @golang/runtime |
Does this issue reproduce with Go 1.22? |
Yes, I could reproduce this with Go 1.22.9. |
Do you have a stack trace for each thread, for example from a debugger? My guess based on the information provided would be that the threads are stuck in some GC-related thing (spinning up GC mark workers? that only happens on the first GC if Alternatively, the threads are going to sleep for STW, which is also based on a |
I'm afraid I'm not familiar with WinDbg. How can I get the stack traces as a file? I'll try tomorrow. Also I can insert printlns in the runtime by -overlay. I'll try this later too. |
I dumped stack traces by x64dbg: Stack traces by x64dbg
|
I also dumped goroutine stacks by delve Goroutine Dumps
Go sourcepackage main
import (
"log"
"os"
"runtime"
"runtime/debug"
"time"
)
func main() {
/*for _, env := range os.Environ() {
log.Println(env)
}*/
/*if runtime.GOOS == "windows" && os.Getenv("SteamClientLaunch") == "1" {
runtime.GOMAXPROCS(max(1, min(2, runtime.NumCPU()-1)))
}*/
log.Println(runtime.Version())
log.Printf("PID: %d", os.Getpid())
go func() {
for {
_ = make([]byte, 256*1024)
time.Sleep(time.Millisecond)
}
}()
for {
time.Sleep(time.Second)
var gcStats debug.GCStats
debug.ReadGCStats(&gcStats)
log.Printf("LastGC: %s, NumGC: %d, PauseTotal: %s", gcStats.LastGC, gcStats.NumGC, gcStats.PauseTotal)
}
} |
Adding println affects the behavior apparently, so I am not sure this was a correct observation, but I saw goroutines got stuck at the stdcall2 with _WaitForSingleObject.
STW was invoked on GOMAXPROCS in my case, but this was not related to the freeze. Freezing happens regardless of STW. |
package main
import (
"log"
"os"
"runtime"
"runtime/debug"
"time"
)
func main() {
debug.SetGCPercent(-1)
go func() {
for {
println("runtime.GC start")
runtime.GC()
println("runtime.GC end")
time.Sleep(5 * time.Millisecond)
}
}()
log.Println(runtime.Version())
log.Printf("PID: %d", os.Getpid())
for i := 0; i < 5; i++ {
go func() {
for {
_ = make([]byte, 256*1024)
time.Sleep(time.Millisecond)
}
}()
}
for {
time.Sleep(time.Second)
var gcStats debug.GCStats
debug.ReadGCStats(&gcStats)
log.Printf("LastGC: %s, NumGC: %d, PauseTotal: %s", gcStats.LastGC, gcStats.NumGC, gcStats.PauseTotal)
}
} The above program manually calls |
Thanks for the details. In the x64dbg stack traces, it looks like every thread is blocked, which seems suspicious. This thread seems particularly suspicious to me:
@qmuntal is it normal for releasing a mutex to block? Or maybe this isn't actually blocked and we got lucky. @hajimehoshi could you collect another set of stack traces to see if they look similar? It's unfortunate that we didn't get symbol names for the Go code itself. I tried building #71242 (comment) with Go 1.23.2, but the addresses don't even come close to matching up. @hajimehoshi could you try manually symbolizing some of these? I think From the goroutine stacks, a GC is clearly running. A few goroutines are waiting to help out with GC assists (which is a bit odd). Two of them we can't see:
These are perhaps stuck in GC mark termination, maybe in here? |
Go programpackage main
import (
"log"
"os"
"runtime"
"runtime/debug"
"time"
)
func main() {
for range 5 {
go func() {
for {
_ = make([]byte, 256*1024)
time.Sleep(time.Millisecond)
}
}()
}
log.Printf("Version: %s", runtime.Version())
log.Printf("PID: %d", os.Getpid())
for {
time.Sleep(time.Second)
var gcStats debug.GCStats
debug.ReadGCStats(&gcStats)
log.Printf("LastGC: %s, NumGC: %d, PauseTotal: %s", gcStats.LastGC, gcStats.NumGC, gcStats.PauseTotal)
}
} Stack traces by x64dbg
I'll try addr2line soon. |
Unfortunately I failed to resolve the addresses: Get-Content .\stacktraces.txt | go tool addr2line .\innovation2007_windows_amd64.exe The result was
Am I missing something? |
By the way, in order to reproduce this, it is possible to do this for free:
|
Thanks, looks like that run is also sitting in ReleaseMutex.
addr2line requires that each line contains only an address, so you'll need to edit the stack trace file to remove everything else. |
Hmm, addr2line still didn't resolve the file like this:
|
I am not sure this would be helpful, but I objdumped the binary and searched the binaries in the suspicous stack traces. I realized the addresses are 'shifted'. Stack traces with ReleaseMutex (I newly dumped this):
|
Here is the resolved symbols in the stack trace below
|
Apologies for the bad instructions, I don't use addr2line much. That stack is really interesting. On Windows, asynchronous preemption uses the Windows thread suspend/resume APIs. It seems that this thread is getting stuck in ResumeThread. Maybe the Steam overlay messes with this API somehow? |
@hajimehoshi Can you reproduce with |
I couldn't reproduce this with |
By the way, is there a compile option to specify Of course, it would be the best that the runtime is fixed to suppress this issue. There seem multiple situations where
EDIT: |
I think we should fix that, though I'm glad you found another workaround with |
Yeah, so would it be possible for
I mean, I want to disable async preemption when I build my games for Steam Windows. So I would specify |
I think this is reasonable, please do file an issue.
That said, I don't think this is the final resolution. Steam seems to be messing with our process in a way that breaks certain Windows API calls. I think the next step is to report this bug to Valve, as current evidence points to a bug in whatever Steam is doing. |
Thanks! #71283
I agree. I've already reported the current situation https://steamcommunity.com/discussions/forum/0/595138100650327297/. I'll try to make a simple C program that invokes I appreciate all of your quick responses! |
Apologies, the |
Go version
go version go1.23.2 windows/amd64
Output of
go env
in your module/workspace:What did you do?
Compile this Go program to an execute file (with
-ldflags="-H=windowsgui"
)EDIT: I could minized the case further. See #71242 (comment)
Replace an exe file in a Steam game with the compiled exe file, and run it via Steam client.
What did you see happen?
The application sometimes freezes for more than 10 seconds. For example, I saw this log
You can see a freeze happens between 22:23:57 and 22:24:18.
What did you expect to see?
The application doesn't freeze.
The text was updated successfully, but these errors were encountered: