Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: hangs in "found a crash, minimizing..." #46634

Open
klauspost opened this issue Jun 7, 2021 · 2 comments
Open

cmd/go: hangs in "found a crash, minimizing..." #46634

klauspost opened this issue Jun 7, 2021 · 2 comments

Comments

@klauspost
Copy link
Contributor

@klauspost klauspost commented Jun 7, 2021

What version of Go are you using (go version)?

λ gotip version
go version devel go1.17-5542c10fbf Fri Jun 4 03:07:33 2021 +0000 windows/amd64

What operating system and processor architecture are you using (go env)?

go env Output
λ gotip env                                                                                                                                  
set GO111MODULE=                                                                                                                             
set GOARCH=amd64                                                                                                                             
set GOBIN=                                                                                                                                   
set GOCACHE=d:\temp\gocache                                                                                                                  
set GOENV=C:\Users\klaus\AppData\Roaming\go\env                                                                                              
set GOEXE=.exe                                                                                                                               
set GOFLAGS=                                                                                                                                 
set GOHOSTARCH=amd64                                                                                                                         
set GOHOSTOS=windows                                                                                                                         
set GOINSECURE=                                                                                                                              
set GOMODCACHE=e:\gopath\pkg\mod                                                                                                             
set GONOPROXY=                                                                                                                               
set GONOSUMDB=                                                                                                                               
set GOOS=windows                                                                                                                             
set GOPATH=e:\gopath                                                                                                                         
set GOPRIVATE=                                                                                                                               
set GOPROXY=https://goproxy.io                                                                                                               
set GOROOT=C:\Users\klaus\sdk\gotip                                                                                                          
set GOSUMDB=sum.golang.org                                                                                                                   
set GOTMPDIR=                                                                                                                                
set GOTOOLDIR=C:\Users\klaus\sdk\gotip\pkg\tool\windows_amd64                                                                                
set GOVCS=                                                                                                                                   
set GOVERSION=devel go1.17-5542c10fbf Fri Jun 4 03:07:33 2021 +0000                                                                          
set GCCGO=gccgo                                                                                                                              
set AR=ar                                                                                                                                    
set CC=gcc                                                                                                                                   
set CXX=g++                                                                                                                                  
set CGO_ENABLED=1                                                                                                                            
set GOMOD=NUL                                                                                                                                
set CGO_CFLAGS=-g -O2                                                                                                                        
set CGO_CPPFLAGS=                                                                                                                            
set CGO_CXXFLAGS=-g -O2                                                                                                                      
set CGO_FFLAGS=-g -O2                                                                                                                        
set CGO_LDFLAGS=-g -O2                                                                                                                       
set PKG_CONFIG=pkg-config                                                                                                                    
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=d:\temp\wintemp\go-build1674875180=/tmp/go-build -gno-record-gcc-switches

What did you do?

Ran a fuzz test, that ended up eating up all memory, causing OOMs to happen. This is Windows, so applications just crash and aren't killed by the OS.

Likely one or more of the workers crashed.

What did you expect to see?

The controller printed found a crash, minimizing..., but never progressed from there.

Left it for about 15 minutes before I killed it.

What did you see instead?

Fuzzer not getting stuck.

@katiehockman katiehockman changed the title [dev.fuzz] Hangs in "found a crash, minimizing..." [dev.fuzz] cmd/go: hangs in "found a crash, minimizing..." Jun 7, 2021
@katiehockman katiehockman added this to the Unreleased milestone Jun 7, 2021
@rolandshoemaker
Copy link
Member

@rolandshoemaker rolandshoemaker commented Jun 18, 2021

This is a complicated problem, since Go doesn't provide any real way to deal with GC backpressure (see #29696 and the associated issues for background). Unless the worker process actually dies, killing the pipe used for RPC communication, the worker/coordinator has no way of knowing that there is a memory problem.

A simple-ish solution would be to just add a timeout to some/all of the RPC calls, so we can just bail out if the worker is taking longer than expected (and we assume it is dead). One question is, if the RPC times out, do we consider that a crash? (i.e. if a fuzz call times out, we copy the input and consider that a crasher).

Once we allow for continuous fuzzing (continuing when a crasher is find, rather than immediately exiting), we would need a way of marking a worker process dead, and ideally restarting it so we don't slowly run out of workers.

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Jul 14, 2021

This is likely going to be a release blocker for Go 1.18.

@katiehockman katiehockman removed this from the Unreleased milestone Sep 14, 2021
@katiehockman katiehockman added this to the Backlog milestone Sep 14, 2021
@katiehockman katiehockman removed this from the Backlog milestone Sep 14, 2021
@katiehockman katiehockman added this to the Go1.18 milestone Sep 14, 2021
@rsc rsc changed the title [dev.fuzz] cmd/go: hangs in "found a crash, minimizing..." cmd/go: hangs in "found a crash, minimizing..." Sep 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants