Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: SIGSEGV at high GOMAXPROCS in init #38474

Closed
prattmic opened this issue Apr 15, 2020 · 6 comments
Closed

runtime: SIGSEGV at high GOMAXPROCS in init #38474

prattmic opened this issue Apr 15, 2020 · 6 comments

Comments

@prattmic
Copy link
Member

@prattmic prattmic commented Apr 15, 2020

What version of Go are you using (go version)?

$ ../bin/go version        
go version devel +435b9dd1a1 Wed Apr 15 15:30:46 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

This does not reproduce on go 1.14.2.

What operating system and processor architecture are you using (go env)?

go env Output
$ ../bin/go env    
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/mpratt/.cache/go-build"
GOENV="/usr/local/google/home/mpratt/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/usr/local/google/home/mpratt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/mpratt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/google/home/mpratt/src/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/google/home/mpratt/src/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/usr/local/google/home/mpratt/src/go/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build240555977=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ ./make.bash
$ GOMAXPROCS=1024 ../bin/go help

What did you expect to see?

Help output

What did you see instead?

$ GOMAXPROCS=1024 ../bin/go help
zsh: segmentation fault  GOMAXPROCS=1024 ../bin/go help

Full backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x000000000040f093 in runtime.profilealloc (mp=0xeb33a0 <runtime.m0>, x=0xc000932000, size=10240) at /usr/local/google/home/mpratt/src/go/src/runtime/malloc.go:1229
1229            return uintptr(fastexprand(MemProfileRate))
(gdb) bt
#0  0x000000000040f093 in runtime.profilealloc (mp=0xeb33a0 <runtime.m0>, x=0xc000932000, size=10240) at /usr/local/google/home/mpratt/src/go/src/runtime/malloc.go:1229
#1  0x000000000040e810 in runtime.mallocgc (size=10240, typ=0xa1cb80, needzero=true, ~r3=<optimized out>) at /usr/local/google/home/mpratt/src/go/src/runtime/malloc.go:1130
#2  0x000000000040ef58 in runtime.newobject (typ=0xa1cb80, ~r1=<optimized out>) at /usr/local/google/home/mpratt/src/go/src/runtime/malloc.go:1179
#3  0x000000000043fbf2 in runtime.procresize (nprocs=1024, ~r1=<optimized out>) at /usr/local/google/home/mpratt/src/go/src/runtime/proc.go:4279
#4  0x00000000004372db in runtime.schedinit () at /usr/local/google/home/mpratt/src/go/src/runtime/proc.go:584
#5  0x0000000000463e45 in runtime.rt0_go () at /usr/local/google/home/mpratt/src/go/src/runtime/asm_amd64.s:214
#6  0x0000000000000002 in ?? ()
#7  0x00007fffffffdbf8 in ?? ()
#8  0x0000000000000002 in ?? ()
#9  0x00007fffffffdbf8 in ?? ()
#10 0x0000000000000000 in ?? ()
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 15, 2020

Yeah, don't start doing profiling allocations before the scheduler is initialized. The crash is because mp.p is 0 in profilealloc, and that is so because it has not yet been set by the call from procresize to acquirep, which happens after the point of the crash above.

@prattmic
Copy link
Member Author

@prattmic prattmic commented Apr 15, 2020

For completeness:

$ git bisect bad                          
3093959ee10f5c28211094e784c954f6a304b9c9 is the first bad commit                                                                              
commit 3093959ee10f5c28211094e784c954f6a304b9c9                  
Author: Ian Lance Taylor <iant@golang.org>                                                                                                    
Date:   Mon Nov 4 14:25:22 2019 -0800                          
                                                                       
    runtime: remove mcache field from m                                
                                                                                                                                              
    Having an mcache field in both m and p is confusing, so remove it from m.
    Always use mcache field from p. Use new variable mcache0 during bootstrap.                                                                
                                                                                                                                              
    Change-Id: If2cba9f8bb131d911d512b61fd883a86cf62cc98 
    Reviewed-on: https://go-review.googlesource.com/c/go/+/205239
    Run-TryBot: Ian Lance Taylor <iant@golang.org>
    TryBot-Result: Gobot Gobot <gobot@golang.org>
    Reviewed-by: Austin Clements <austin@google.com>            
                                                                       
 src/runtime/malloc.go   | 19 +++++++++++++++----                                                                                             
 src/runtime/mgcsweep.go |  2 +-                                                                                                                                                                                                                                                            
 src/runtime/mheap.go    | 29 ++++++++++++++++++++---------                                                                                   
 src/runtime/proc.go     | 29 ++++++++++++-----------------                                                                                   
 src/runtime/runtime1.go |  5 -----                                                                                                           
 src/runtime/runtime2.go |  1 -                           
 src/runtime/stack.go    | 12 ++++++------           
 7 files changed, 54 insertions(+), 43 deletions(-)              
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 15, 2020

Interesting, it's true that before that change profilealloc updated mp.mcache but after that change it updates mp.p.ptr().mcache. So only after that change did profilealloc require that mp.p be initialized.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 15, 2020

I'll send a CL.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 15, 2020

Change https://golang.org/cl/228420 mentions this issue: runtime: use mcache0 if no P in profilealloc

@gopherbot gopherbot closed this in 7ea40f6 Apr 17, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 17, 2020

Change https://golang.org/cl/228786 mentions this issue: runtime: allow proflock above globalAlloc.mutex

gopherbot pushed a commit that referenced this issue Apr 21, 2020
During schedinit, these may occur in:

mProf_Malloc
  stkbucket
    newBucket
      persistentalloc
        persistentalloc1

mProf_Malloc
  setprofilebucket
    fixalloc.alloc
      persistentalloc
        persistentalloc1

These seem to be legitimate lock orderings.

Additionally, mheap.speciallock had a defined rank, but it was never
actually used. That is fixed now.

Updates #38474

Change-Id: I0f6e981852eac66dafb72159f426476509620a65
Reviewed-on: https://go-review.googlesource.com/c/go/+/228786
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dan Scales <danscales@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.