Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: go runtime GC panic #45401

Open
eastmoutain opened this issue Apr 6, 2021 · 1 comment
Open

runtime: go runtime GC panic #45401

eastmoutain opened this issue Apr 6, 2021 · 1 comment

Comments

@eastmoutain
Copy link

@eastmoutain eastmoutain commented Apr 6, 2021

What version of Go are you using (go version)?

$ go version

I have tried go1.13.4, go1.13.15, go1.14, go1.14.15

Does this issue reproduce with the latest release?

my code depends on the third part code, go1.15 and above would cause compilation fail

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/thomas/.cache/go-build"
GOENV="/home/thomas/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/thomas/gopath/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/thomas/gopath"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build922420275=/tmp/go-build -gno-record-gcc-switches"

What did you do?

compile my project(a gvisor varietas container) with go, run the container for some hours.

go runtime GC will panic, I have caught two different paincs:
The first one: runtime/mgcwork.go:checknonempty() and checkempty()

func (b *workbuf) checknonempty() {
    if b.nobj == 0 {
        throw("workbuf is empty")
    }
}

and

func (b *workbuf) checkempty() {
    if b.nobj != 0 {
        throw("workbuf is not empty")
    }
}

the second: runtime/mgcsweep.go:sweep()

    nfreed := s.allocCount - nalloc
    if nalloc > s.allocCount {
        print("runtime: nelems=", s.nelems, " nalloc=", nalloc, " previous allocCount=", s.allocCount, " nfreed=", nfreed, "\n")
        throw("sweep increased allocation count")
    }

What did you expect to see?

It should not panic in go runtime library

What did you see instead?

panic frequently in go runtime GC

Trials

I added a FENCE instruction in both push() and pop() function in runtime/lfstack.go

func (head *lfstack) push(node *lfnode) {
   ...
    for {
        old := atomic.Load64((*uint64)(head))
        node.next = old
        atomic.Mfence()  // mfence instruction
        if atomic.Cas64((*uint64)(head), old, new) {
            break
        }
    }
}
func (head *lfstack) pop() unsafe.Pointer {
    for {
        old := atomic.Load64((*uint64)(head))
        if old == 0 {
            return nil
        }
        node := lfstackUnpack(old)
        atomic.Mfence()  // mfence instruction
        next := atomic.Load64(&node.next)
        if atomic.Cas64((*uint64)(head), old, next) {
            return unsafe.Pointer(node)
        }
    }
}

It makes no change to the result, but it runs more time that before.

@ALTree
Copy link
Member

@ALTree ALTree commented Apr 6, 2021

This won't be easy to investigate without a reproducer. Two questions:

  1. Does your program (or any of its dependencies) use cgo? Does compiling and running it with GODEBUG=cgocheck=2 raise any error?
  2. Does compiling your program with -race and then running it print any race detector warnings?

Also you haven't pasted any of the compilation(??)/runtime errors you are getting. I'd suggest doing so.

@ALTree ALTree changed the title go runtime GC panic runtime: go runtime GC panic Apr 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants