Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: SIGPROF arriving on a foreign thread before any cgo call is made will crash the process #9456

Closed
minux opened this issue Dec 27, 2014 · 4 comments
Assignees
Milestone

Comments

@minux
Copy link
Member

@minux minux commented Dec 27, 2014

runtime.badsignal will try to deliver the signal received by foreign threads to sigqueue using the cgo callback mechanism, however, cgocallback will not work if the process has not made any cgo call yet.

A simple reproduction program:

package main

/*
#include <signal.h>
#include <pthread.h>
volatile int x;
static void *thread(void *p) {
    (void)p;
    while (x == 0)
        ;
    pthread_kill(pthread_self(), SIGPROF);
    return NULL;
}
__attribute__((constructor)) void test() {
    pthread_t tid;
    pthread_create(&tid, 0, thread, NULL);
}
*/
import "C"
import "sync/atomic"
import "unsafe"

func main() {
    atomic.StoreInt32((*int32)(unsafe.Pointer(&C.x)), 1)
    select {}
}

This example is contrived, but imagine the thread is created by an external library linked in, and the user is profiling Go code. See https://groups.google.com/forum/#!topic/golang-nuts/SMhWSUsfPag for a real-world example (the symptom is the same, and disabling the cgocallback line in function runtime.badsignal fixes it, but I'm still not sure about how the program could create the extra thread during benchmarking without any cgocall. The problem could also be in other part of runtime.)

Tentatively labeled 1.4.1, as there is no meaningful workaround for 1.4.

@minux minux added the repo-main label Dec 27, 2014
@minux minux self-assigned this Dec 27, 2014
@minux minux added this to the Go1.4.1 milestone Dec 27, 2014
@minux minux changed the title runtime: SIGPROF might arrive on a foreign thread before any cgo call is made will crash the process runtime: SIGPROF arriving on a foreign thread before any cgo call is made will crash the process Dec 27, 2014
@kortschak
Copy link
Contributor

@kortschak kortschak commented Dec 27, 2014

Sorry, I was not able to construct a reproducer that does not depend on being in tests. However, this code reproduces the failure if it is used to replace the covariancematrix_test.go file in github.com/gonum/stat.

Run using go test -cpuprofile foo.prof.

(github.com/gonum/blas/cblas is installed using OpenBLAS).

@minux
Copy link
Member Author

@minux minux commented Dec 27, 2014

Thanks. I've verified that the essence of the problem is the same as the
contrived example I created. Will fix and add a test to runtime.

OpenBLAS has a global constructor (https://github.com/xianyi/OpenBLAS/blob/develop/driver/others/memory.c#L1291)
that creates worker threads correspond to the each of processors in the system.

The worker threads are periodically waken up to check if there is work for
them (https://github.com/xianyi/OpenBLAS/blob/develop/driver/others/blas_server.c#L268),
so it's possible that during the long data preparation steps (where the Go
code has not made any cgo call yet), SIGPROF comes to one of the waken worker thread
of OpenBLAS and trigger the problem.

@minux minux closed this in 5da9c8c Dec 31, 2014
minux added a commit that referenced this issue Jan 14, 2015
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes #9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
@btracey
Copy link
Contributor

@btracey btracey commented May 27, 2015

I'm seeing this periodically in Go 1.4.1. It's especially confusing because it happens seemingly randomly, and the most recent code doesn't import any cgo code, although I do have "export CGO_LDFLAGS="-L/Users/brendan/software/OpenBLAS -lopenblas" in my .bash_profile. I'd like to be more helpful, but I don't know how to make a consistent reproducer. I think I've only seen it when using "go run" if that has any relevance.

@minux
Copy link
Member Author

@minux minux commented May 27, 2015

@golang golang locked and limited conversation to collaborators Jun 25, 2016
wheatman added a commit to wheatman/go-akaros that referenced this issue Jun 25, 2018
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman added a commit to wheatman/go-akaros that referenced this issue Jun 26, 2018
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman added a commit to wheatman/go-akaros that referenced this issue Jul 9, 2018
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman added a commit to wheatman/go-akaros that referenced this issue Jul 20, 2018
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman added a commit to wheatman/go-akaros that referenced this issue Jul 30, 2018
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.