Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: throwOnGCWork (and general builder instability) #29124

Closed
eliasnaur opened this Issue Dec 6, 2018 · 5 comments

Comments

Projects
None yet
4 participants
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Dec 6, 2018

@aclements

This comment has been minimized.

Copy link
Member

aclements commented Dec 6, 2018

linux/amd64: https://build.golang.org/log/2ce33c8fa9981d37f3d5a81e384285c0be6df37a
android/arm: https://build.golang.org/log/34a763d60d9908f39856dd7380948081bef8fff9

Dup of #27993. (Definitely a release-blocker. I'm actively working on this one.)

https://build.golang.org/log/8b9d9127e958601e4a0a3aa71243463d814bd92e
https://build.golang.org/log/8aa5d8614243f1aea5354ba3598e66143de3249c
https://build.golang.org/log/80435e9f9573e9eec360dcb0a69772ca4ccf991a
https://build.golang.org/log/24e04fb21e1383cd002c3a575398f38cf15c7532

Dup of #25519. Only affects debug call injection, so not high-priority. This isn't technically Android-specific, though it seems to happen a lot there, so maybe we should just disable that test on Android.

https://build.golang.org/log/994e945a2a0f4e4ac2efdbabf9c7fedcf402553e

Stuck doing a syscall.Open?

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Dec 6, 2018

https://build.golang.org/log/994e945a2a0f4e4ac2efdbabf9c7fedcf402553e

Stuck doing a syscall.Open?

I believe the vet process used ~100% cpu when I sent it SIGQUIT. So an infinite loop somehow.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Dec 14, 2018

Change https://golang.org/cl/154112 mentions this issue: runtime: fix hangs in TestDebugCall*

gopherbot pushed a commit that referenced this issue Dec 17, 2018

runtime: fix hangs in TestDebugCall*
This fixes a few different issues that led to hangs and general
flakiness in the TestDebugCall* tests.

1. This fixes missing wake-ups in two error paths of the SIGTRAP
   signal handler. If the goroutine was in an unknown state, or if
   there was an unknown debug call status, we currently don't wake the
   injection coordinator. These are terminal states, so this resulted
   in a hang.

2. This adds a retry if the target goroutine is in a transient state
   that prevents us from injecting a call. The most common failure
   mode here is that the target goroutine is in _Grunnable, but this
   was previously masked because it deadlocked the test.

3. Related to 2, this switches the "ready" signal from the target
   goroutine from a blocking channel send to a non-blocking channel
   send. This makes it much less likely that we'll catch this
   goroutine while it's in the runtime performing that send.

4. This increases GOMAXPROCS from 2 to 8 during these tests. With the
   current setting of 2, we can have at most the non-preemptible
   goroutine we're injecting a call in to and the goroutine that's
   trying to make it exit. If anything else comes along, it can
   deadlock. One particular case I observed was in TestDebugCallGC,
   where runtime.GC() returns before the forEachP that prepares
   sweeping on all goroutines has finished. When this happens, the
   forEachP blocks on the non-preemptible loop, which means we now
   have at least three goroutines that need to run.

Fixes #25519.

Updates #29124.

Change-Id: I7bc41dc0b865b7d0bb379cb654f9a1218bc37428
Reviewed-on: https://go-review.googlesource.com/c/154112
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@aclements

This comment has been minimized.

Copy link
Member

aclements commented Dec 18, 2018

Closing as a dup of #27993 and #25519.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.