Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: Go program "just terminates" with exit 0 #15873

Closed
purpleidea opened this issue May 29, 2016 · 11 comments

Comments

Projects
None yet
7 participants
@purpleidea
Copy link

commented May 29, 2016

While hacking on a medium sized golang project, I've found that certain things seem to cause my program to terminate, but without any trace or exit code. I'm assuming this is a golang bug, since there's no clear reason why the program would exit without reason.

I've got a binary that I compiled that experiences this, and I can reproduce it on other machines.

I'm running golang:
go version go1.5.4 linux/amd64
On Fedora 23.

If this is a known issue someone can point me to, I'd appreciate it, otherwise please let me know what I can run to debug this.

Here is the output of my most recent terminate:

./mgmt run
06:12:56 main.go:65: This is: mgmt, version: 0.0.3-48-ga89c332-dirty
06:12:56 main.go:66: Main: Start: 1464516776777766845
06:12:56 main.go:120: Main: Seeds: No seeds specified!
06:12:56 etcd.go:344: Etcd: Bootstrapping...
06:12:56 etcd.go:1089: Trace: Etcd: nominateCallback()
06:12:56 asm_amd64.s:438: Trace: Etcd: nominateCallback(): Finished!
$ echo $?
0

Of particular note is that the last line contains the contents of a log.Printf command, which is some debugging text I put there, but it doesn't happen at asm_amd64.s:438 (I've enabled logging of file and line numbers)

Thanks for your time, and I'm happy to post the binary if anyone is interested.

@minux

This comment has been minimized.

Copy link
Member

commented May 29, 2016

@purpleidea

This comment has been minimized.

Copy link
Author

commented May 29, 2016

@minux Of course I agree with you, otherwise I wouldn't have reported this. There's nothing I'm calling that calls Exit, and the etcd authors said there is nothing that does it either. When it happened an earlier time I figured it was my fault, but when I got the log with the assembly code, I realized it was probably a golang bug. There's no reason I would see that asm line AFAIK.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented May 29, 2016

It's pretty hard to diagnose this without the source of your program. Can
you please provide a minimal example that demonstrates the problem.

On Mon, May 30, 2016 at 7:53 AM, James notifications@github.com wrote:

@minux https://github.com/minux Of course I agree with you, otherwise I
wouldn't have reported this. There's nothing I'm calling that calls Exit,
and the etcd authors said there is nothing that does it either. When it
happened an earlier time I figured it was my fault, but when I got the log
with the assembly code, I realized it was probably a golang bug. There's no
reason I would see that asm line AFAIK.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#15873 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcAwTXRE8fhcX4Qy9-vtllIMZrltwHks5qGgrtgaJpZM4IpQiM
.

@purpleidea

This comment has been minimized.

Copy link
Author

commented May 29, 2016

On Sun, May 29, 2016 at 6:11 PM, Dave Cheney notifications@github.com
wrote:

It's pretty hard to diagnose this without the source of your program. Can
you please provide a minimal example that demonstrates the problem.

Once I can distill the code down, I will. I suppose the binary won't help?
Does the ASM message tell you anything?

@minux

This comment has been minimized.

Copy link
Member

commented May 30, 2016

@purpleidea

This comment has been minimized.

Copy link
Author

commented May 30, 2016

This seems to be related to issues where a null pointer would get used and in the protobuf golang package. So in theory if your code has zero bugs (mine doesn't) then you won't hit this. So something is either failing in a strange way, or something relating to protobufs is suppressing the trace dump on null pointer.

@ianlancetaylor ianlancetaylor changed the title Golang program "just terminates" with exit 0 runtime: Go program "just terminates" with exit 0 May 30, 2016

@ianlancetaylor ianlancetaylor added this to the Go1.8 milestone May 30, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 30, 2016

We need a test case or the backtrace that minux suggested.

@awishformore

This comment has been minimized.

Copy link

commented Aug 24, 2016

I am running into the exact same problem in one of our microservices. The only thing we were able to observe is that some network polling fails before it happens, but that might or might not be related.

I searched the entire code base (including vendored libraries), and the only place where it calls os.Exit(0) is an the end of the main function. Before it exits, there are a few lines of log output which are never written.

It simply shouldn't be possible for the application to exit with code 0 and have no output at all, other than normal operation. This is blocking our deployment big time right now.

It happens on a timescale of several hours to several days, so it's really hard to track anything down.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 24, 2016

We need a test case.

@awishformore

This comment has been minimized.

Copy link

commented Aug 26, 2016

Our issue was, in fact, unrelated to this. The shutdown messages were just hidden in front of hundreds of warnings in the log, as execution was terminating and closing everything down. The reason for the shutdown was an unexpected signal other than SIGINT, SIGTERM and SIGHUP being issued, and us deciding to catch all signals. After we stopped catching the signal, the software stopped "crashing". We still don't know what issued the signal, since we found nothing in OS logs.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 8, 2016

Timeout. Closing this bug. Without details or a repro, we can't help.

@bradfitz bradfitz closed this Sep 8, 2016

@golang golang locked and limited conversation to collaborators Sep 9, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.