Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: SIGKILL, SIGABRT appear to cause server termination yet zombie processes with address still bound persist #16599

Closed
odeke-em opened this issue Aug 4, 2016 · 4 comments

Comments

Projects
None yet
5 participants
@odeke-em
Copy link
Member

commented Aug 4, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    • go version devel +50edddb Tue Aug 2 21:31:58 2016 +0000 darwin/amd64
    • go1.6
  2. What operating system and processor architecture are you using (go env)?
    GOARCH="amd64"
    GOBIN="/Users/emmanuelodeke/go/bin"
    GOEXE=""
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOOS="darwin"
    GOPATH="/Users/emmanuelodeke/go"
    GORACE=""
    GOROOT="/Users/emmanuelodeke/go/src/go.googlesource.com/go"
    GOTOOLDIR="/Users/emmanuelodeke/go/src/go.googlesource.com/go/pkg/tool/darwin_amd64"
    CC="clang"
    GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v3/7z434qpx5v3bw0wh8h2myfpw0000gn/T/go-build464848761=/tmp/go-build -gno-record-gcc-switches -fno-common"
    CXX="clang++"
    CGO_ENABLED="1
  3. What did you do?
    Ran a simple server. Sent it a SIGKILL, SIGABRT to the program and it exited. However, the socket stays bound to the address of the "killed" server hence any future reuses of the same port/address are impossible.
$ go run server.go --port-addr 4567 &
[1] 62314
2016/08/04 01:44:40 serving at :4567
$ kill -9 62314
[1]+  Killed: 9               go run server.go --port-addr 4567
$ go run server.go --port-addr 4567
2016/08/04 01:46:36 serving at :4567
2016/08/04 01:46:36 listen tcp :4567: bind: address already in use
exit status 1

You can also try with SIGABRT ( kill -6 ) and that produces the same end result in which you can no longer bind to the address when the command is repeated.

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
https://play.golang.org/p/LorluoHyLW

  1. What did you expect to see?
    I expected SIGKILL and SIGABRT to shutdown not only the server and unbind itself from the port just like SIGINT would. If this is acceptable behavior, I haven't found any documentation for it in net/http.
  2. What did you see instead?
    When https://play.golang.org/p/LorluoHyLW is rerun, it gives such an error
2016/08/04 01:42:06 listen tcp :4567: bind: address already in use

The issue exists on Go1.6 so this isn't a regression.

@odeke-em odeke-em changed the title net/http: SIGKILL, SIGABRT appear to cause program termination yet zombie processes with address still bound persist net/http: SIGKILL, SIGABRT appear to cause server termination yet zombie processes with address still bound persist Aug 4, 2016

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2016

I'm pretty sure this is a generic tcp stack wat. Listening addresses can go
into their version of TIMEWAIT2 until any tcp handshakes in progress have
timed out.

On Thu, 4 Aug 2016, 19:04 Emmanuel T Odeke notifications@github.com wrote:

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

  • go version devel +50edddb Tue Aug 2 21:31:58 2016 +0000 darwin/amd64

    • go1.6
      2.

    What operating system and processor architecture are you using (go env
    )?
    GOARCH="amd64"
    GOBIN="/Users/emmanuelodeke/go/bin"
    GOEXE=""
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOOS="darwin"
    GOPATH="/Users/emmanuelodeke/go"
    GORACE=""
    GOROOT="/Users/emmanuelodeke/go/src/go.googlesource.com/go"
    GOTOOLDIR="/Users/emmanuelodeke/go/src/
    go.googlesource.com/go/pkg/tool/darwin_amd64"
    CC="clang"
    GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics
    -Qunused-arguments -fmessage-length=0
    -fdebug-prefix-map=/var/folders/v3/7z434qpx5v3bw0wh8h2myfpw0000gn/T/go-build464848761=/tmp/go-build
    -gno-record-gcc-switches -fno-common"
    CXX="clang++"
    CGO_ENABLED="1
    3.

    What did you do?
    Ran a simple server. Sent it a SIGKILL, SIGABRT to the program and it
    exited. However, the socket stays bound to the address of the "killed"
    server hence any future reuses of the same port/address are impossible.

  • Steps, run either https://play.golang.org/p/LorluoHyLW or its
    variation https://play.golang.org/p/GvjCDDwaNf, get the program's PID
    and then SIGTERM it.

$ go run server.go --port-addr 4567 &
[1] 62314
2016/08/04 01:44:40 serving at :4567
$ kill -9 62314
[1]+ Killed: 9 go run server.go --port-addr 4567
$ go run server.go --port-addr 4567
2016/08/04 01:46:36 serving at :4567
2016/08/04 01:46:36 listen tcp :4567: bind: address already in useexit status 1

You can also try with SIGABRT ( kill -6 ) and that produces the same end
result in which you can no longer bind to the address when the command is
repeated.

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
https://play.golang.org/p/LorluoHyLW

What did you expect to see?
I expected SIGKILL and SIGABRT to shutdown not only the server and
unbind itself from the port just like SIGINT would. If this is acceptable
behavior, I haven't found any documentation for it in net/http.
2.

What did you see instead?
When https://play.golang.org/p/LorluoHyLW is rerun, it gives such an
error

2016/08/04 01:42:06 listen tcp :4567: bind: address already in use

The issue exists on Go1.6 so this isn't a regression.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#16599, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAcA03y8E1Oqkv25yId9XJWeNkDZTu5ks5qcasAgaJpZM4JcfL5
.

@rhysh

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2016

The pid you see belongs to the go run command, which compiles your program and runs it as a child process. When you send SIGKILL to that pid, the go run process dies but the child (your program) continues to run.

You'll get the behavior you want if you compile your program once with go build -o ./server server.go and run it as ./server --port-addr 4567 &.

@odeke-em

This comment has been minimized.

Copy link
Member Author

commented Aug 4, 2016

Thank you @rhysh for the analysis. Makes sense, and since SIGKILL nor SIGSTOP cannot be stopped, we don't have a way for go run to catch this signal and then kill its children.
For those that need a way to kill these zombies lsof -i :<PORT>

$ lsof -i :4567 | awk '{ print $2 }'
$ # Then find the PID and kill it with -9 e.g
$ lsof -i :4321 | awk '{ print $2 }' | awk 'if ($2 ~ /[0-9 { print $1 }' | while read F;do kill -9 $F;done

Unless @bradfitz or @ianlancetaylor think something else can be done, then this is working as intended.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Aug 4, 2016

Well, definitely not a net/http bug as the subject implies. But as the net/http owner I will happily push "close". :-)

@bradfitz bradfitz closed this Aug 4, 2016

@golang golang locked and limited conversation to collaborators Aug 4, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.