net/http: File descriptor leakage

[magisterquis]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8.1 openbsd/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="openbsd"
GOOS="openbsd"
GOPATH="/home/stuart/go"
GORACE=""
GOROOT="/home/stuart/.go/1.8.1"
GOTOOLDIR="/home/stuart/.go/1.8.1/pkg/tool/openbsd_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -fmessage-length=0"
CXX="g++"
CGO_ENABLED="0"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

Build Host:

GOOS=linux go build -o f.linux.amd64 f.go

Execution Host:

./f.linux.amd64

Code: http://play.golang.org/p/gKiRi7x6V5

What did you expect to see?

HTTP requests, and file descriptors deallocated.

What did you see instead?

File descriptors are not released. lsof -Pnp <pid> indicates sockets hang around once they're no longer in a CLOSE_WAIT state.

$ lsof -Pnp 19380
COMMAND     PID            USER   FD      TYPE   DEVICE SIZE/OFF     NODE NAME
f.linux.a 19380 stuart  cwd       DIR    253,3     4096 75497664 /home/stuart.mcmurray
f.linux.a 19380 stuart  rtd       DIR    253,3     4096      128 /
f.linux.a 19380 stuart  txt       REG    253,3  5523864 75509093 /home/stuart.mcmurray/f.linux.amd64
f.linux.a 19380 stuart    0u      CHR    136,0      0t0        3 /dev/pts/0
f.linux.a 19380 stuart    1u      CHR    136,0      0t0        3 /dev/pts/0
f.linux.a 19380 stuart    2u      CHR    136,0      0t0        3 /dev/pts/0
f.linux.a 19380 stuart    3u     sock      0,6      0t0 48758872 protocol: TCP
f.linux.a 19380 stuart    4r      CHR      1,9      0t0     1033 /dev/urandom
f.linux.a 19380 stuart    5u  a_inode      0,9        0     5719 [eventpoll]
f.linux.a 19380 stuart    6u     sock      0,6      0t0 48753565 protocol: TCP
f.linux.a 19380 stuart    7u     sock      0,6      0t0 48753578 protocol: TCP
f.linux.a 19380 stuart    8u     sock      0,6      0t0 48757923 protocol: TCP
f.linux.a 19380 stuart    9u     sock      0,6      0t0 48757941 protocol: TCP
f.linux.a 19380 stuart   10u     sock      0,6      0t0 48757954 protocol: TCP
f.linux.a 19380 stuart   11u     IPv4 48757964      0t0      TCP 10.40.15.176:49254->52.23.67.15:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   12u     sock      0,6      0t0 48758954 protocol: TCP
f.linux.a 19380 stuart   13u     sock      0,6      0t0 48758964 protocol: TCP
f.linux.a 19380 stuart   14u     sock      0,6      0t0 48757994 protocol: TCP
f.linux.a 19380 stuart   15u     sock      0,6      0t0 48759828 protocol: TCP
f.linux.a 19380 stuart   16u     IPv4 48756119      0t0      TCP 10.40.15.176:49259->52.23.67.15:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   17u     IPv4 48756125      0t0      TCP 10.40.15.176:52080->34.197.186.182:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   18u     IPv4 48758042      0t0      TCP 10.40.15.176:49261->52.23.67.15:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   19u     IPv4 48759033      0t0      TCP 10.40.15.176:38018->107.23.175.12:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   20u     IPv4 48758057      0t0      TCP 10.40.15.176:37774->34.192.54.247:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   21u     IPv4 48756183      0t0      TCP 10.40.15.176:38020->107.23.175.12:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   22u     IPv4 48759062      0t0      TCP 10.40.15.176:49265->52.23.67.15:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   23u     IPv4 48758075      0t0      TCP 10.40.15.176:37777->34.192.54.247:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   24u     IPv4 48759073      0t0      TCP 10.40.15.176:37778->34.192.54.247:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   25u     IPv4 48758086      0t0      TCP 10.40.15.176:38024->107.23.175.12:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   26u     IPv4 48759088      0t0      TCP 10.40.15.176:52089->34.197.186.182:443 (CLOSE_WAIT)
f.linux.a 19380 stuart   27u     IPv4 48758103      0t0      TCP 10.40.15.176:38026->107.23.175.12:443 (CLOSE_WAIT)
-snip-

It seems net/http doesn't close file descriptors when it's done with them. The fd ulimit is set to 1024.

[bradfitz]

You don't close the Body in this case:

        if http.StatusOK != res.StatusCode {
                log.Printf("Ping Status: %v", res.Status)
                return
        }

I can't reproduce this on a server that's only returning 200 OKs

After a couple minutes:

$ lsof -p 15475
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
repro   15475 bradfitz  cwd    DIR    8,1    16384 531000 /home/bradfitz
repro   15475 bradfitz  rtd    DIR    8,1     4096      2 /
repro   15475 bradfitz  txt    REG    8,1  4057184 655685 /tmp/go-build186024032/command-line-arguments/_obj/exe/repro
repro   15475 bradfitz  mem    REG    8,1  1738176 540080 /lib/x86_64-linux-gnu/libc-2.19.so
repro   15475 bradfitz  mem    REG    8,1   137384 532910 /lib/x86_64-linux-gnu/libpthread-2.19.so
repro   15475 bradfitz  mem    REG    8,1   140928 533432 /lib/x86_64-linux-gnu/ld-2.19.so
repro   15475 bradfitz    0u   CHR  136,2      0t0      5 /dev/pts/2
repro   15475 bradfitz    1u   CHR  136,2      0t0      5 /dev/pts/2
repro   15475 bradfitz    2u   CHR  136,2      0t0      5 /dev/pts/2
repro   15475 bradfitz    4u  0000    0,9        0   7634 anon_inode
repro   15475 bradfitz    5r   CHR    1,9      0t0   1033 /dev/urandom

This is well tested in the net/http package. Be sure to always close your response Body.

Closing, but let me know if I'm missing something.

[magisterquis]

That was it. My mistake.

I appreciate it.