net/http: *http.Transport is never collected

[powerman]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?

go version go1.6.2 linux/amd64

1. What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/powerman/gocode"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="x86_64-pc-linux-gnu-gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="x86_64-pc-linux-gnu-g++"
CGO_ENABLED="1"

1. What did you do?

package main

import (
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "runtime"
    "time"
)

func main() {
    for {
        c := &http.Client{
            Transport: &http.Transport{
                ExpectContinueTimeout: 1 * time.Second, // A
            },
        }
        resp, err := c.Get("http://google.com")
        if resp != nil {
            io.Copy(ioutil.Discard, resp.Body) // B
            resp.Body.Close()                  // B
        }
        fmt.Printf("resp=%t, err=%v\n", resp != nil, err)
        time.Sleep(time.Second)
        c.Transport.(*http.Transport).CloseIdleConnections() // C
        runtime.SetFinalizer(c, func(interface{}) { fmt.Println("--- http.Client GONE") })
        runtime.SetFinalizer(c.Transport, func(interface{}) { fmt.Println("--- http.Transport GONE") })
        runtime.GC()
    }
}

1. What did you expect to see?

resp=true, err=<nil>
--- http.Client GONE
resp=true, err=<nil>
--- http.Client GONE
--- http.Transport GONE
resp=true, err=<nil>
--- http.Client GONE
--- http.Transport GONE
resp=true, err=<nil>
--- http.Client GONE
--- http.Transport GONE
resp=true, err=<nil>
^Csignal: interrupt

1. What did you see instead?

Provided program works correctly and gives output shown above. But if we'll comment any of lines marked // A or // B or // C then http.Transport won't be garbage-collected anymore and output became:

resp=true, err=<nil>
--- http.Client GONE
resp=true, err=<nil>
--- http.Client GONE
resp=true, err=<nil>
--- http.Client GONE
resp=true, err=<nil>
--- http.Client GONE
resp=true, err=<nil>
^Csignal: interrupt

I know it's recommended to share same http.Client for all application, but I'm developing load-testing application which simulate thousands of independent clients and thus require own simulated client to use own http.Client and own http.Transport objects to behave more like real independent clients.

[adg]

I'm not sure what you expect to happen here. All the code is working exactly as documented. In particular, you must always Close the response body when using http.Client.

You can turn off keepalives entirely in your custom http.Transport if you like. There are also configuration knobs on Client that you may wish to turn.

[powerman]

I expect to have this code working correctly without needs to manually call CloseIdleConnections().

I don't see any issues with needs to call Body.Close(), but relationship between this issue and ExpectContinueTimeout looks weird for me.

[adg]

I expect to have this code working correctly without needs to manually call CloseIdleConnections().

Unless you disable Keep-Alive connections in the Transport, then you should expect the Transport to maintain a pool of idle connections. If you don't close those connections or re-use the Transport, you will leak Transports.

relationship between this issue and ExpectContinueTimeout looks weird for me.

Hmm, well ExpectContinueTimeout disables http2 in the client, so maybe that's related. You're saying that you see Transports escaping even if you don't call CloseIdleConnections?

[powerman]

Unless you disable Keep-Alive connections in the Transport, then you should expect the Transport to maintain a pool of idle connections.

Sure. But also I expect Transport object to clean up after itself when it doesn't used anymore - i.e. when all references to that object is gone.

If you don't close those connections or re-use the Transport, you will leak Transports.

Is this mean only correct way to create own Transport is to manually make sure it will be garbage-collected when it doesn't needed anymore? For example, in simple case when this Transport isn't shared between multiple clients use something like this (untested)?

tr := &http.Transport{…}
client := &http.Client{Transport: tr}
runtime.SetFinalizer(client, func(c *http.Client){ c.Transport.CloseIdleConnections() })

If this is the case, then it probably should be noted in documentation with warning about leaking sockets and goroutines otherwise.

Hmm, well ExpectContinueTimeout disables http2 in the client, so maybe that's related. You're saying that you see Transports escaping even if you don't call CloseIdleConnections?

Without defining ExpectContinueTimeout it leaks no matter is I call CloseIdleConnections() or not.

[adg]

Sure. But also I expect Transport object to clean up after itself when it doesn't used anymore - i.e. when all references to that object is gone.

But that's not how things usually work in Go. I would expect to call some kind of Close method.

I would not recommend using SetFinalizer (at all, its inclusion in Go was a mistake IMO), but rather just calling CloseIdleConnections when you are done. Alternately, you can disable KeepAlives in the Transport and then you needn't worry about CloseIdleConnections.

The interaction between connection pooling (I assume) and ExpectContinueTimeout does surprise me. There might be something here. Leaving assigned to @bradfitz in case he has idaes.

[powerman]

I would not recommend using SetFinalizer

Can you please explain why? Or give a link if this was already explained somewhere? How it's differ from defer (in spirit) in this use case?

But that's not how things usually work in Go. I would expect to call some kind of Close method.

I agree, but I believe this is somewhat corner/subtle case and it's worth to be mentioned in http package documentation.

[adg]

Can you please explain why? Or give a link if this was already explained somewhere? How it's differ from defer (in spirit) in this use case?

There are no guarantees about if/when a finalizer will be executed. A finalizer may be silently replaced by a subsequent call to SetFinalizer, with no warning from the compiler or runtime. In short, they are not reliable enough to be broadly useful.

On the other hand, defer is well-defined and its behavior is deterministic. You know that all deferred functions were executed if a function call returns.