Go version
go version go1.27.0 linux/amd64
Output of go env in your module/workspace:
GOOS=linux
GOARCH=amd64
GOVERSION=go1.27.0
What did you do?
(Also reproduced, identically, with go1.24.7, go1.25.6, go1.26.1, and tip.)
Ran the program below that attempts to shutdown an http2 server while a request is in-flight (in, say, the TLS handshake). The blocking inside the server's GetCertificate callback makes the race deterministic but isn't required to invoke the race.
The cause here is also leading to #59038 but the broader view made this ticket seem worthy.
(This was originally found by upgrading github.com/jmhodges/howsmyssl to Go 1.27.0 and then trying to remove our previously hacked up ServeConn way of doing http/2.)
// Shutdown hangs if an HTTP/2 connection finishes its TLS handshake after
// Shutdown's one-shot GOAWAY sweep, then never opens a stream.
package main
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"fmt"
"io"
"math/big"
"net"
"net/http"
"os"
"time"
)
func cert() tls.Certificate {
key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
tmpl := &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "probe"},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(time.Hour),
KeyUsage: x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
IPAddresses: []net.IP{net.IPv4(127, 0, 0, 1)},
}
der, _ := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
leaf, _ := x509.ParseCertificate(der)
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key, Leaf: leaf}
}
func main() {
crt := cert()
pool := x509.NewCertPool()
pool.AddCert(crt.Leaf)
inHandshake := make(chan struct{}) // closed when the server enters the TLS handshake
releaseHandshake := make(chan struct{}) // closed to let the handshake complete
srv := &http.Server{
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}),
TLSConfig: &tls.Config{
GetCertificate: func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
close(inHandshake)
<-releaseHandshake
return &crt, nil
},
},
}
li, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
panic(err)
}
go srv.ServeTLS(li, "", "")
// Dial and start the TLS handshake; it stalls in GetCertificate.
tcpConn, err := net.Dial("tcp", li.Addr().String())
if err != nil {
panic(err)
}
tlsConn := tls.Client(tcpConn, &tls.Config{
RootCAs: pool,
ServerName: "127.0.0.1",
NextProtos: []string{"h2"},
})
handshakeDone := make(chan error, 1)
go func() { handshakeDone <- tlsConn.Handshake() }()
<-inHandshake // conn is accepted and mid-handshake: not yet registered with the HTTP/2 server
shutdownDone := make(chan error, 1)
shutdownStart := time.Now()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
go func() { shutdownDone <- srv.Shutdown(ctx) }()
// Give Shutdown's OnShutdown hook (the one-shot GOAWAY sweep of the
// http2 server's activeConns set, which is empty right now) time to run.
time.Sleep(200 * time.Millisecond)
// Let the handshake finish. The connection negotiates h2 and registers
// with the HTTP/2 server, which has already done its GOAWAY sweep.
close(releaseHandshake)
if err := <-handshakeDone; err != nil {
panic(err)
}
// Speak enough HTTP/2 to be a healthy idle connection: client preface
// plus an empty SETTINGS frame. Never open a stream.
io.WriteString(tlsConn, "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n")
tlsConn.Write([]byte{0, 0, 0, 0x4, 0, 0, 0, 0, 0}) // SETTINGS, len 0, stream 0
go io.Copy(io.Discard, tlsConn)
err = <-shutdownDone
elapsed := time.Since(shutdownStart).Round(time.Millisecond)
if err != nil {
fmt.Printf("BUG: Shutdown returned %q after %v (hung until context deadline; no GOAWAY sent)\n", err, elapsed)
os.Exit(1)
}
fmt.Printf("Shutdown drained cleanly in %v\n", elapsed)
}What did you see happen?
On every Go version tested (1.24.7, 1.25.6, 1.26.1, 1.27.0, tip 49178db), on linux/amd64 and on 1.26.6 and tip on darwin/arm64, the program always emits:
BUG: Shutdown returned "context deadline exceeded" after 5.001s (hung until context deadline; no GOAWAY sent)
That means that Shutdown always burns exactly its full context. A SIGQUIT dump during the hang shows Shutdown in its closeIdleConns poll loop, one http2serverConn parked in its serve select with no streams, and its peer sitting in an http2ClientConn.readLoop in the client's pool. Nothing is in startGracefulShutdown.
Because the connection is also fully functional, a request sent on it after Shutdown has begun is served successfully. In that success case, the connection does eventually die, because the HTTP/2 server re-checks keep-alive state when the stream closes but the never-used connection is what hangs Shutdown forever.
What did you expect to see?
Shutdown to return once in-flight work is done. In the example program above, it should be within milliseconds, since the conn has no streams. Every conn accepted before the listeners closed should get a GOAWAY regardless of where its handshake was when Shutdown started.
Go version
go version go1.27.0 linux/amd64
Output of
go envin your module/workspace:What did you do?
(Also reproduced, identically, with go1.24.7, go1.25.6, go1.26.1, and tip.)
Ran the program below that attempts to shutdown an http2 server while a request is in-flight (in, say, the TLS handshake). The blocking inside the server's GetCertificate callback makes the race deterministic but isn't required to invoke the race.
The cause here is also leading to #59038 but the broader view made this ticket seem worthy.
(This was originally found by upgrading github.com/jmhodges/howsmyssl to Go 1.27.0 and then trying to remove our previously hacked up ServeConn way of doing http/2.)
What did you see happen?
On every Go version tested (1.24.7, 1.25.6, 1.26.1, 1.27.0, tip 49178db), on linux/amd64 and on 1.26.6 and tip on darwin/arm64, the program always emits:
That means that
Shutdownalways burns exactly its full context. A SIGQUIT dump during the hang shows Shutdown in its closeIdleConns poll loop, one http2serverConn parked in its serve select with no streams, and its peer sitting in an http2ClientConn.readLoop in the client's pool. Nothing is in startGracefulShutdown.Because the connection is also fully functional, a request sent on it after Shutdown has begun is served successfully. In that success case, the connection does eventually die, because the HTTP/2 server re-checks keep-alive state when the stream closes but the never-used connection is what hangs Shutdown forever.
What did you expect to see?
Shutdown to return once in-flight work is done. In the example program above, it should be within milliseconds, since the conn has no streams. Every conn accepted before the listeners closed should get a GOAWAY regardless of where its handshake was when Shutdown started.