net/http: HTTP2: not-consistent invalid error when server closes connection, possible race condition

[vblz]

What version of Go are you using (go version)?

$ go version
go version go1.25.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes, including tip-20251128-alpine3.22 docker image

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
AR='ar'
CC='cc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='c++'
GCCGO='gccgo'
GO111MODULE='on'
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/v/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/v/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/yc/2rq4vj6x19q6bq783_rn8mdw0000gn/T/go-build3209620510=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/v/projects/http2poc/go.mod'
GOMODCACHE='/Users/v/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/v/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/opt/go/libexec'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/v/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/opt/go/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25.4'
GOWORK=''
PKG_CONFIG='pkg-config'
GOROOT/bin/go version: go version go1.25.4 darwin/arm64
GOROOT/bin/go tool compile -V: compile version go1.25.4
uname -v: Darwin Kernel Version 25.1.0: Mon Oct 20 19:33:00 PDT 2025; root:xnu-12377.41.6~2/RELEASE_ARM64_T6020
ProductName:		macOS
ProductVersion:		26.1
BuildVersion:		25B78
lldb --version: lldb-1703.0.234.3
Apple Swift version 6.2.1 (swiftlang-6.2.1.4.8 clang-1700.4.4.1)

What did you do?

I have a TLS server that requires a client certificate and has HTTP/2 enabled.
The client does not send a certificate / send invalid certifcate, so the TLS handshake is expected to fail with tls: certificate required.

The following minimal code to reproduce the issue and program to run it several time and check returned errors:

func reproduceTLSClientError() string {
	server := httptest.NewUnstartedServer(nil)
	server.TLS = &tls.Config{
		ClientAuth: tls.RequireAndVerifyClientCert,
	}
	server.EnableHTTP2 = true
	server.StartTLS()
	defer server.Close()

	_, err := server.Client().Get(server.URL)
	return err.Error()
}

Full code

go.dev/play, but it does not allow to run more than a couple of test runs

package main

import (
	"crypto/tls"
	"fmt"
	"net/http/httptest"
	"strings"
)

const times = 10000

// This function is where the problem manifests: it sometimes returns
// different error strings for the same situation.
func reproduceTLSClientError() string {
	server := httptest.NewUnstartedServer(nil)
	server.TLS = &tls.Config{
		ClientAuth: tls.RequireAndVerifyClientCert,
	}
	server.EnableHTTP2 = true
	server.StartTLS()
	defer server.Close()

	_, err := server.Client().Get(server.URL)
	return err.Error()
}

var errors = map[string]int{
	"remote error: tls: certificate required": 0,
	"client conn could not be established":    0,
	"write: broken pipe":                      0,
	"write: connection reset by peer":         0,
}

func countError(errTxt string) {
	for k := range errors {
		if strings.HasSuffix(errTxt, k) {
			errors[k]++
			return
		}
	}
	errors[errTxt]++
}

func main() {
	for i := 0; i < times; i++ {
		countError(reproduceTLSClientError())
	}
	fmt.Printf("%+v\n", errors)
}

What did you expect to see?

Since the server is consistently rejecting the connection with tls.RequireAndVerifyClientCert and sending a TLS alert “Certificate Required”, I expected every failed Get/RoundTrip to return the TLS error: remote error: tls: certificate required
This is exactly what happens when HTTP/2 is disabled and the client uses HTTP/1.1.

What did you see instead?

With HTTP/2 enabled on the server, over many runs I see a mix of error messages for the same scenario:

• remote error: tls: certificate required (most of the time)
• client conn could not be established
• write: broken pipe
• write: connection reset by peer

For example, after 10,000 iterations one run printed:

map[
client conn could not be established:56
remote error: tls: certificate required:9933
write: broken pipe:10
write: connection reset by peer:1]

Whereas with HTTP/2 disabled I consistently get:

map[
client conn could not be established:0
remote error: tls: certificate required:10000
write: broken pipe:0
write: connection reset by peer:0]

So HTTP/2 introduces non-determinism in which error is returned for the same “server requires client certificate, client doesn’t provide a valid one” situation.

For debugging I also ran with HTTP/2 debug logging enabled.
Last error in the debug logs is Transport readFrame error on conn 0x14000082700: (*tls.permanentError) remote error: tls: certificate required, but the client still got write tcp 127.0.0.1:54649->127.0.0.1:54642: write: broken pipe

GODEBUG=http2debug=2 output

2025/12/02 20:09:49 http2: Transport failed to get client conn for 127.0.0.1:54638: http2: no cached connection was available
2025/12/02 20:09:49 http2: Transport creating client conn 0x1400017ea80 to 127.0.0.1:54638
2025/12/02 20:09:49 http: TLS handshake error from 127.0.0.1:54641: tls: client didn't provide a certificate
2025/12/02 20:09:49 http2: Framer 0x14000170380: wrote SETTINGS len=24, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_FRAME_SIZE=16384, MAX_HEADER_LIST_SIZE=10485760
2025/12/02 20:09:49 http2: Framer 0x14000170380: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2025/12/02 20:09:49 http2: Transport encoding header ":authority" = "127.0.0.1:54638"
2025/12/02 20:09:49 http2: Transport encoding header ":method" = "GET"
2025/12/02 20:09:49 http2: Transport encoding header ":path" = "/"
2025/12/02 20:09:49 http2: Transport encoding header ":scheme" = "https"
2025/12/02 20:09:49 http2: Transport encoding header "accept-encoding" = "gzip"
2025/12/02 20:09:49 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2025/12/02 20:09:49 http2: Transport readFrame error on conn 0x1400017ea80: (*tls.permanentError) remote error: tls: certificate required
2025/12/02 20:09:49 http2: Framer 0x14000170380: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=36
2025/12/02 20:09:49 http2: Framer 0x14000170380: wrote RST_STREAM stream=1 len=4 ErrCode=CANCEL
2025/12/02 20:09:49 http2: Framer 0x14000170380: wrote PING len=8 ping="\xa0\xbc8\xf1\xe3!\f\xb7"
2025/12/02 20:09:49 RoundTrip failure: remote error: tls: certificate required
2025/12/02 20:09:50 http2: Transport failed to get client conn for 127.0.0.1:54642: http2: no cached connection was available
2025/12/02 20:09:50 http2: Transport creating client conn 0x14000082700 to 127.0.0.1:54642
2025/12/02 20:09:50 http2: Framer 0x140000f00e0: wrote SETTINGS len=24, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_FRAME_SIZE=16384, MAX_HEADER_LIST_SIZE=10485760
2025/12/02 20:09:50 http2: Framer 0x140000f00e0: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2025/12/02 20:09:50 http: TLS handshake error from 127.0.0.1:54649: tls: client didn't provide a certificate
2025/12/02 20:09:50 http2: Transport encoding header ":authority" = "127.0.0.1:54642"
2025/12/02 20:09:50 http2: Transport encoding header ":method" = "GET"
2025/12/02 20:09:50 http2: Transport encoding header ":path" = "/"
2025/12/02 20:09:50 http2: Transport encoding header ":scheme" = "https"
2025/12/02 20:09:50 http2: Transport encoding header "accept-encoding" = "gzip"
2025/12/02 20:09:50 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2025/12/02 20:09:50 http2: Framer 0x140000f00e0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=36
2025/12/02 20:09:50 http2: Framer 0x140000f00e0: wrote RST_STREAM stream=1 len=4 ErrCode=CANCEL
2025/12/02 20:09:50 http2: Framer 0x140000f00e0: wrote PING len=8 ping="i\xb4y\x1a\xb8\xb1\xe8M"
2025/12/02 20:09:50 http2: Transport readFrame error on conn 0x14000082700: (*tls.permanentError) remote error: tls: certificate required
2025/12/02 20:09:50 RoundTrip failure: write tcp 127.0.0.1:54649->127.0.0.1:54642: write: broken pipe
--- FAIL: Test_POC (0.51s)
    poc_test.go:24: got error: Get "https://127.0.0.1:54642": write tcp 127.0.0.1:54649->127.0.0.1:54642: write: broken pipe

Additional observations (packet capture)

In a packet capture of a failing HTTP/2 case that ends with write: broken pipe, I see the following sequence on the TCP stream:

1. TLS 1.3 handshake completes up to the point where the server sends Alert (Level: Fatal, Description: Certificate Required).
2. Immediately after that, the server sends a FIN, ACK.
3. After the fatal TLS alert, the client still sends an HTTP/2 preface: Magic, SETTINGS[0], WINDOW_UPDATE[0].
4. Then there is a TLSv1.3 Alert (Level: Warning, Description: Close Notify)and finally a TCP RST.

This suggests that:

• The server correctly rejects the client without a certificate and sends a TLS fatal alert “Certificate Required”.
• However, the HTTP/2 transport on the client side continues to write HTTP/2 frames (preface, SETTINGS, possibly HEADERS) on a connection that has already been closed (or is in the process of being closed) due to the TLS alert.
• Depending on the exact timing, the client either surfaces the TLS error (remote error: tls: certificate required) or a lower‑level OS error (write: broken pipe, connection reset by peer), or errors like client conn could not be established.

From the API user’s point of view this is surprising, especially since the TLS stack clearly knows the underlying cause ((*tls.permanentError) remote error: tls: certificate required in the HTTP/2 debug logs), and HTTP/1.1 consistently returns that error.

It looks like there may be a race between the TLS handshake failure/alert and the HTTP/2 transport starting to send its preface and headers, so that sometimes the application gets the TLS error and sometimes it only sees the subsequent write error on the already-closed TCP connection.

[gabyhelp]

Related Issues

• net/http: client sends RST_STREAM after receiving END_STREAM #76239
• net/http: http.Client starts returning errors when tls.Config is shared #73911 (closed)
• net/http: data race in HTTP/2 body close #61596 (closed)
• net/http: Errors from persistConn.readLoop not returned to client still writing a request #62447
• x/net/http2: #41784 (closed)
• net/http: transport caches permanently broken persistent connections if write error happens during h2 handshake #34978 (closed)
• crypto/tls: misleading "remote error: tls: certificate required" error when client cert CA not in server's accepted CAs list #75009
• x/net/http2: implementation issue #48111
• x/net/http2: Spec conformance fails #38647 (closed)
• x/net/http2: transport race in closing stream while processing DATA frame, kills connection. #47076

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

CC @neild.