net/http: http.Get fails with "connection reset by peer" after update to Go 1.23 but works flawlessly up to 1.22.8

Go version

go version go1.23.2 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY='*.mycompany.com'
GONOSUMDB='*.mycompany.com'
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE='*.mycompany.com'
GOPROXY='https://repo.mycompany.com/artifactory/api/go/go'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/builds/dev-containers/dev-container-features/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3761320793=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I have a very simple repro app that just downloads a file via http.get
The code for this is:

package main

import (
	"fmt"
	"io"
	"net/http"
)

const versionsIndexUrl = "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages"

func main() {
	versionFileContent, err := Download(versionsIndexUrl)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Successfully downloaded %d byte(s)\n", len(versionFileContent))
}

func Download(downloadUrl string) ([]byte, error) {
	resp, err := http.Get(downloadUrl)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("failed to download file '%s'. Status code: %d", downloadUrl, resp.StatusCode)
	}
	bodyBytes, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("failed to read body: %w", err)
	}
	return bodyBytes, nil
}

This code is compiled and runs in a container in a GitLab pipeline. The http_proxy and https_proxy variables are set as a proxy is needed for the CI to connect to the outside world.

What did you see happen?

The code above works great if the go version in go.mod ist at max 1.22.8. As soon as I set it to 1.23.0 or higher, the download always fails with

panic: Get "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages": read tcp 10.127.1.6:55264-><proxy-address>: read: connection reset by peer

This is 100% reproducible in our environment. 1.22.8 (or lower) always works, starting from 1.23.0 it always fails without changing anything else.

What did you expect to see?

In other environments (also on the lokal workstation with also needs a proxy), this works regardless of the go version but on all GitLab runners, it always fails and I am pretty much at a loss on what I could do more to fix this.

Related Issues and Documentation

net/http: provide better error message when proxy is misconfigured #29809
net: TCPConn.ReadFrom hangs when io.Reader is TCPConn or UnixConn, Linux kernel < 5.1 #63795 (closed)
net/http: permanently broken connection with error "read: connection reset by peer" when response body is not closed #36700
net/http: Error fetching URL: Get "http://www.google.com": proxyconnect tcp: dial tcp: lookup http: no such host #61436 (closed)
proxy.golang.org, cmd/go, net/http: http2 PROTOCOL_ERRORS from the proxy #69544 (closed)
net/http: `readLoopPeekFailLocked: <nil>` from client when server sends response immediately #31259
net/http: http.Get: connection reset by peer on macOS #23158 (closed)
net/http: Get request hangs until timeout, but responds immediately over curl #61823 (closed)
vhost doesn't work correctly in the http protocol in net/http package #58601 (closed)
net/http: ServeContent()/ServeFile() doesn't return expected response when WriteTimeout happens #43822 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

@neild

CC @neild @golang/security

Thanks for providing a small test case. Unfortunately, I can't recreate the problem. In all cases when I run your program I get

Successfully downloaded 11438 byte(s)

I've tried with various language versions in go.mod and various compiler versions.

Thanks for your answer. For us, it works on most machines but fails on all 30+ ci agents and I have no more idea what could cause this issue. Is there some way to debug this further?

I think you need to look at your proxy. It's the one that's terminating the connections.

Try GODEBUG=tlskyber=0 (and in general when debugging a breakage between releases, try switching the updated GODEBUGs one by one). If it fixes it, you have a proxy affected by https://tldr.fail or otherwise incompatible with post-quantum key shares in a non-compliant way.

I think you need to look at your proxy. It's the one that's terminating the connections.

But when I compile with go 1.22.8 (or lower) it works with the same proxy, only when using go 1.23.0 or higher and everything else is exactly the same, it fails.

Try GODEBUG=tlskyber=0 (and in general when debugging a breakage between releases, try switching the updated GODEBUGs one by one). If it fixes it, you have a proxy affected by https://tldr.fail or otherwise incompatible with post-quantum key shares in a non-compliant way.

Thank your for your hint. I now went thru all the GODEBUG Values, my go env (which is used for building and then immediately running the application) shows GODEBUG='httpservecontentkeepheaders=1,x509keypairleaf=0,tls3des=1,jstmpllitinterp=1,x509negativeserial=1,tlskyber=0,winreadlinkvolume=0,winsymlink=0,asynctimerchan=1' but I still get the same error with go 1.23 but not with 1.22. Is there anything else I could try?

Nevermind. I set it in the wrong layer (my app runs nested in one more container that was missing the GODEBUG).
I've added godebug tlskyber=0 to my go.mod and now it works correctly so it is indeed a problem with one of our corporate proxies. Thank you for all the support (and new information about GODEBUG). I'll close this as it is not an issue with Go itself.

Glad we could help! If you can share any details about the proxy, sometimes we can backchannel to the vendors, or at least document it for the next person.

I'll see if I can get that information and post back here and also in tldr.fail.

Sorry for the late reply. But here are the data:
The software ist Broadcom ProxySG
The issue was fixed by the vendor with version 7.3.22.1

I have same problem

go1.20.3
go1.22.8
go1.22.12

When I attempt to fetch 500 items at once using http.Get（url very long）,
the request succeeds with the http scheme but fails with the https scheme, resulting in a "read: connection reset by peer" error.

&tls.Config{InsecureSkipVerify: true} invalid
no this problem in postman
no this problem when fetch 50 items at once(using http.Get)

fix(listenbrainz): disable post-quantum TLS to avoid connection reset…

feat: add httpclient factory to disable post-quantum TLS by default

Navigation Menu

Repository navigation

net/http: http.Get fails with "connection reset by peer" after update to Go 1.23 but works flawlessly up to 1.22.8 #70139

Go version

Output of `go env` in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

net/http: http.Get fails with "connection reset by peer" after update to Go 1.23 but works flawlessly up to 1.22.8 #70139

Description

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Activity

gabyhelp commented on Oct 31, 2024

ianlancetaylor commented on Oct 31, 2024

Roemer commented on Oct 31, 2024

seankhliao commented on Oct 31, 2024

FiloSottile commented on Oct 31, 2024

Roemer commented on Oct 31, 2024

Roemer commented on Nov 1, 2024

Roemer commented on Nov 1, 2024

FiloSottile commented on Nov 1, 2024

Roemer commented on Nov 1, 2024

Roemer commented on Jan 27, 2025

fanybook commented on Apr 15, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Output of `go env` in your module/workspace: