net/http: http.Get fails with "connection reset by peer" after update to Go 1.23 but works flawlessly up to 1.22.8

[Roemer]

Go version

go version go1.23.2 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY='*.mycompany.com'
GONOSUMDB='*.mycompany.com'
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE='*.mycompany.com'
GOPROXY='https://repo.mycompany.com/artifactory/api/go/go'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/builds/dev-containers/dev-container-features/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3761320793=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I have a very simple repro app that just downloads a file via http.get
The code for this is:

package main

import (
	"fmt"
	"io"
	"net/http"
)

const versionsIndexUrl = "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages"

func main() {
	versionFileContent, err := Download(versionsIndexUrl)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Successfully downloaded %d byte(s)\n", len(versionFileContent))
}

func Download(downloadUrl string) ([]byte, error) {
	resp, err := http.Get(downloadUrl)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("failed to download file '%s'. Status code: %d", downloadUrl, resp.StatusCode)
	}
	bodyBytes, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("failed to read body: %w", err)
	}
	return bodyBytes, nil
}

This code is compiled and runs in a container in a GitLab pipeline. The http_proxy and https_proxy variables are set as a proxy is needed for the CI to connect to the outside world.

What did you see happen?

The code above works great if the go version in go.mod ist at max 1.22.8. As soon as I set it to 1.23.0 or higher, the download always fails with

panic: Get "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages": read tcp 10.127.1.6:55264-><proxy-address>: read: connection reset by peer

This is 100% reproducible in our environment. 1.22.8 (or lower) always works, starting from 1.23.0 it always fails without changing anything else.

What did you expect to see?

In other environments (also on the lokal workstation with also needs a proxy), this works regardless of the go version but on all GitLab runners, it always fails and I am pretty much at a loss on what I could do more to fix this.

[gabyhelp]

Related Issues and Documentation

• net/http: provide better error message when proxy is misconfigured #29809
• net: TCPConn.ReadFrom hangs when io.Reader is TCPConn or UnixConn, Linux kernel < 5.1 #63795 (closed)
• net/http: permanently broken connection with error "read: connection reset by peer" when response body is not closed #36700
• net/http: Error fetching URL: Get "http://www.google.com": proxyconnect tcp: dial tcp: lookup http: no such host #61436 (closed)
• proxy.golang.org, cmd/go, net/http: http2 PROTOCOL_ERRORS from the proxy #69544 (closed)
• net/http: `readLoopPeekFailLocked: <nil>` from client when server sends response immediately #31259
• net/http: http.Get: connection reset by peer on macOS #23158 (closed)
• net/http: Get request hangs until timeout, but responds immediately over curl #61823 (closed)
• vhost doesn't work correctly in the http protocol in net/http package #58601 (closed)
• net/http: ServeContent()/ServeFile() doesn't return expected response when WriteTimeout happens #43822 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

CC @neild @golang/security

Thanks for providing a small test case. Unfortunately, I can't recreate the problem. In all cases when I run your program I get

Successfully downloaded 11438 byte(s)

I've tried with various language versions in go.mod and various compiler versions.

[Roemer]

Thanks for your answer. For us, it works on most machines but fails on all 30+ ci agents and I have no more idea what could cause this issue. Is there some way to debug this further?

[seankhliao]

I think you need to look at your proxy. It's the one that's terminating the connections.

[FiloSottile]

Try GODEBUG=tlskyber=0 (and in general when debugging a breakage between releases, try switching the updated GODEBUGs one by one). If it fixes it, you have a proxy affected by https://tldr.fail or otherwise incompatible with post-quantum key shares in a non-compliant way.

[Roemer]

I think you need to look at your proxy. It's the one that's terminating the connections.

But when I compile with go 1.22.8 (or lower) it works with the same proxy, only when using go 1.23.0 or higher and everything else is exactly the same, it fails.

[Roemer]

Try GODEBUG=tlskyber=0 (and in general when debugging a breakage between releases, try switching the updated GODEBUGs one by one). If it fixes it, you have a proxy affected by https://tldr.fail or otherwise incompatible with post-quantum key shares in a non-compliant way.

Thank your for your hint. I now went thru all the GODEBUG Values, my go env (which is used for building and then immediately running the application) shows GODEBUG='httpservecontentkeepheaders=1,x509keypairleaf=0,tls3des=1,jstmpllitinterp=1,x509negativeserial=1,tlskyber=0,winreadlinkvolume=0,winsymlink=0,asynctimerchan=1' but I still get the same error with go 1.23 but not with 1.22. Is there anything else I could try?

[Roemer]

Nevermind. I set it in the wrong layer (my app runs nested in one more container that was missing the GODEBUG).
I've added godebug tlskyber=0 to my go.mod and now it works correctly so it is indeed a problem with one of our corporate proxies. Thank you for all the support (and new information about GODEBUG). I'll close this as it is not an issue with Go itself.

[FiloSottile]

Glad we could help! If you can share any details about the proxy, sometimes we can backchannel to the vendors, or at least document it for the next person.

[Roemer]

I'll see if I can get that information and post back here and also in tldr.fail.

[Roemer]

Sorry for the late reply. But here are the data:
The software ist Broadcom ProxySG
The issue was fixed by the vendor with version 7.3.22.1