Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/tls: TLS 1.3 client fails to connect to some servers #41910

Closed
carlmjohnson opened this issue Oct 10, 2020 · 13 comments
Closed

crypto/tls: TLS 1.3 client fails to connect to some servers #41910

carlmjohnson opened this issue Oct 10, 2020 · 13 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@carlmjohnson
Copy link
Contributor

What version of Go are you using (go version)?

go version go1.15.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

GO111MODULE="on"
GOARCH="amd64"
GOBIN="/Users/adhoc/bin"
GOCACHE="/Users/adhoc/Library/Caches/go-build"
GOENV="/Users/adhoc/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/adhoc/src/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/adhoc/src/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/adhoc/src/spl/almanack/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/p7/jc4qc9n94r3f6ylg0ssh1rq00000gs/T/go-build149709333=/tmp/go-build -gno-record-gcc-switches -fno-common"

But it also happens on an AWS Lambda.

What did you do?

Attempted to get a URL from a certain server with net/http.

Here is an example URL: https://www.inquirer.com/resizer/LUs6Sqe2nxqwxVEe0NHgiLSh6i0=/arc-anglerfish-arc2-prod-pmn/public/GG5L3ZCM5JCOTDVPYIURDF3HHE.jpg

What did you expect to see?

The URL should load an image.

Interestingly, Slack fails to preview this URL, so I think that means they're using Go internally. The image should load in your browser.

What did you see instead?

The server sends an error message to net/http, but not to browsers, curl, or fasthttp.

It's unclear if Go is doing something weird and the server doesn't like it or vice versa. The server appears to be an Akamai CDN. (It's controlled by a friendly third party. I can ask for more details on Monday.) It's possible they've decided that Go is evil and are blocking it somehow, but changing the User-Agent does not help, so however they detect Go, it isn't in the obvious manner of banning a User-Agent or IP.

Here's a reproducer:

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"

	"github.com/valyala/fasthttp"
)

func main() {
	const exampleURL = "https://www.inquirer.com/resizer/LUs6Sqe2nxqwxVEe0NHgiLSh6i0=/arc-anglerfish-arc2-prod-pmn/public/GG5L3ZCM5JCOTDVPYIURDF3HHE.jpg"

	// Fails with net/http
	rsp, err := http.Get(exampleURL)
	check(err)
	blob, err := ioutil.ReadAll(rsp.Body)
	check(err)
	rsp.Body.Close()
	printBlob(blob)

	// Succeeds with fasthttp
	_, blob, err = fasthttp.Get(nil, exampleURL)
	check(err)
	printBlob(blob)
}

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func printBlob(blob []byte) {
	allASCII := true
	for _, c := range blob {
		if c > 127 {
			allASCII = false
			break
		}
	}
	if allASCII {
		fmt.Printf("got: %q\n", blob)
	} else {
		fmt.Printf("got: %0x", blob[:80])
	}
}

Output:

got: "<!DOCTYPE html> <html> <body>  <h1>Invalid Site</h1>  </body> </html>"
got: ffd8ffe000104a46494600010100000100010000ffe20c584943435f50524f46494c4500010100000c484c696e6f021000006d6e74725247422058595a2007ce00020009000600310000616373704d53

In other words, fasthttp gets the JPEG. Curl does as well.

@carlmjohnson
Copy link
Contributor Author

carlmjohnson commented Oct 12, 2020

Minor update: This reproduces with the www.inquirer.com homepage, so it's not route specific.

@carlmjohnson
Copy link
Contributor Author

Double update: Setting tls.Config{ MaxVersion: tls.VersionTLS12 } fixes the issue, so something is going wrong in TLS1.3 negotiation!

@bradfitz bradfitz changed the title net/http: failure to communicate to server that connects with browsers, curl, and fasthttp crypto/tls: TLS 1.3 client fails to connect to some servers Oct 12, 2020
@bradfitz bradfitz added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 12, 2020
@bradfitz bradfitz added this to the Go1.16 milestone Oct 12, 2020
@bradfitz
Copy link
Contributor

/cc @FiloSottile

carlmjohnson added a commit to spotlightpa/almanack that referenced this issue Oct 12, 2020
carlmjohnson added a commit to spotlightpa/almanack that referenced this issue Oct 12, 2020
@carlmjohnson
Copy link
Contributor Author

Curl appears to connect properly with TLS1.3 enabled by OpenSSL.


* Connected to www.inquirer.com (172.232.7.82) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /usr/local/etc/openssl@1.1/cert.pem
  CApath: /usr/local/etc/openssl@1.1/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [29 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4135 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: OU=Domain Control Validated; CN=www.inquirer.com
*  start date: May 15 18:56:42 2019 GMT
*  expire date: May 15 18:56:42 2021 GMT
*  subjectAltName: host "www.inquirer.com" matched cert's "www.inquirer.com"
*  issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x7f849e00a200)
} [5 bytes data]
> GET / HTTP/2
> Host: www.inquirer.com
> user-agent: curl/7.72.0
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [249 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [249 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200 
< content-type: text/html;charset=UTF-8
< server: openresty
< x-akamai-transformed: 9 - 0 pmb=mRUM,2
< cache-control: private, max-age=60
< expires: Mon, 12 Oct 2020 15:45:50 GMT
< date: Mon, 12 Oct 2020 15:44:50 GMT
< set-cookie: arc-country=US; expires=Mon, 12-Oct-2020 15:59:50 GMT; path=/
< set-cookie: arc-zipcode=21201-21203+21205-21206+21209-21218+21223-21224+21229-21231+21233+21235+21239-21241+21250-21252+21263-21264+21270+21273+21275+21278-21282+21284-21285+21287-21290+21297-21298; expires=Mon, 12-Oct-2020 15:59:50 GMT; path=/
< server-timing: cdn-cache; desc=HIT
< server-timing: edge; dur=1
< content-security-policy: upgrade-insecure-requests
< arc-zipcode: 21201-21203+21205-21206+21209-21218+21223-21224+21229-21231+21233+21235+21239-21241+21250-21252+21263-21264+21270+21273+21275+21278-21282+21284-21285+21287-21290+21297-21298
< arc-country: US
< 
{ [2248 bytes data]
(rest omitted)

@carlmjohnson
Copy link
Contributor Author

Is it possible this is related to #40521?

@carlmjohnson
Copy link
Contributor Author

@FiloSottile I was hoping this would be resolved by go 1.15.4, but it looks like no such luck:

$ go1.15.4 run /Users/adhoc/Desktop/reproducer.go
https://www.inquirer.com/resizer/LUs6Sqe2nxqwxVEe0NHgiLSh6i0=/arc-anglerfish-arc2-prod-pmn/public/GG5L3ZCM5JCOTDVPYIURDF3HHE.jpg
got: "<!DOCTYPE html> <html> <body>  <h1>Invalid Site</h1>  </body> </html>"

@networkimprov
Copy link

cc @ianlancetaylor - possible release blocker?

@carlmjohnson
Copy link
Contributor Author

This has been broken in prod for the last month, so I don't think it's a release issue per se?

@carlmjohnson
Copy link
Contributor Author

Could still be faulty on Akamai's side, but it's strange that it works in Curl with TLSL1.3 turned on.

@FiloSottile FiloSottile self-assigned this Nov 14, 2020
@odeke-em
Copy link
Member

Howdy @carlmjohnson, you mentioned that this issue has been around for a while per #41910 (comment). We’ve been swamped for Go1.16 development cycles, yet we are almost getting Go1.16 out the door. I shall move this to Go1.17, and when it gets looked at, we shall backport it.

@odeke-em odeke-em modified the milestones: Go1.16, Go1.17 Dec 25, 2020
@carlmjohnson
Copy link
Contributor Author

Yeah, it started Oct 10, presumably because of a change on Akamai’s side. I’m planning to upgrade to 1.16 right away so I’m fine with a fix coming mid-cycle in 1.16 or whenever.

@networkimprov
Copy link

See also #43250 re TLS 1.3.

@carlmjohnson
Copy link
Contributor Author

This no longer reproduces with Go 1.15 or 1.16. I assume that whatever was going wrong has been fixed on the inquirer.com side, presumably by Akamai.

@golang golang locked and limited conversation to collaborators Feb 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

6 participants