What version of Go are you using (go version)?
go version go1.15.2 darwin/amd64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/Users/adhoc/bin"
GOCACHE="/Users/adhoc/Library/Caches/go-build"
GOENV="/Users/adhoc/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/adhoc/src/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/adhoc/src/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/adhoc/src/spl/almanack/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/p7/jc4qc9n94r3f6ylg0ssh1rq00000gs/T/go-build149709333=/tmp/go-build -gno-record-gcc-switches -fno-common"
But it also happens on an AWS Lambda.
What did you do?
Attempted to get a URL from a certain server with net/http.
Here is an example URL: https://www.inquirer.com/resizer/LUs6Sqe2nxqwxVEe0NHgiLSh6i0=/arc-anglerfish-arc2-prod-pmn/public/GG5L3ZCM5JCOTDVPYIURDF3HHE.jpg
What did you expect to see?
The URL should load an image.
Interestingly, Slack fails to preview this URL, so I think that means they're using Go internally. The image should load in your browser.
What did you see instead?
The server sends an error message to net/http, but not to browsers, curl, or fasthttp.
It's unclear if Go is doing something weird and the server doesn't like it or vice versa. The server appears to be an Akamai CDN. (It's controlled by a friendly third party. I can ask for more details on Monday.) It's possible they've decided that Go is evil and are blocking it somehow, but changing the User-Agent does not help, so however they detect Go, it isn't in the obvious manner of banning a User-Agent or IP.
Here's a reproducer:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"github.com/valyala/fasthttp"
)
func main() {
const exampleURL = "https://www.inquirer.com/resizer/LUs6Sqe2nxqwxVEe0NHgiLSh6i0=/arc-anglerfish-arc2-prod-pmn/public/GG5L3ZCM5JCOTDVPYIURDF3HHE.jpg"
// Fails with net/http
rsp, err := http.Get(exampleURL)
check(err)
blob, err := ioutil.ReadAll(rsp.Body)
check(err)
rsp.Body.Close()
printBlob(blob)
// Succeeds with fasthttp
_, blob, err = fasthttp.Get(nil, exampleURL)
check(err)
printBlob(blob)
}
func check(err error) {
if err != nil {
log.Fatal(err)
}
}
func printBlob(blob []byte) {
allASCII := true
for _, c := range blob {
if c > 127 {
allASCII = false
break
}
}
if allASCII {
fmt.Printf("got: %q\n", blob)
} else {
fmt.Printf("got: %0x", blob[:80])
}
}Output:
got: "<!DOCTYPE html> <html> <body> <h1>Invalid Site</h1> </body> </html>"
got: ffd8ffe000104a46494600010100000100010000ffe20c584943435f50524f46494c4500010100000c484c696e6f021000006d6e74725247422058595a2007ce00020009000600310000616373704d53
In other words, fasthttp gets the JPEG. Curl does as well.
What version of Go are you using (
go version)?go version go1.15.2 darwin/amd64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env)?go envOutputBut it also happens on an AWS Lambda.
What did you do?
Attempted to get a URL from a certain server with net/http.
Here is an example URL: https://www.inquirer.com/resizer/LUs6Sqe2nxqwxVEe0NHgiLSh6i0=/arc-anglerfish-arc2-prod-pmn/public/GG5L3ZCM5JCOTDVPYIURDF3HHE.jpg
What did you expect to see?
The URL should load an image.
Interestingly, Slack fails to preview this URL, so I think that means they're using Go internally. The image should load in your browser.
What did you see instead?
The server sends an error message to net/http, but not to browsers, curl, or fasthttp.
It's unclear if Go is doing something weird and the server doesn't like it or vice versa. The server appears to be an Akamai CDN. (It's controlled by a friendly third party. I can ask for more details on Monday.) It's possible they've decided that Go is evil and are blocking it somehow, but changing the User-Agent does not help, so however they detect Go, it isn't in the obvious manner of banning a User-Agent or IP.
Here's a reproducer:
Output:
In other words, fasthttp gets the JPEG. Curl does as well.