Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: proxy returns 503 response but http client returns error #30560

Open
nnathan opened this Issue Mar 4, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@nnathan
Copy link

nnathan commented Mar 4, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build551463569=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I'm running a squid proxy server listening on 127.0.0.1:3128 which is running as squid user with an iptables rule that drops outgoing port 53 packets emitted by the squid user. This forces squid to respond to all proxied requests using a dns name with a 503 response carrying a header X-Squid-Error: ERR_DNS_FAIL 0.

I'm also writing a go program that performs a healthcheck when squid returns a response with X-Squid-Error header by making a proxied GET request to a URL using the default http client. The proxying is enabled using the http_proxy and https_proxy environment variables.

When querying a HTTP URL the http client returns the 503 response from squid with the X-Squid-Error header.

What did you expect to see?

When querying a HTTPS URL the http client should return the 503 response from squid.

What did you see instead?

When querying a HTTPS URL the http client returns an error and nil response, with the error returning "Get https://www.google.com: Service Unavailable".

The same query to https://www.google.com using curl with the https_proxy environment variable set returns the squid 503 response.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Mar 4, 2019

@nnathan can you please provide a sample program that someone else can use to reproduce the problem you are having. Thank you.

@nnathan

This comment has been minimized.

Copy link
Author

nnathan commented Mar 4, 2019

Sure.

The program kind of relies on squid running that responds with a 503 and returns an X-Squid-Error header.

  1. The squid.conf is:
#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8

acl SSL_ports port 443
acl SSL_ports port 2443
acl SSL_ports port 5222
acl SSL_ports port 8243
acl SSL_ports port 8280
acl SSL_ports port 9443
acl SSL_ports port 9445
acl SSL_ports port 9763
acl SSL_ports port 22
acl SSL_ports port 25
acl SSL_ports port 4120
acl SSL_ports port 4119
acl SSL_ports port 4122
acl Safe_ports port 4120
acl Safe_ports port 4119
acl Safe_ports port 4122
acl Safe_ports port 80
acl Safe_ports port 21
acl Safe_ports port 22
acl Safe_ports port 25
acl Safe_ports port 443
acl Safe_ports port 70
acl Safe_ports port 210
acl Safe_ports port 1025-65535
acl Safe_ports port 280
acl Safe_ports port 488
acl Safe_ports port 591
acl Safe_ports port 777
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/spool/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
  1. To force DNS for squid to fail, I run: iptables -A OUTPUT -m owner --uid-owner squid -p udp --dport 53 -j DROP

  2. Now the following go program runs a webserver on port 33128 which will serve a 200 response if it didn't detect a 503/X-Squid-Error response, otherwise it will serve a 500 response with a diagnostic.

main.go:

package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"
)

var listenPort = flag.Int("listen_port", 33128, "health check webserver listen port")

var healthCheckURL = flag.String("health_check_url", "http://www.google.com", "URL to perform health check on")

func main() {
	flag.Parse()

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain; charset=utf-8")

		resp, err := http.Get(*healthCheckURL)

		if err != nil {
			log.Printf("error: %v", err)
			w.WriteHeader(http.StatusInternalServerError)
			w.Write([]byte(fmt.Sprintf("error: %v\n", err)))
			return
		}

		if resp.StatusCode == 503 {
			resp.Body.Close()

			squidErr := resp.Header.Get("X-Squid-Error")

			if squidErr != "" {
				log.Printf("squid failure error detected, X-Squid-Error: %s", squidErr)
				w.WriteHeader(http.StatusInternalServerError)
				w.Write([]byte(fmt.Sprintf("error detected: X-Squid-Error: %s\n", squidErr)))
				return
			}
		}

		w.WriteHeader(http.StatusOK)
		w.Write([]byte("Everying OK - did not detect any squid errors\n"))
	})

	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *listenPort), nil))
}

To run: http_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 go run main.go -health_check_url https://www.google.com

(change the https above to http to observe the different behaviours, whereby https will return a error on the GET and http will return a proper response)

With http://www.google.com:

# curl -v http://127.0.0.1:33128
* About to connect() to 127.0.0.1 port 33128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 33128 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:33128
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Date: Mon, 04 Mar 2019 05:54:11 GMT
< Content-Length: 46
< 
error detected: X-Squid-Error: ERR_DNS_FAIL 0
* Connection #0 to host 127.0.0.1 left intact

With https://www.google.com:

# curl -v http://127.0.0.1:33128
* About to connect() to 127.0.0.1 port 33128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 33128 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:33128
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Date: Mon, 04 Mar 2019 05:54:43 GMT
< Content-Length: 55
< 
error: Get https://www.google.com: Service Unavailable
* Connection #0 to host 127.0.0.1 left intact

And with https://www.google.com/ bypassing the healthcheck and using the proxy directly:

# https_proxy=http://127.0.0.1:3128/ http_proxy=http://127.0.0.1:3128/ curl -k -v https://www.google.com
* About to connect() to proxy 127.0.0.1 port 3128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< Server: squid/3.5.20
< Mime-Version: 1.0
< Date: Mon, 04 Mar 2019 05:56:07 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3725
< X-Squid-Error: ERR_DNS_FAIL 0
< Vary: Accept-Language
< Content-Language: en
< 
* Received HTTP code 503 from proxy after CONNECT
* Connection #0 to host 127.0.0.1 left intact
curl: (56) Received HTTP code 503 from proxy after CONNECT
@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Mar 4, 2019

Thank you for your reply. Can you please try to reduce the program. I suggest removing the http server by moving the logic from the anonymous HandleFunc into main.

@nnathan

This comment has been minimized.

Copy link
Author

nnathan commented Mar 4, 2019

Yep sure.

So here is a condensed version without http server cruft from earlier:

package main

import (
	"log"
	"net/http"
)

func try(url string) {
	log.Printf("trying url: %s", url)
	resp, err := http.Get(url)

	if err != nil {
		log.Printf("error: %v", err)
		return
	}

	if resp.StatusCode == 503 {
		resp.Body.Close()

		squidErr := resp.Header.Get("X-Squid-Error")

		if squidErr != "" {
			log.Printf("squid failure error detected, X-Squid-Error: %s", squidErr)
			return
		}
	}

	log.Printf("Everything OK - no squid errors")
}

func main() {
	try("http://www.google.com")
	try("https://www.google.com")
}

Here is the output when running using the proxy:

$ http_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 go run main.go
2019/03/04 06:02:44 trying url: http://www.google.com
2019/03/04 06:03:19 squid failure error detected, X-Squid-Error: ERR_DNS_FAIL 0
2019/03/04 06:03:19 trying url: https://www.google.com
2019/03/04 06:03:19 error: Get https://www.google.com: Service Unavailable
@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Mar 4, 2019

@nnathan

This comment has been minimized.

Copy link
Author

nnathan commented Mar 4, 2019

This problem occurs specifically when proxying https connections which uses the CONNECT verb.

What I've learned is that there are two responses: a response for the CONNECT, and a response for the (tunnelled) proxy request.

I guess it makes sense to treat a non-successful response to CONNECT as an error, however this means a loss of information of the underlying error (which in this case is embedded as an X-Squid-Error header). For my purposes this isn't a huge issue, checking the response from a regular HTTP request is sufficient.

@julieqiu

This comment has been minimized.

Copy link

julieqiu commented Mar 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.