New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Transport does not handle 302 redirects with link-local server address correctly #20669

Open
max2k1 opened this Issue Jun 14, 2017 · 20 comments

Comments

Projects
None yet
6 participants
@max2k1

max2k1 commented Jun 14, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/go"
GORACE=""
GOROOT="/opt/go"
GOTOOLDIR="/opt/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

I need to get some data from link-local neighbor:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	if resp, err := http.Get("http://[fe80::1%25eth0]:8002/info.json"); err == nil {
		defer resp.Body.Close()
		fmt.Println(resp)
	} else {
		fmt.Println(err)
	}
}

What did you expect to see?

Something like this:

&{200 OK 200 HTTP/1.1 1 1 map[Connection:[keep-alive] Set-Cookie:[uid=AAAAAVlBPd2KJ2cFAwMUAg==; path=/] Server:[nginx] Date:[Wed, 14 Jun 2017 13:45:01 GMT] Content-Type:[application/json]] 0xc42007a600 -1 [chunked] false false map[] 0xc4200e8300 <nil>}

What did you see instead?

Get http://[fe80::1]:8002/info.sh: dial tcp [fe80::1]:8002: connect: invalid argument

It happened, because http://[fe80::1%eth0]:8002/info.json returned 302-redirect to /info.sh and http-library lost zone-prefix somewhere while following redirect.
Proof:

curl -v "http://\[fe80::1%25eth0\]:8002/info.json"
*   Trying fe80::1...
* Connected to fe80::1 (fe80::1) port 8002 (#0)
> GET /info.json HTTP/1.1
> Host: [fe80::1]:8002
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 302 Moved Temporarily
< Server: nginx
< Date: Wed, 14 Jun 2017 13:49:17 GMT
< Content-Type: text/html
< Content-Length: 154
< Location: http://[fe80::1]:8002/info.sh
< Connection: keep-alive
< Set-Cookie: uid=AAAAAVlBPt2KJ2cFAwMVAg==; path=/
<
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host fe80::1 left intact

@bradfitz bradfitz changed the title from http.Get does not handle 302 redirects with link-local server address correctly to net/http: Transport does not handle 302 redirects with link-local server address correctly Jun 14, 2017

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jun 14, 2017

Where is this behavior defined in a spec?

That 302 response's Location header is Location: http://[fe80::1]:8002/info.sh and doesn't have a zone. Are we really supposed to keep state and associate a zone with an authority for some duration?

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jun 14, 2017

/cc @mikioh

@max2k1

This comment has been minimized.

max2k1 commented Jun 14, 2017

Where is this behavior defined in a spec?
That 302 response's Location header is Location: http://[fe80::1]:8002/info.sh and doesn't have a zone.

It seems to me, there are no RFCs concerning this behaviour. Web server just use requst's HOST-header, we've send to it, in response's Location-header, it has no idea about zones, because they're local to client. It is client, who should decide, how to reach specified location.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jun 14, 2017

I don't want to complicate Go's code for a niche use case that's not even defined.

@bradfitz bradfitz modified the milestones: Unplanned, Go1.10 Jun 14, 2017

@max2k1

This comment has been minimized.

max2k1 commented Jun 14, 2017

But it just does not work now: strange exception after correct http-server response... It's weird. Modern curl, for example, deals with that kind of addresses in responses easily.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jun 14, 2017

Curl is probably not handling it and just handling connection cache management differently and you're getting lucky.

If you find evidence that curl is explicitly handling this in some way, then I might be convinced to do the same.

Please go grep the curl source code and figure out whether curl deals with IPv6 zones.

@mikioh

This comment has been minimized.

Contributor

mikioh commented Jun 14, 2017

See https://tools.ietf.org/html/rfc6874. Handling "beyond routing scope" is one of classic IPv6 issues and that's the reason why IPv6 site-local unicast addresses are deprecated: https://tools.ietf.org/html/rfc3879.

So using IPv6 literals in HTTP has some restriction. I'm guessing that one control plane solution could be the combo of a) using mDNS for propagating on-link adjacency information and b) using non-IPv6 literals, node/service names propagated by mDNS, in HTTP.

@max2k1

This comment has been minimized.

max2k1 commented Jun 14, 2017

that's the reason why IPv6 site-local unicast addresses are deprecated: https://tools.ietf.org/html/rfc3879.

fe80::1 is not kind of site-local, but a link-local address – they will not be deprecated in the near future for sure :). It is like an 192.168.0.1 in ipv4 world, because of a common pattern to set it on default gw devices and route through it. How often do you see http://192.168.0.1/something URL's? I think you see them often enough. Now try to imagine, that you favourite language http-transport does not handle 302 to 192.168.0.1 address...

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jun 14, 2017

@max2k1, I think you messed up your quoting there. I'm not sure how to read that comment.

Somebody needs to translate this into English for me. I speak a little networking, but not as a native language.

@max2k1

This comment has been minimized.

max2k1 commented Jun 14, 2017

Curl is probably not handling it and just handling connection cache management differently and you're getting lucky.

Curl does recognise different scopes of ip6-addresses here: https://github.com/curl/curl/blob/93567071b744975e191c5732d9cbe4b0d529f4c9/lib/if2ip.c#L69-L76
and selects correct interface there: https://github.com/curl/curl/blob/93567071b744975e191c5732d9cbe4b0d529f4c9/lib/if2ip.c#L115-L193

@mikioh

This comment has been minimized.

Contributor

mikioh commented Jun 14, 2017

translate this into English

See https://tools.ietf.org/html/rfc6874#section-4. RFC 3879 describes the rest of pains on operating "scope boundaries."

It is like an 192.168.0.1 in ipv4 world

Nope, see https://tools.ietf.org/html/rfc3927. The equivalent of IPv4 private addresses is https://tools.ietf.org/html/rfc4193.

@max2k1

This comment has been minimized.

max2k1 commented Jun 14, 2017

@max2k1, I think you messed up your quoting there. I'm not sure how to read that comment.
Somebody needs to translate this into English for me. I speak a little networking, but not as a native language.

Sorry, I'll try to be simpler.
fe80::1 is a link-local address. Usually, you do not need global addresses in ipv6 world to communicate over one shared link (ethernet segment for example). Instead, you got automatically generated link-local addresses on every interface of you network station (fe80::/64) and they can be used to reach any of your neighbour. So every network interface of any ipv6-enabled device always have link-local address (one, or more).
It's a common practice in ipv6-enabled networks to use link-local addresses (fe80::1 usually for "default gateways") as routers' addresses, because it is very handy. That is why I compared fe80::1 with 192.168.0.1. 192.168.0.1 is widely used as default gateway address and, because of that, frequently used in http-requests. So, in IPv6-only world it is common to query over http link-local addresses. That is why I think, that it is important to fully support this case in golang http-transport.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jun 14, 2017

And why doesn't it work now?

Per RFC 6874, we already remove the zone identifier from the outgoing Host header: https://github.com/golang/go/blob/release-branch.go1.8/src/net/http/request.go#L678

That was added in 957255f and has tests.

So we already can successfully make requests to http://[fe80::1%whatever]:80.

But what says that when that server replies with a redirect code + Location header response without a zone, which zone are we supposed to use?

@max2k1

This comment has been minimized.

max2k1 commented Jun 14, 2017

But what says that when that server replies with a redirect code + Location header response without a zone, which zone are we supposed to use?

We know local interface of this tcp-connection. So this is zone id.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jun 14, 2017

Okay, I think I finally understand what you want.

In the future, please write your bug report with all the information.

I would consider a patch for this if it's minimally invasive, but I don't intend to work on this myself.

@mikioh

This comment has been minimized.

Contributor

mikioh commented Jun 15, 2017

I have no strong opinion on this feature because I have no measurement for IPv6 addressing/routing scope boundary handling in transport- and/or session-layer protocols, such as HTTP.

FWIW, the net/http package provides the CheckRedirect field of http.Client for such use cases.

@SCKelemen

This comment has been minimized.

Contributor

SCKelemen commented Jul 6, 2017

@gopherbot

This comment has been minimized.

gopherbot commented Sep 20, 2017

Change https://golang.org/cl/64911 mentions this issue: net/http: remove ipv6 link-local address zone ID from connection cache key

@crvv

This comment has been minimized.

Contributor

crvv commented Sep 20, 2017

This looks like an undefined behavior of HTTP, so I think supporting this explicitly is not necessary.
Other implementations (I checked cURL and Python) don't support this explicitly.

cURL extracts zone ID from link-local address and store them separately. cURL uses both of them to setup connections and use address as key to cache connections. So the second request sent by cURL uses the previous connection and can send request successfully.

Python doesn't handle zone ID at all.

The relevant cURL code is
https://github.com/curl/curl/blob/8839c05fba1f8415eec17eff8ac60cc3a50eb51e/lib/url.c#L4749
https://github.com/curl/curl/blob/8839c05fba1f8415eec17eff8ac60cc3a50eb51e/lib/conncache.c#L123

I think using the same logic(do not use zone ID in cache key) in Go is enough.

@mikioh

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment