New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

website: binary download links involve redirection #22648

Closed
bradfitz opened this Issue Nov 9, 2017 · 10 comments

Comments

Projects
None yet
4 participants
@bradfitz
Member

bradfitz commented Nov 9, 2017

We used to have really nice download links that just worked.

Now there's this HTTP redirection, making curl usage more annoying:

pi@raspberrypi:~ $ curl -O https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-armv6l.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   626  100   626    0     0   2087      0 --:--:-- --:--:-- --:--:--  2100


pi@raspberrypi:~ $ cat go1.9.2.linux-armv6l.tar.gz 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://r13---sn-nx57yn7k.gvt1.com/edgedl/go/go1.9.2.linux-armv6l.tar.gz?cms_redirect=yes&amp;expire=1510278830&amp;ip=2620:0:1009:fd00:e066:33c6:5590:624&amp;ipbits=0&amp;mm=28&amp;mn=sn-nx57yn7k&amp;ms=nvh&amp;mt=1510264098&amp;mv=u&amp;pl=56&amp;shardbypass=yes&amp;sparams=expire,ip,ipbits,mm,mn,ms,mv,pl,shardbypass&amp;signature=2E1506D41A75DC13E8293B0274C93AE43A6285FD.5C967816ABA01ED0CF2F7BB3DC6EBE673E3B2566&amp;key=cms1">here</A>.
</BODY></HTML>

We can't serve from the "edge" without all that stuff? We do for other things, and others on the Internet do too.

What is going on?

@bradfitz bradfitz added this to the Unplanned milestone Nov 9, 2017

@andybons

This comment has been minimized.

Show comment
Hide comment
@andybons

andybons Nov 9, 2017

Member

The old urls, while prettier and direct, are blocked in some countries. This not only allows us to serve faster downloads to our users more cheaply, but also reach the countries that block GCS.

I’m not sure how much of the internal guts I’m able to share publically as to why Goog’s edge cache behaves the way that it does, but I’ve asked.

In the meantime, curl -O -L https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-armv6l.tar.gz

Member

andybons commented Nov 9, 2017

The old urls, while prettier and direct, are blocked in some countries. This not only allows us to serve faster downloads to our users more cheaply, but also reach the countries that block GCS.

I’m not sure how much of the internal guts I’m able to share publically as to why Goog’s edge cache behaves the way that it does, but I’ve asked.

In the meantime, curl -O -L https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-armv6l.tar.gz

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Nov 10, 2017

Member

Noted elsewhere, I see Chrome uses https://dl.golang.org/* URLs directly. What is different between Go and Chrome, that Chrome gets to use pretty & non-redirecting URLs?

Member

bradfitz commented Nov 10, 2017

Noted elsewhere, I see Chrome uses https://dl.golang.org/* URLs directly. What is different between Go and Chrome, that Chrome gets to use pretty & non-redirecting URLs?

@andybons

This comment has been minimized.

Show comment
Hide comment
@andybons

andybons Nov 10, 2017

Member

storage.googleapis.com is definitely blocked.

We can use dl.google.com. It's not blocked, but when I asked about this I was told that it costs more money and is less reliable.

Member

andybons commented Nov 10, 2017

storage.googleapis.com is definitely blocked.

We can use dl.google.com. It's not blocked, but when I asked about this I was told that it costs more money and is less reliable.

@golang golang deleted a comment from forskning Nov 10, 2017

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Nov 10, 2017

Member

Let's switch to dl.google.com. If it's good enough for Chrome, it's good enough for us.

Member

bradfitz commented Nov 10, 2017

Let's switch to dl.google.com. If it's good enough for Chrome, it's good enough for us.

@andybons

This comment has been minimized.

Show comment
Hide comment
@andybons

andybons Nov 10, 2017

Member

OK. Sounds good.

Member

andybons commented Nov 10, 2017

OK. Sounds good.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Nov 10, 2017

Change https://golang.org/cl/76971 mentions this issue: godoc: use dl.google.com instead of redirector.gvt1.com

gopherbot commented Nov 10, 2017

Change https://golang.org/cl/76971 mentions this issue: godoc: use dl.google.com instead of redirector.gvt1.com

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Nov 11, 2017

Member

We used to have really nice download links that just worked.

@bradfitz, what I've always used (for the last 4-6~ major releases) was:

curl -L https://golang.org/dl/go1.9.2.linux-amd64.tar.gz

I thought that was the "canonical nice download link". Is it not the canonical nice link, or were you not aware that it exists?

For reference, it currently redirects to https://redirector.gvt1.com/edgedl/...:

$ curl -i https://golang.org/dl/go1.9.2.linux-amd64.tar.gz
HTTP/2 302 
location: https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-amd64.tar.gz
...

I guess this a question/comment... I want to find out if other people are aware of the https://golang.org/dl/... links and what is their status (canonical? secondary? deprecated?). Thanks.

Member

dmitshur commented Nov 11, 2017

We used to have really nice download links that just worked.

@bradfitz, what I've always used (for the last 4-6~ major releases) was:

curl -L https://golang.org/dl/go1.9.2.linux-amd64.tar.gz

I thought that was the "canonical nice download link". Is it not the canonical nice link, or were you not aware that it exists?

For reference, it currently redirects to https://redirector.gvt1.com/edgedl/...:

$ curl -i https://golang.org/dl/go1.9.2.linux-amd64.tar.gz
HTTP/2 302 
location: https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-amd64.tar.gz
...

I guess this a question/comment... I want to find out if other people are aware of the https://golang.org/dl/... links and what is their status (canonical? secondary? deprecated?). Thanks.

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Nov 11, 2017

Member

Looking at golang/tools@9c477ba more closely, I see that what's been modified is the redirect URL that https://golang.org/dl/... returns (and I see that change isn't live yet).

So, before that commit, it was:

nice golang.org/dl URL -> ugly https://redirector.gvt1.com/edgedl/... URL

And after:

nice golang.org/dl URL -> nice https://dl.google.com/go/... URL

(Arrow represents an HTTP redirect.)

I guess that partially answers my question, but my followup question then is, have you considered using curl -L https://golang.org/dl/... and not caring what the underlying redirect URL it uses under the hood? (If the answer is "yes I've considered it", that's completely fine; I just want to confirm.)

Member

dmitshur commented Nov 11, 2017

Looking at golang/tools@9c477ba more closely, I see that what's been modified is the redirect URL that https://golang.org/dl/... returns (and I see that change isn't live yet).

So, before that commit, it was:

nice golang.org/dl URL -> ugly https://redirector.gvt1.com/edgedl/... URL

And after:

nice golang.org/dl URL -> nice https://dl.google.com/go/... URL

(Arrow represents an HTTP redirect.)

I guess that partially answers my question, but my followup question then is, have you considered using curl -L https://golang.org/dl/... and not caring what the underlying redirect URL it uses under the hood? (If the answer is "yes I've considered it", that's completely fine; I just want to confirm.)

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Nov 13, 2017

Member

@shurcooL, I don't want people to see URLs with "redirector.gvt1.com" in them for hostnames. That says neither Google nor Golang in it and looks spammy.

Member

bradfitz commented Nov 13, 2017

@shurcooL, I don't want people to see URLs with "redirector.gvt1.com" in them for hostnames. That says neither Google nor Golang in it and looks spammy.

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Nov 13, 2017

Member

@bradfitz Makes sense. Thanks for confirming!

Member

dmitshur commented Nov 13, 2017

@bradfitz Makes sense. Thanks for confirming!

@mblair mblair referenced this issue Dec 8, 2017

Merged

go 1.10beta1 (devel) #21448

4 of 4 tasks complete

dochang added a commit to dochang/homebrew-binary that referenced this issue Feb 10, 2018

dochang added a commit to dochang/homebrew-binary that referenced this issue Feb 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment