Use SSL for AUTO_BOOSTRAP #7000
Conversation
| HOST_DMD_ROOT=$(TMP)/.host_dmd-$(HOST_DMD_VER) | ||
| # dmd.2.072.2.osx.zip or dmd.2.072.2.linux.tar.xz | ||
| HOST_DMD_BASENAME=dmd.$(HOST_DMD_VER).$(OS)$(if $(filter $(OS),freebsd),-$(MODEL),) | ||
| # http://downloads.dlang.org/releases/2.x/2.072.2/dmd.2.072.2.linux.tar.xz | ||
| HOST_DMD_URL=http://downloads.dlang.org/releases/2.x/$(HOST_DMD_VER)/$(HOST_DMD_BASENAME) | ||
| HOST_DMD_URL=https://s3-us-west-2.amazonaws.com/downloads.dlang.org/releases/$(HOST_DMD_YEAR)/$(HOST_DMD_BASENAME) |
There was a problem hiding this comment.
How long is that link going to keep working? It looks awfully fragile, with apparently having the datacenter location in it.
The solution here is rather to either enable proper SSL for downloads.dlang.org, or just add a SHA256 check to the downloads.
There was a problem hiding this comment.
How long is that link going to keep working? It looks awfully fragile, with apparently having the datacenter location in it.
For as long as the S3 bucket is alive (S3 buckets are bound to a datacenter and can't be moved to a different location or renamed).
The solution here is rather to either enable proper SSL for downloads.dlang.org,
See also: https://trello.com/c/mWHZxwo5/89-https-for-s3-downloadsdlangorg
or just add a SHA256 check to the downloads.
This is going to be annoying for updates and ensuring that sha256sum is installed on all machines. Also we can't do the streaming extraction anymore.
There was a problem hiding this comment.
We may decide to move hosting the downloads away from Amazon to some other service at some point, which would make building older D versions overly onerous. It would be much better if we could avoid exposing "implementation details" like the real download server, and use dlang.org links.
If it is too complicated to add SSL to downloads.dlang.org for some reason, even some redirects on https://dlang.org would be better. In fact, why not move the HTML pages of downloads.dlang.org to the dlang.org server?
There was a problem hiding this comment.
Also we can't do the streaming extraction anymore.
If you really wanted to, you could still do it with tee and process substitution.
There was a problem hiding this comment.
It would be much better if we could avoid exposing "implementation details" like the real download server, and use dlang.org links.
Well, we could have downloads.dlang.io where the year mapping is hard-coded, but then all traffic would flow to a single point of failure and the entire point of using S3 is gone.
In fact, why not move the HTML pages of downloads.dlang.org to the dlang.org server?
Because they are automatically generated and I bet @MartinNowak wouldn't like sending yet another PR on every update.
There was a problem hiding this comment.
Well, we could have downloads.dlang.io where the year mapping is hard-coded, but then all traffic would flow to a single point of failure and the entire point of using S3 is gone.
??? The website would just do a redirect. It would be no more of a single point of failure than the dlang.org website.
Not sure if W/A would like the downloads going though a server/domain not owned by the foundation, though.
Because they are automatically generated and I bet @MartinNowak wouldn't like sending yet another PR on every update.
I did not say "part of dlang.org github repository", I just said same server, because apparently Jan has no problem setting SSL up.
There was a problem hiding this comment.
The website would just do a redirect. It would be no more of a single point of failure than the dlang.org website.
I think the best solution would be to either
- upload releases to GH releases too
- restructure the download,s.t. they are year independent
- use the releases to a provider that supports SSL out-of-the-box
Note that we could use Cloudfront on AWS to achieve SSL, but sadly that would/might be really expensive (not sure how much traffic is currently generated).
because apparently Jan has no problem setting SSL up.
There was a problem hiding this comment.
upload releases to GH releases too
GitHub used to have a downloads feature, and then they abruptly took that away. Today, all links to those downloads are broken.
restructure the download,s.t. they are year independent
That looks like a separate nuisance.
https://dlang.org/ works fine. Is it a case of us never asking SSL for that domain?
|
We have a story for TLS on S3, using their raw bucket links doesn't seem to be the way to go. |
The redirects are S3 internal (downloads/2.x/2.075 -> downloads/2017/2.075), so we'd need to hardcode those on some dlang.org hosted download pages. For https://dlang.org -> https://s3 we'd furthermore need to hardcode the bucket links. https://github.com/dlang/downloads.dlang.org/blob/master/make-links Since I generally don't have the best connection to dlang.org, I'd be cautious to move all our downloads (including all the CI systems) through a bottleneck redirect at dlang.org. |
|
Changing the AUTO_BOOTSTRAP is a bad point to being with a S3 HTTPS transition. |
I believe this doesn't matter, because on a modern system a D compiler should be present and package maintainers use source tarballs (or digger) to do the bootstrapping.
Unfortunately, the year needs to be set as well as the release layout is a bit weird.
See also: