HTTP links on https://curl.haxx.se/download.html

[hannob]

On the curl download page
https://curl.haxx.se/download.html

there are a lot of HTTP links to pages that either provide HTTPS or default to HTTPS. Also there seem to be some dead links.

It's a lot of links, so not sure how to best tackle this, but one could probably start by making a script that goes through all of them and checks for http->https redirects and 404 errors.

[bagder]

Yeah, I think there are three parts to this:

1. Dead links (that truly are 404 forever and not just temporary). Remove.
2. to-HTTPS redirects. Update the URLs so that we link users correctly.
3. All the links that are HTTP-only or FTP. Consider to remove completely? Maybe mark them in some way?

I started on 1 and 2 a while ago and have converted a whole bunch over to HTTPS already.

[bagder]

Most them that are available over HTTPS are converted now.

[bagder]

I've removed several dead entries and I've chased down HTTPS links - turned out to be work since many of the download sites don't automatically redirect to HTTPS even when they do offer HTTPS alternatives!

There are now 152 entries in the download database out of 247 that are still using "unsafe" URLs. But without us knowing any safe alternative URLs to them. I made our internal system visualize this better to make it more apparent to us adding and editing entries.

That means 95 (38%) are using HTTPS links right now. I think this is as far as we can get right now.

[pghmcfc]

The mirror.city-fan.org (Paul Howarth) links are now available over https.

[bagder]

@pghmcfc excellent!

[bagder]

Recount:

Out of the 245 download links currently stored, 115 are using "unsafe" URLs (FTP or HTTP). That's 130 or 53% using HTTPS links.

[rockdaboot]

BTW, @hannob also opened an issue against the Wget website, which should now be clean.
I just kicked out all links to domains that doesn't provide HTTPS and changed the others from HTTP to HTTPS. But then there were several HTML parts automatically included from FSF templates that contained HTTP links. An issue to the FSF web administration has been quickly reacted on - now the pages seems clean.

So, maybe it's worth a try to file bugs against the HTTP-only domain admins to ask for HTTPS.

[bagder]

maybe it's worth a try to file bugs against the HTTP-only domain admins

Yeah, could be. But that's an awful lot of work since that's like 100 different sites, most of them without a clear bug/issue tracker where such bugs could be filed. I suspect most of them would have to be contacted by email and I'm just generally pessimistic about non-HTTPS site owners acting on single users asking them to switch to HTTPS...

[rockdaboot]

Hmmm, if it's so many, is there a way to find out the admin's email to automatically generate emails ? Maybe whois could be a helper ?

[bagder]

I'm down to 54 unique host names with linked download resources not available over HTTPS. Most of these host really old versions of curl too, which indicates that they're not eagerly maintained:

azalea.ics.agh.edu.pl
cgit.openembedded.org
codex.sourcemage.org
download.qnx.com
downloads.openwrt.org
embedded.org.ua
ftp.halifax.rwth-aachen.de
ftp.linux-mips.org
ftp.netbsd.org
ftp.nluug.nl
ftp.pld-linux.org
ftp.redhat.com
ftp.sco.com
ftp.splack.org
ftp.thewrittenword.com
ftp.turbolinux.co.jp
git.dragonflybsd.org
git.minix3.org
h20293.www2.hp.com
hg.slitaz.org
hpux.connect.org.uk
hydra.nixos.org
libpak.neoni.net
mik.dyndns.pro
mirrors.slackware.com
modules.lunar-linux.org
os4depot.net
os.archlinuxarm.org
pdb.finkproject.org
pkg.openindiana.org
pkgsrc-repo.uk.openindiana.org
pkgsrc.se
pmandin.atari.org
pub.allbsd.org
repository.maemo.org
slackintosh.workaround.ch
syllable.cvs.sourceforge.net
web.syllable.org
winampplugins.co.uk
wraptastic.org
www.aixtools.net
www.aljex.com
www.archhurd.org
www.confusedbycode.com
www.gnu-darwin.org
www.linuxfromscratch.org
www.magicermine.com
www.myslug.de
www.oss4aix.org
www.paehl.com
www.rahul.net
www.sisyphus.ru
www.smedley.info
www.strcprstskrzkrk.co.uk

[rockdaboot]

Just had quick look with

for i in $x; do i=`echo $i|cut -d'.' -f2-`; echo $i";"`whois $i|grep @|cut -d':' -f2`; done

with your list in $x.

You'll find a few persons (at least the emails look like it), but also many general addresses from big hosters. Likely those won't answer, so your are left with some hand-picked email addresses in the end.
Not sure if it's worth it.

[rockdaboot]

You should make a second try, just hand-tested some:

• h20293.www2.hp.com doesn't exist at all
• ftp.netbsd.org has HTTPS
• git.dragonflybsd.org has HTTPS

And maybe it is worth to open an issue for the Redhat server.

[bagder]

Thanks, fixed!

This is not really a game that ends. Servers will come and go over time so surely some will break soon and there will soon appear HTTPS alternatives for currently unsafe ones. We will continue updating links going forward as we analyze the daily download-scanner logs and hope that people keep reporting issues.

[bagder]

Closing this issue. Thanks everyone!

[rockdaboot]

A last tip, latest statistics stuff for Wget2 (GSOC 2017 project):
Put your domain list with https:// prepended into list.txt and execute

wget2 --timeout=3 --stats-tls=csv:- --spider --tries=1 -i list.txt

to get a (machine readable) CSV report about TLS features.

This is the current list with domains that seem to support HTTPS:

Hostname,TLSVersion,FalseStart,TFO,ALPN,Resumed,HTTPVersion,Certificates,Duration
pkg.openindiana.org,TLS1.2,0,1,-,1,1,2,521
ftp.turbolinux.co.jp,TLS1.0,0,1,-,1,1,2,881
hydra.nixos.org,TLS1.2,0,1,http/1.1,1,1,2,2713
ftp.linux-mips.org,TLS1.2,0,1,http/1.1,1,1,2,2517
web.syllable.org,TLS1.2,0,1,h2,1,2,3,1092
www.linux-mips.org,TLS1.2,0,1,http/1.1,1,1,2,980
ftp.netbsd.org,TLS1.2,0,1,-,0,1,3,3701
git.dragonflybsd.org,TLS1.2,0,1,http/1.1,1,1,3,857
downloads.openwrt.org,TLS1.2,0,1,http/1.1,1,1,2,731
ftp.nluug.nl,TLS1.2,0,1,-,1,1,2,781
mirrors.slackware.com,TLS1.2,0,1,http/1.1,1,1,2,1102
www.aljex.com,TLS1.2,0,1,h2,1,2,3,1413
winampplugins.co.uk,TLS1.2,0,1,h2,1,2,3,751
www.archhurd.org,TLS1.2,0,1,h2,1,2,2,748
ftp.halifax.rwth-aachen.de,TLS1.2,0,1,h2,1,2,4,775
pub.allbsd.org,TLS1.2,0,1,http/1.1,1,1,2,2330