Support HTTPS

[purcell]

We've previously declined to provide this (see #1877) on the basis that signing packages (#1749) is the long-term path to ensuring tamper-evident installations.

However, signing will only benefit those with Emacs 24.4 or greater, and now that we've settled MELPA on a dedicated TLD we can easily get a (free) SSL cert. So let's do that.

Thoughts:

• StartSSL provides free certs for a TLD and one subdomain (usually "www.", but "stable." in our case)
• CloudFlare would proxy and SSL the sites for free, but this would stop us from calculating download statistics
• If we go SSL-only, which we should, then we need to be prepared for problems when users' Emacsen do not have access to the open_ssl or gnutls helper programs. This seems to have affected Windows users following Marmalade's switch to SSL-only.

Suggested steps:

• @milkypostman verifies his own identity with StartSSL using a @melpa.org email address
• @milkypostman obtains a free cert using that verified identity
• Either one of us sets up the cert in nginx on the servers, and probably enables SPDY too. I've done this a bunch, so can easily take care of this.
• We update the "getting started" page and README to recommend HTTPS, and start using that URL ourselves.
• When ready, 301 the HTTP URLs to HTTPS.

To check:

• Protocol changes are considered a domain change by Webmaster Tools, because you could technically run completely different sites on https: and http:. Can't recall if there's an effect on Google Analytics, dlvr.it etc., but it's worth looking out for.

/cc @technomancy (thanks for the prod)

[Fanael]

If we go SSL-only, which we should, then we need to be prepared for problems when users' Emacsen do not have access to the open_ssl or gnutls helper programs.

I'm building the world right now, and I think the best solution would be setting up a helpline where one could vent all their frustration caused by autotools.

[technomancy]

An unfortunate wrinkle in this equation: http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg01000.html

The 24.4 release of Emacs introduces an unfortunate bug in package.el that causes dependencies of TLS-fetched packages to be downloaded without TLS. Might want to put this on hold until the package.el bug is fixed.

[purcell]

@technomancy Thanks for the heads-up. I'd seen Nic had encountered issues switching Marmalade over. Will wait a bit and see how this plays out.

[rmgrimm]

Glad to see this is being considered!

When implementing this, please keep both HTTP and HTTPS open. I've been in both situations where one is blocked and the other available, depending on company firewall/transparent proxy policy.

[glyph]

One way to provide TLS without having to manage it on melpa.org yourself would be to use a github pages domain (https://milkypostman.github.io/melpa) to host all the packages.

If you want to keep the packages on MELPA, I have a lot of experience configuring servers to support TLS and I know of several zero-cost valid cert options, and I'd be happy to volunteer to administer this aspect of the process, if someone wanted to give me appropriate credentials.

I think that many people on this ticket are underestimating the severity of this problem. MELPA is, currently, an obvious vector for malware, and the only reason it hasn't been exploited in this manner is that it's relatively obscure. However, determined attackers will soon realize that people with lots of elevated privileges (system administrators, programmers with administrative credentials) use this repository and will likely target it in the future if they have not done so already. For example, @rmgrimm - if you can't access HTTPS, then I don't think you should be able to access MELPA. It's irresponsible to have the capability to secure users connections, but refuse to do so on the basis that there might be some users in some environments that are so heavily surveilled that they are not allowed to make secure connections at all.

[glyph]

One possible solution to Windows users without access to OpenSSL would be to use a script that uses the native Windows HTTP client, like https://serverfault.com/questions/29707/download-file-from-vbscript/29720#29720

[glyph]

Bump.

Although package signing is surely the real future-proof way to do this, HTTP-only MELPA remains very low-hanging fruit for attackers to put malware onto Emacs users' machines. There's rudimentary support for HTTPS in package.el already; please just let me use it.

[milkypostman]

Ok I'll take care of this. I think part of the problem was that there was a
bug in package.el with http a then we never got around to it. Your wish
shall be granted.
On Sun, Aug 9, 2015 at 12:33 Glyph notifications@github.com wrote:

Bump.

Although package signing is surely the real future-proof way to do this,
HTTP-only MELPA remains very low-hanging fruit for attackers to put malware
onto Emacs users' machines. There's rudimentary support for HTTPS in
package.el already; please just let me use it.

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[glyph]

Thanks so much, @milkypostman !

[glyph]

Anyone interested in this issue will probably also be interested to know that you need these custom variables in order to actually verify TLS:

 '(tls-checktrust t)
 '(tls-program
   (quote
    ("gnutls-cli --x509cafile \"$(python -m certifi)\" -p %p %h")))

By default, Emacs doesn't verify TLS certificates, nor TLS hostnames, and includes a number of fallbacks which will fall through to not verifying if you happen to not have gnutls-cli installed.

You may want to replace the python -m certifi there with a static path to your CA bundle, for example, /etc/ssl/certs/ca-certificates.crt if you're on linux; certifi is a Python package that distributes a root-CA-trust bundle for other programs to use.

[tzz]

@glyph Emacs has built-in GnuTLS integration in the C code that makes HTTP/S verification simple without requiring tls-checktrust etc. and it works on almost every platform.

For the Network Security Manager, the piece that actually manages certificates and warns you about expired ones: http://lars.ingebrigtsen.no/2014/12/01/the-emacs-network-security-manager/

For the GnuTLS integration, see https://www.gnu.org/software/emacs/manual/html_mono/emacs-gnutls.html

Emacs 25 could be the release that officially deprecates the old tls-* variables.

[glyph]

@tzz - It doesn't work on the Emacs from either https://emacsformacosx.com or from Ubuntu. So the advice stands: for a given user, their Emacs is probably insecure and needs to be adjusted. In five years when Emacs 25 is common, if they remove the build options that allow it to build without GnuTLS, then users might be able to start relying on it.

[tzz]

@glyph you're absolutely right: I thought your instructions were incomplete, not wrong. Sorry I wasn't explicit about that! So maybe you could join them with mine? I'll be glad to include that in the GnuTLS integration docs I cited as a "how to use HTTP/S reliably" guide. I don't think one exists right now anywhere so it will be a nice addition.

[glyph]

@tzz Ah, I guess I misread. Honestly what I do is I do is head over to *scratch*, then

(url-insert-contents-of-file "https://www.github.com/")
(url-insert-contents-of-file "https://www.financialcryptography.com/")

and the former should work, the latter should fail. If the latter succeeds then I get sad and check my configuration.

[milkypostman]

I tried to setup the SSL with startSSL and they are having problems right now which doesn't bode well. but I will try again tomorrow. Just wanted to update.

[purcell]

@milkypostman You can get free certs from them for the root domain plus one subdomain, so I guess that should cover both melpa.org and stable.melpa.org...

[glyph]

@milkypostman thanks for the update. StartSSL's uptime isn't great, but luckily once you're set up you only need to use their site once a year :).

[milkypostman]

are there any recommended SSL provides? I'm happy to just pay the amount.
StartSSL is having some problem with THEIR ssl certs,so I'm a bit hesitant
to even get one from them now. even if it's $100 a year I don't care that
much. just want to go with someone reliable.

On Fri, Aug 14, 2015 at 4:19 PM, Glyph notifications@github.com wrote:

@milkypostman https://github.com/milkypostman thanks for the update.
StartSSL's uptime isn't great, but luckily once you're set up you only need
to use their site once a year :).

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[glyph]

What problems with their certificate are you referring to? If you're willing to spend a little money, there's SSLMate which automates more of the process and is still pretty cheap.

[milkypostman]

The problem is that I login but cannot get to the control panel. When I go
to the control panel it gives an ssl error.
On Mon, Aug 17, 2015 at 16:11 Glyph notifications@github.com wrote:

What problems with their certificate are you referring to? If you're
willing to spend a little money, there's SSLMate https://sslmate.com
which automates more of the process and is still pretty cheap.

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[glyph]

When you say you "login", what do you mean? Typically logging in takes you straight to the control panel. Are you sure you've properly installed your client authentication certificate in your browser?

[glyph]

(I am able to get to my account, so it is not a problem with the StartSSL site, I don't think.)

[milkypostman]

OK, so I must admit I wasn't aware of client auth certs before using this. So either it's in my download folder somewhere or I lost it. I'll try again tonight. I see what's happening.

[glyph]

Glad I could help then :). I hope you didn't lose it!

[milkypostman]

OK, give it a shot. It should work now!

On Tue, Aug 18, 2015 at 11:33 AM, Glyph notifications@github.com wrote:

Glad I could help then :). I hope you didn't lose it!

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[tzz]

I was excited by this announcement https://www.eff.org/deeplinks/2014/11/certificate-authority-
encrypt-entire-web when it came in late 2014.

https://letsencrypt.org/ is supposed to go live by Q4 2015, so I'd wait for that.

[tzz]

@glyph continuing the discussion here since you posted the instructions here... I posted a patch against the Emacs GnuTLS integration manual to emacs-devel: https://lists.gnu.org/archive/html/emacs-devel/2015-08/msg00693.html

If you could review that, it would be great (feel free to follow up on emacs-devel or to me directly). Also I don't know if you want to be mentioned in the commit, but if you do, I don't know what name to use.

[milkypostman]

cool, thanks for the update ted. I already added the ssl certs from
startssl but this would be great in the future!

On Wed, Aug 19, 2015 at 7:32 AM Ted Zlatanov notifications@github.com
wrote:

@glyph https://github.com/glyph continuing the discussion here since
you posted the instructions here... I posted a patch against the Emacs
GnuTLS integration manual to emacs-devel:
https://lists.gnu.org/archive/html/emacs-devel/2015-08/msg00693.html

If you could review that, it would be great (feel free to follow up on
emacs-devel or to me directly). Also I don't know if you want to be
mentioned in the commit, but if you do, I don't know what name to use.

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[glyph]

Contacting host: elpa.gnu.org:443
Opening TLS connection to `elpa.gnu.org'...
Opening TLS connection with `gnutls-cli --x509cafile "$(python -m certifi)" -p 443 elpa.gnu.org'...done
Opening TLS connection to `elpa.gnu.org'...done
Contacting host: elpa.gnu.org:443
Opening TLS connection to `melpa.org'...
Opening TLS connection with `gnutls-cli --x509cafile "$(python -m certifi)" -p 443 melpa.org'...done
Opening TLS connection to `melpa.org'...done
Contacting host: melpa.org:443
35 packages can be upgraded; type `U' to mark them for upgrading.

\o/
 |
/ \

[glyph]

This could be closed now, although I think that updating https://melpa.org/#/getting-started to point at https would be good to do before calling it done.

[marsam]

Thanks a lot for this. I was really waiting for the https version.

[purcell]

This could be closed now, although I think that updating https://melpa.org/#/getting-started to point at https would be good to do before calling it done.

Done in 37c241a, but I think we should also consider setting up HSTS headers.

[glyph]

Not to discourage you, but, will gnutls-cli or Emacs actually respect HSTS headers?

[purcell]

Not to discourage you, but, will gnutls-cli or Emacs actually respect HSTS headers?

I don't really care -- it's just a good thing to do for HTTPS sites.

[glyph]

I don't really care -- it's just a good thing to do for HTTPS sites.

Fair enough. First order of business would be removing the HTTP site and replacing it with a redirect, yes?

[purcell]

Yes. Maybe just for the html content, though, because - at least for a while - I'd like to allow people to still fetch packages over http. I don't want to have to urgently deal with any fallout like Marmalade's switch to https.

[milkypostman]

glyph, do i need to post your snippet along with the url to get things
working for users in the "getting started" section?

On Sat, Aug 22, 2015 at 11:34 PM, Steve Purcell notifications@github.com
wrote:

Yes. Maybe just for the html content, though, because - at least for a
while - I'd like to allow people to still fetch packages over http. I don't
want to have to urgently deal with any fallout like Marmalade's switch to
https.

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[glyph]

@milkypostman - Yes, although that snippet is probably insufficient by itself. Assuming that users can configure a working certifi package as an exercise for the reader if they're not already Python programmers is deeply over-optimistic. I will try to write a blog post you can reference explaining this as concisely as I can while still being complete enough for people to get things working on multiple platforms.

[glyph]

(For that matter I have never actually tested this on Windows, where I might learn some more exciting things.)

[milkypostman]

is it easier if you can assume that users have the tickets available in
/etc? because i would assume that most users do.

On Sun, Aug 23, 2015 at 2:37 PM, Glyph notifications@github.com wrote:

(For that matter I have never actually tested this on Windows, where I
might learn some more exciting things.)

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[glyph]

Only users on Linux do, and only if they have particular packages installed (ca-certificates on debian, not sure what it is on fedora).

[glyph]

It looks like Windows is quite a pickle. The official emacs builds don't include GnuTLS, and getting GnuTLS installed is tricky (msysgit doesn't come with it; you need a full Cygwin, it looks like).

[milkypostman]

imo windows users are on their own.

On Mon, Aug 24, 2015 at 11:15 AM Glyph notifications@github.com wrote:

It looks like Windows is quite a pickle. The official emacs builds don't
include GnuTLS, and getting GnuTLS installed is tricky (msysgit doesn't
come with it; you need a full Cygwin, it looks like).

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[glyph]

imo windows users are on their own.

I think that we may have to say that for now; this is a complicated puzzle. http://www.lonecpluspluscoder.com/2015/08/adding-tls-support-to-emacs-24-5-on-windows/ suggests that it's not all that hard to get a build of GnuTLS going, but … that suggests downloading it from Sourceforge, which, guess what, doesn't offer HTTPS on its downloads, so GnuTLS itself becomes a vector for an unauthenticated binary. Ugh!

[nyambol]

"imo windows users are on their own."

Then put a sign on the front page of melpa, telling windows users up front that your site is broken, unless they manually patch their version of emacs, to work around it. Why make them spend the time and effort to troubleshoot a problem you already know exists?

[milkypostman]

so does it not work at all?

On Sun, Dec 6, 2015 at 2:47 AM, Michael Powe notifications@github.com
wrote:

"imo windows users are on their own."

Then put a sign on the front page of melpa, telling windows users up front
that your site is broken, unless they manually patch their version of
emacs, to work around it. Why make them spend the time and effort to
troubleshoot a problem you already know exists?

—
Reply to this email directly or view it on GitHub
#2109 (comment)
.

[nyambol]

Not in cygwin, not with the standard Windows binary. I found at least 3 different sets of "instructions" for manually installing GnuTLS, none of which worked. Apparently, there is a rather delicate combination of dependencies that must be met.

The standard win32 emacs does not have gnutls built in, according to ldd. The cygwin-64 emacs shows that it's there, but emacs itself does not see it.

powem@powem-dell-win7 ~
$ !ldd
ldd which emacs | grep tls
cyggnutls-28.dll => /usr/bin/cyggnutls-28.dll (0x3fd5d0000)

Testing in emacs:

Debugger entered--Lisp error: (void-function open-gnutls-stream)
(open-gnutls-stream "tls" "tls-buffer" "melpa.org" "https")
eval((open-gnutls-stream "tls" "tls-buffer" "melpa.org" "https") nil)
eval-last-sexp-1(nil)
eval-last-sexp(nil)
call-interactively(eval-last-sexp nil nil)
command-execute(eval-last-sexp)

Chasing this stuff down can turn into a real rabbit hole. I'm trying to avoid falling in. I have in the past built emacs on linux, and that in itself can be a monster operation. I have no desire to do it in winblows, as I'm sure it's a route to a drinking problem.

thanks.

[milkypostman]

c5fbe5a makes a note about this.

[glyph]

My blog post at https://glyph.twistedmatrix.com/2015/11/editor-malware.html covers Windows - although I've only tested these instructions with the standard Windows binary, not Cygwin.

[tarsius]

Support HTTPS

We do. Is there anything left to be done here?

[tarsius]

Should the link from https://github.com/melpa/melpa to http://melpa.org changed to https?

[milkypostman]

done.