Find new TLS certificate vendor

[clone1018]

This morning I'm getting an SSL error for Gratipay.com

I'm running Google Chrome Version 57.0.2987.21 beta (64-bit)

[clone1018]

From Wikipedia:

On 24 October 2016, Mozilla announced on its security blog that, following its discovery of the purchase of StartCom by another Certificate Authority called WoSign during its investigation on numerous issues with that CA, and that both have failed to disclose this transaction,[12] Mozilla will stop trusting certificates that are issued after 21 October 2016 starting with Firefox 51.[13] On 1 November 2016, Google announced that it too would stop trusting certificates issued after 21 October 2016 starting with Chrome 56.[14] On 30 November 2016, Apple products will block certificates from WoSign and StartCom root CAs if the "Not Before" date is on or after 1 Dec 2016 00:00:00 GMT/UTC. [15]

[chadwhitacre]

Good catch, @clone1018. Here is Google's announcement (3+ months ago).

[clone1018]

@whit537 One thing to remember, if migration is the solution CloudFlare does offer SSL

[chadwhitacre]

Migration is clearly the solution. StartSSL is hosed.

[EdOverflow]

A dedicated SSL certificate on Cloudflare costs 5$ a month.

[chadwhitacre]

Re: Cloudflare: gratipay/inside.gratipay.com#957.

[chadwhitacre]

Also: https://letsencrypt.org/.

[chadwhitacre]

https://blog.heroku.com/announcing_heroku_free_ssl_beta_and_flexible_dyno_hours

[clone1018]

I've used LetsEncrypt and it was very nice. I wonder if there's a stigma to using a free SSL provider for a payment website

[chadwhitacre]

SNI should be fine from a browser support pov:

Desktop Browsers

• Firefox 2 and above
• Internet Explorer 7 on Windows Vista and above
• Windows Vista or OS X 10.6 with:
• Chrome 5.0.342.0
• Opera 14
• Safari 4

Mobile Browsers

• Mobile Safari on iOS 4.0 and above
• Android 4.0 (“Ice Cream Sandwich”) and above
• Windows Phone 7 and above

[chadwhitacre]

I wonder if there's a stigma to using a free SSL provider for a payment website

I think Let's Encrypt has a good reputation. No?

[clone1018]

I think the stigma is "You're a payment provider but you can't even afford a real secure cert?". I personally don't have an issue with it but I've heard that before.

[chadwhitacre]

Alright, I think the lowest-hanging fruit here is Let's Encrypt + Heroku SSL. We should probably take the $20/mo we'll save at Heroku and donate it to Let's Encrypt.

[chadwhitacre]

Would that help address the "can't even afford" issue, @clone1018?

[EdOverflow]

In my opinion, Let's Encrypt is perfectly fine.

[chadwhitacre]

We should probably take the $20/mo we'll save at Heroku and donate it to Let's Encrypt.
A dedicated SSL certificate on Cloudflare costs 5$ a month.

Sooooo ... $10/mo donation?

[chadwhitacre]

https://certbot.eff.org/docs/using.html#manual

[chadwhitacre]

It's a Python app! :-)

/etc/letsencrypt
total 0
drwxr-xr-x    2 root  wheel    68 Feb  9 12:01 ./
drwxr-xr-x  109 root  wheel  3706 Feb  9 12:01 ../
[gratipay] $ certbot certonly
Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.11.1', 'console_scripts', 'certbot')()
  File "/usr/local/Cellar/certbot/0.11.1/libexec/lib/python2.7/site-packages/certbot/main.py", line 865, in main
    make_or_verify_needed_dirs(config)
  File "/usr/local/Cellar/certbot/0.11.1/libexec/lib/python2.7/site-packages/certbot/main.py", line 821, in make_or_verify_needed_dirs
    os.geteuid(), config.strict_permissions)
  File "/usr/local/Cellar/certbot/0.11.1/libexec/lib/python2.7/site-packages/certbot/main.py", line 814, in make_or_verify_core_dir
    raise errors.Error(_PERM_ERR_FMT.format(error))
Error: The following error was encountered:
[Errno 13] Permission denied: '/var/lib/letsencrypt'
If running as non-root, set --config-dir, --work-dir, and --logs-dir to writeable paths.

[gratipay] $

[chadwhitacre]

Okay, I guess this is the way:

$ mkdir ssl
$ certbot certonly --config-dir=ssl --work-dir=ssl --logs-dir=ssl
Saving debug log to /Users/whit537/personal/gratipay/gratipay.com/ssl/letsencrypt.log

How would you like to authenticate with the ACME CA?
-------------------------------------------------------------------------------
1: Place files in webroot directory (webroot)
2: Spin up a temporary webserver (standalone)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel):vendors@gratipay.com

-------------------------------------------------------------------------------
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf. You must agree
in order to register with the ACME server at
https://acme-v01.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel: a

-------------------------------------------------------------------------------
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about EFF and
our work to encrypt the web, protect its users and defend digital rights.
-------------------------------------------------------------------------------
(Y)es/(N)o: n
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel):

[chadwhitacre]

P.S. We were only a month away from cert expiration anyway.

[chadwhitacre]

Doc to revise: http://inside.gratipay.com/howto/install-an-ssl-certificate.

[chadwhitacre]

I'm actually not too sad to be removing "Search 'startssl attempted phone call' in my personal Gmail" from our documentation. ☺️

[chadwhitacre]

Let’s Encrypt CA issues short-lived certificates (90 days). Make sure you renew the certificates at least once in 3 months.

https://certbot.eff.org/docs/using.html#renewing-certificates

That's gonna be a shift for us.

[chadwhitacre]

I guess we should go ahead and drop gittip.org (gratipay/inside.gratipay.com#877) now.

[chadwhitacre]

I'm putting together a config file that we can store and reuse.

[clone1018]

Renew with each payday :D?

[chadwhitacre]

@clone1018 Hehe. :-) Or maybe once a month, as part of payday duties?

[chadwhitacre]

Sorting through domain verification ... looks like there are a number of options. DNS would be the easiest, but afaict certbot --manual only supports simpleHttp(?).

[chadwhitacre]

A clue! certbot/certbot#4153

[chadwhitacre]

Actually, I think HTTP validation will be easier month-to-month than DNS validation. It'll be easier for us to update some config vars at Heroku than to modify DNS.

[chadwhitacre]

Can we enforce HTTPS though?

[chadwhitacre]

And will we get a single token/authorization for each domain (good), or different for each (bad)?

[chadwhitacre]

Because if we get a single pair for each domain, then that's fairly straightforward to store in Heroku config. If it's different for each domain then we may even want to put this in a dashboard page.

[chadwhitacre]

This looks to me like it's different for each domain:

(Pdb) pp achalls
[KeyAuthorizationAnnotatedChallenge(challb=ChallengeBody(chall=HTTP01(token='\x8a\xd4\xab\x05\xd9\x9fnD\x82\xc3\xdb\xef\xbe\xce0\x8b\x8b\xe2\x14\xd1\x8c\xb9\x90\xeaD\xc0\xc7\t(\x10\xfb\x14'), status=Status(pending), validated=None, uri=u'https://acme-v01.api.letsencrypt.org/acme/challenge/JOcG5i0--6DMuLIFSWV7J9GT5BLjVkB_Ah5RwmwlcAE/612995837', error=None), domain=u'gittip.co', account_key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x104eb28d0>)>)),
 KeyAuthorizationAnnotatedChallenge(challb=ChallengeBody(chall=HTTP01(token='h\xfd\x02\xec\xb5W\xf1\xc0`\xf9w\xb5e\xa1y\xc3\xda\xbaU\xf8@\xc0\x1c\x90\xa5\x01\xf9\xce\xf3\x0b\xa6J'), status=Status(pending), validated=None, uri=u'https://acme-v01.api.letsencrypt.org/acme/challenge/j-_1Q-DrKJT1SN2MWG-oq6SeCWZKFMwllTwipMdFPuE/612995876', error=None), domain=u'gittip.com', account_key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x104eb28d0>)>)),
 KeyAuthorizationAnnotatedChallenge(challb=ChallengeBody(chall=HTTP01(token="+\x92\xa6\xc1\xcb\x94\xb4\x86l\xc0'$\x85\\\xb6!\x9a\x03\xe2\x91\x1c4T\x98\xa8\x00\xa5\x12\xd2\xaeA\xbd"), status=Status(pending), validated=None, uri=u'https://acme-v01.api.letsencrypt.org/acme/challenge/Z8qfzcntShqbizpfbLQ7vxCEOERZe1OTdCR_jo66AdY/612995903', error=None), domain=u'gratipay.co', account_key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x104eb28d0>)>)),
 KeyAuthorizationAnnotatedChallenge(challb=ChallengeBody(chall=HTTP01(token='Pmx\x07\x18\x05\xc7tL\x18\xe2\xf6\xa7\x83\x7f\xd5\xda|v\xdd\xd4\xdf\x9cA\x10\xcf\xf5\x8e\xf4\x07\x89"'), status=Status(pending), validated=None, uri=u'https://acme-v01.api.letsencrypt.org/acme/challenge/E9Q4YOVQEHs4xVHu2TUpNQLk3eUf5GJRB7u6v7J74zQ/612995939', error=None), domain=u'gratipay.com', account_key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x104eb28d0>)>)),
 KeyAuthorizationAnnotatedChallenge(challb=ChallengeBody(chall=HTTP01(token='\x1dM\\Q\xd1\x7f\xc5i\xa5\x9c\x8e\xeaW\xeb\x9cqK[\xb6H\xfc\x9f\xec4_\xd6;B\xdd\x00\x17H'), status=Status(pending), validated=None, uri=u'https://acme-v01.api.letsencrypt.org/acme/challenge/-BvmSDRDf4y9ycDHDjw9521WouRKUY-iiYy-d8fXuz0/612995985', error=None), domain=u'gratipay.net', account_key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x104eb28d0>)>)),
 KeyAuthorizationAnnotatedChallenge(challb=ChallengeBody(chall=HTTP01(token='\x83\xf5\xed\xc4:\x9b\xc6\xca\xe2On\xaa\x0fPw\xb0\xd6\x98\x83U\xe8\xc6\xb3\xc05r\xe5\xb5z\r\xb1>'), status=Status(pending), validated=None, uri=u'https://acme-v01.api.letsencrypt.org/acme/challenge/R1iQHAcF7v5tBAGVF_40AkTOkiQ5wFN6IlpOAn_VPHc/612996018', error=None), domain=u'gratipay.org', account_key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x104eb28d0>)>))]
(Pdb)

That's at https://github.com/certbot/certbot/blob/v0.11.1/certbot/plugins/manual.py#L112.

[chadwhitacre]

Yeah, they're different. Okay!

(Pdb) continue

-------------------------------------------------------------------------------
Make sure your web server displays the following content at
http://gittip.co/.well-known/acme-challenge/itSrBdmfbkSCw9vvvs4wi4viFNGMuZDqRMDHCSgQ-xQ before continuing:

itSrBdmfbkSCw9vvvs4wi4viFNGMuZDqRMDHCSgQ-xQ.xUuLmhhsQVoA1eLx120E-nqe4fo-R277Gs2p71enaDs

If you don't have HTTP server configured, you can run the following
command on the target server (as root):

mkdir -p /tmp/certbot/public_html/.well-known/acme-challenge
cd /tmp/certbot/public_html
printf "%s" itSrBdmfbkSCw9vvvs4wi4viFNGMuZDqRMDHCSgQ-xQ.xUuLmhhsQVoA1eLx120E-nqe4fo-R277Gs2p71enaDs > .well-known/acme-challenge/itSrBdmfbkSCw9vvvs4wi4viFNGMuZDqRMDHCSgQ-xQ
# run only once per server:
$(command -v python2 || command -v python2.7 || command -v python2.6) -c \
"import BaseHTTPServer, SimpleHTTPServer; \
s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \
s.serve_forever()" 
-------------------------------------------------------------------------------
Press Enter to Continue
> /usr/local/Cellar/certbot/0.11.1/libexec/lib/python2.7/site-packages/certbot/plugins/manual.py(113)perform()
-> import pdb; pdb.set_trace()
(Pdb) c

-------------------------------------------------------------------------------
Make sure your web server displays the following content at
http://gittip.com/.well-known/acme-challenge/aP0C7LVX8cBg-Xe1ZaF5w9q6VfhAwByQpQH5zvMLpko before continuing:

aP0C7LVX8cBg-Xe1ZaF5w9q6VfhAwByQpQH5zvMLpko.xUuLmhhsQVoA1eLx120E-nqe4fo-R277Gs2p71enaDs

[chadwhitacre]

And do challenges survive redirects? Because we canonicalize domains before we hit simplates.

[chadwhitacre]

I'm trying to hit the place where it seems like the challenge is issued. But I'm not able to yet.

If it's a requests.get ... I think it might follow redirects? 🙈

[chadwhitacre]

This is so skunkerific. 😩

[chadwhitacre]

It appears that simple_verify is unused. :-|

$ ack simple_verify acme certbot
acme/challenges.py
213:    def simple_verify(self, chall, domain, account_public_key):
273:    `simple_verify`.
280:    def simple_verify(self, chall, domain, account_public_key, port=None):
379:    `simple_verify`.
452:    def simple_verify(self, chall, domain, account_public_key,

acme/challenges_test.py
106:    def test_simple_verify_failure(self):
109:        verified = self.response.simple_verify(self.chall, "local", public_key)
112:    def test_simple_verify_success(self):
114:        verified = self.response.simple_verify(self.chall, "local", public_key)
178:    def test_simple_verify_bad_key_authorization(self):
180:        self.response.simple_verify(self.chall, "local", key2.public_key())
183:    def test_simple_verify_good_validation(self, mock_get):
186:        self.assertTrue(self.response.simple_verify(
191:    def test_simple_verify_bad_validation(self, mock_get):
193:        self.assertFalse(self.response.simple_verify(
197:    def test_simple_verify_whitespace_validation(self, mock_get):
202:        self.assertTrue(self.response.simple_verify(
207:    def test_simple_verify_connection_error(self, mock_get):
209:        self.assertFalse(self.response.simple_verify(
213:    def test_simple_verify_port(self, mock_get):
214:        self.response.simple_verify(
339:    def test_simple_verify_bad_key_authorization(self):
341:        self.response.simple_verify(self.chall, "local", key2.public_key())
344:    def test_simple_verify(self, mock_verify_cert):
347:            mock.sentinel.verification, self.response.simple_verify(
354:    def test_simple_verify_false_on_probe_error(self, mock_probe_cert):
356:        self.assertFalse(self.response.simple_verify(

acme/standalone_test.py
104:        return resource.response.simple_verify(

[chadwhitacre]

Okay, I realized that I had to confirm all six domains before certbot would start actually hitting them. Once I do the result suggests that it does follow redirects, since I'm getting a 404 error instead of a 302.

   Type:   unauthorized
   Detail: Invalid response from
   http://gittip.com/.well-known/acme-challenge/X68UjfeMEpMRcqhcXZbHp49Bjpbs7uTvz1mYWzHedq0:
   "<!DOCTYPE html>
   <html lang="en">
   <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">

       <title>404 - Eep! - Gra"

   Domain: gratipay.net
   Type:   unauthorized
   Detail: Invalid response from
   http://gratipay.net/.well-known/acme-challenge/0ob6DqTGWQKU0ZTkC9RH974qo30tg5CrDfdI354BOWQ:
   "<!DOCTYPE html>
   <html lang="en">
   <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">

       <title>404 - Eep! - Gra"

   Domain: gratipay.co
   Type:   unauthorized
   Detail: Invalid response from
   http://gratipay.co/.well-known/acme-challenge/7tmbvcPpPCtZ0VzNrZkmqUiM0iXqroVir7b3b4Tb-r8:
   "<!DOCTYPE html>
   <html lang="en">
   <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">

       <title>404 - Eep! - Gra"

   Domain: gratipay.com
   Type:   unauthorized
   Detail: Invalid response from
   http://gratipay.com/.well-known/acme-challenge/fY4mMTgma9MeKJfFUo-WSRJZLTpZpyAOI40DShwPgIY:
   "<!DOCTYPE html>
   <html lang="en">
   <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">

       <title>404 - Eep! - Gra"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.
$ 

[chadwhitacre]

Alright, the way the protocol works appears to be that we never directly hit http://gratipay.com/.well-known/acme-challenge/deadbeef from the client. My hunch is that that only ever happens on the server side, with the result communicated back to the client somehow for error reporting.

Press Enter to Continue
Waiting for verification...
> /usr/local/Cellar/certbot/0.11.1/libexec/lib/python2.7/site-packages/acme/client.py(643)_send_request()
-> return response
(Pdb) response
<Response [202]>
(Pdb) c
> /usr/local/Cellar/certbot/0.11.1/libexec/lib/python2.7/site-packages/acme/client.py(643)_send_request()
-> return response
(Pdb) response
<Response [200]>
(Pdb) c
Cleaning up challenges
Failed authorization procedure. gratipay.org (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://gratipay.org/.well-known/acme-challenge/6Sm2DpX4zwtR_LiINGmHR2BDGVrkESU6q1xCLEXQzIg: "<!DOCTYPE html>
<html lang="en">
<head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">
    
    <title>404 - Eep! - Gra"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: gratipay.org
   Type:   unauthorized
   Detail: Invalid response from
   http://gratipay.org/.well-known/acme-challenge/6Sm2DpX4zwtR_LiINGmHR2BDGVrkESU6q1xCLEXQzIg:
   "<!DOCTYPE html>
   <html lang="en">
   <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">

       <title>404 - Eep! - Gra"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.
$

[chadwhitacre]

Yeah, here it is:

(Pdb) print response.text
{
  "identifier": {
    "type": "dns",
    "value": "gratipay.org"
  },  
  "status": "invalid",
  "expires": "2017-02-16T23:47:57Z",
  "challenges": [
    {
      "type": "tls-sni-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3VquqWCWTV576rr4jOaIrIWd8ngMd1_EEZZm0AiTZjQ/614621449",
      "token": "l6ac6k4jQQvfdZ8eXTuc712bR1vIBBUWJVpmWApViW4"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3VquqWCWTV576rr4jOaIrIWd8ngMd1_EEZZm0AiTZjQ/614621450",
      "token": "ni6w5JrayWFjwgyK8SHLJB3KvXjOKKBE4oyY7YBGGgo"
    },
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:acme:error:unauthorized",
        "detail": "Invalid response from http://gratipay.org/.well-known/acme-challenge/VQIPCPDWj3qskyK7vVZWu-ROn-P6A7wOAuWzkwMyCoQ: \"\u003c!DOCTYPE html\u003e\n\u003chtml lang=\"en\"\u003e\n\u003chead prefix=\"og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#\"\u003e\n    \n    \u003ctitle\u003e404 - Eep! - Gra\"",
        "status": 403
      },
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3VquqWCWTV576rr4jOaIrIWd8ngMd1_EEZZm0AiTZjQ/614621451",
      "token": "VQIPCPDWj3qskyK7vVZWu-ROn-P6A7wOAuWzkwMyCoQ",
      "keyAuthorization": "VQIPCPDWj3qskyK7vVZWu-ROn-P6A7wOAuWzkwMyCoQ.xUuLmhhsQVoA1eLx120E-nqe4fo-R277Gs2p71enaDs",
      "validationRecord": [
        {
          "url": "http://gratipay.org/.well-known/acme-challenge/VQIPCPDWj3qskyK7vVZWu-ROn-P6A7wOAuWzkwMyCoQ",
          "hostname": "gratipay.org",
          "port": "80",
          "addressesResolved": [
            "54.225.168.249",
            "54.221.240.156",
            "23.23.160.184"
          ],
          "addressUsed": "54.225.168.249"
        },
[snip]

[chadwhitacre]

Alright, well, I guess that's enough to go on. The server-side (which I think is open source and written in Go?) appears to follow redirects. 👍

[chadwhitacre]

Okay! So let's build an admin capability to set verification token/authorization pairs per domain.

[chadwhitacre]

PR in #4331 ... will return here in a bit for the certbot side.

[chadwhitacre]

I just accidentally closed Chrome, and means that it updated when I reopened. I'm now on Chrome 56, but gratipay.com is still green for me.

[chadwhitacre]

We also need to account for grtp.co and assets.gratipay.com.

[chadwhitacre]

Moving to gratipay/inside.gratipay.com#1005 for the certbot side. Leaving this open to drive the deploy.

[chadwhitacre]

Alright, once #4333 is out we should be ready to roll. I think we should go in order of increasing risk:

• grtp.co
• assets.gratipay.com
• gratipay.com

[chadwhitacre]

Actually, let's go with gratipay/inside.gratipay.com#1005 ...