SNI support for login.{persona,anosrep}.org for multiple SSL virtual hosts on one IP

[lmorchard]

It seems that prod and stage Persona do not support the SNI extension for SSL. SNI would allow me to host multiple domains with SSL on the same IP address (ie. my single VM for hobby projects).

Instead, it looks like Persona gets the same default SSL cert from my server for all domains - and that's only correct for one of my domains (i.e. whuru.com), so only one of my domains works with Persona. That includes running a full-on IdP or even for delegating one of my personal domains to another IdP.

If the Mozilla infra can't be changed to support SNI, this would be handy to note in docs.

FWIW, I've been able to detect that my server is using SNI with curl -3. That could also be handy for implementers to know why their IdPs aren't working. It was really confusing to me, because otherwise my copy of cURL supports SNI, thus obscuring the issue.

[lmorchard]

FWIW, RE: docs, I just added this as a tip to the page on MDN.

[lloyd]

at the very least we can improve error messaging to help IdP developers isolate the issue.

[ozten]

+1

If we could get check_primary_support running in production and accessible as a webtool, that would be a good bang for debugging buck.

Maybe the go2market crew can take a stab at that?

[djc]

Not sure what version of Node.js is used for Persona, but it looks like this has been fixed upstream:

nodejs/node-v0.x-archive@d07f75a9e (landed in 0.8)

[lmorchard]

It doesn't seem like this is a node problem in production & staging, though. As I understand it, there's a squid proxy cache in front of the service that takes care of fetching service docs & etc, and I'd guess the SNI support is missing there. Persona running in dev and on my laptop doesn't have this issue.

[djc]

Ah, I guess the proxy could be the problem, as well. Still, Persona on your laptop working doesn't preclude this from being a Node version issue. As for SNI support in Squid, I found the following:

http://wiki.squid-cache.org/Features/BumpSslServerFirst
http://comments.gmane.org/gmane.comp.web.squid.devel/15201

Which seems to indicate that SNI support should be better in the upcoming 3.3 version of Squid.

[chemistrydioxide]

I think that SNI should be made a MUST to implement for Persona consumers/verification services. I mean, nobody can afford a dedicated IP for each of their domain names. Even large organizations will eventually want to run their own Persona IdP for each domain they serve (think Google Apps for your domain) - and putting like 1000 customers into 1 multi domain cert doesn't sound like a good idea either.

[callahad]

SNI is unsupported in IE 8 on Windows XP. Allowing SNI would allow IdPs to lock out IE8/XP users. Then again, I don't really see a problem with giving IdP's that ability and making sure that scripts/check_primary_support, checkmyidp.org, and the docs issue warnings about it.

Hm. @gene1wood, any idea what would be involved in making this happen? Should it wait until after / be part of the migration to AWS?

[callahad]

(If it's simple enough to Just Do It Now, that would be pretty great, too)

[chemistrydioxide]

I don't think that SNI would even cause a hard lockout for IE8/XP users.
Instead, the server would send them a wrong certificate and the user
would have to accept that. Alternatively, SNI could be added to IE8
(using a plugin/proxy/whatever). Or XP users could use a different
browser.

Mit freundlichen Grüßen / With kind regards
ChO₂

ChO₂ Email: chemistrydioxide AT lodaya DOT im

-------- Weitergeleitete Nachricht --------
Von: Dan Callahan notifications@github.com
Reply-to: mozilla/browserid <reply
+i-7527795-3b34526b3b822f5da31a786afdb4da395b997c1d-3464006@reply.github.com>
An: mozilla/browserid browserid@noreply.github.com
Kopie: chemistrydioxide chemistrydioxide@lodaya.im
Betreff: Re: [browserid] SNI support for login.{persona,anosrep}.org for
multiple SSL virtual hosts on one IP (#2583)
Datum: Sat, 09 Feb 2013 12:48:16 -0800

(If it's simple enough to Just Do It Now, that would be pretty great,
too)

—
Reply to this email directly or view it on GitHub..

[callahad]

Training users to accept invalid certificates is the last thing we want to do. ;)

[gene1wood]

It sounds like we're talking about the production and staging persona server side components being able to initiate connections out to and IdP which has its support document, provisioning page and authentication page being served over SSL from a webserver taking advantage of SNI.

The entity that would need to have SNI client support enabled then would be the part of staging or production persona which initiates connections out to interact with these pages.

If these connections traverse our squid proxy servers then the method used is that node.js establishes an http connection to the proxies which in turn establish an SSL connection out to the IdP

https://github.com/mozilla/browserid/blob/fe731c56ea3604ba79566a5da796be4c25ff404f/lib/primary.js#L143

We run squid-3.1.10-1.el6_2.4.x86_64 in staging and production. SNI support appears to have been added to squid in 3.2.0.8

http://bazaar.launchpad.net/~squid/squid/3.2/view/head:/src/forward.cc#L767

Alternatively, browserid could begin using the CONNECT method to tunnel the request through the proxies

http://muffin.doit.org/docs/rfc/tunneling_ssl.html

thereby putting the obligation of handling SNI on node.

[lmorchard]

"It sounds like we're talking about the production and staging persona server side components being able to initiate connections out to and IdP which has its support document, provisioning page and authentication page being served over SSL from a webserver taking advantage of SNI."

Yes, exactly that. This issue is not so much about end-users' browsers as Persona itself using SNI for the HTTP requests it makes.

[callahad]

This issue is not so much about end-users' browsers as Persona itself using SNI for the HTTP requests it makes.

Which isn't to say that end-user browsers are not also important -- they needs to load both the authentication and provisioning pages -- but that's correct. This issue is about getting SNI working in staging and production, for outbound requests originating on our servers.

[skomorokh]

In some ways this represents a barrier to a diverse ecosystem of identity providers---it reduces the ability for SAs and hobbyists to experiment with small scale deployments since they'll need to dedicate an IP for it. Certainly it's a big part of why I'm not running one.

What separates Persona from being just another monolithic single point of failure is the ability for completely independent entities to host your id. Important to make that process as inviting as possible!

[lmorchard]

Not to pile on, but it's frustrating that this issue is going on 6 months old now. Needing an IP for each identity provider - even the ones delegating to another provider - is pretty much what stopped my tinkering cold. This issue blocks me from doing much more, because it's not worth been worth the trouble yet.

[chemistrydioxide]

Is there any chance that this gets fixed in the near future? I'm still waiting for the day when I can start my own IdP.

[chemistrydioxide]

Is there any problem with Squid 3.2 that forces Persona to stick with Squid 3.1? If no, why is v3.1 still being used?

[lloyd]

/cc @gene1wood What are the chances we could conflate a migration to squid 3.2 with our AWS move?

[callahad]

We need to be careful with enabling SNI heterogeneously across data centers. Wouldn't want to break IdPs during a failover.

[gerv]

Note: it's not "IE8 on Windows XP" which doesn't support Persona, it's any version of IE, or anything else which uses platform crypto (so Chrome?), on Windows XP. http://en.wikipedia.org/wiki/Server_Name_Indication

Given that XP still makes up a scarily large percentage of worldwide machines, and XP is still supported for 1 more year, people relying on this need to be very sure of their target market.

[lmorchard]

FWIW, a few Mozilla sites already use SNI. But, this issue isn't about Persona serving things up with SNI - it's about Persona being liberal in what it accepts and working with sites that have chosen to use SNI. Personally, I don't much care whether IE on XP works with my hobby sites. Others might, and if they do, they can choose not to use SNI. Either way, it would be really nice if Persona supported both.

[lloyd]

@callahad I think if we had heterogenous support for a short period, the failure rate would be sufficiently high that no one would rely on it. But we could roll out in a single config change after we're 100% aws.

Either way, bottom line were looking at about two months before this is resolved...

[ozten]

Getting feedback from some Wizards at Open Source Bridge that no SNI makes Persona a non-starter for them.

(plus everyone contributing to this bug :) )

What is our plan? Does AWS use Squid? Did we move to 3.2 yet? Should we switch to CONNECT for a Node based workaround?

Anything @fmarier and I can do this weekend to help?

[jcea]

Trying to deploy Persona Identity Provider and yes, this is a showstoper :-(.

Do we have a target date to get this issue solved?.

[gene1wood]

The impediment at the moment to moving to Squid 3.2 is the absence of a RHEL/CentOS/ScientificLinux blessed RPM. I'll look into building based on this work : http://repo.ngtech.co.il/rpm/centos/6/x86_64/

[jcea]

ETA?. I need SNI support for deploying a 3.000 members Persona verifier.

If I can be of any help, please, let me know.

[gene1wood]

We have a Squid 3.2 RPM now thanks to @mostlygeek and I need to find how to enable it. At the moment on squid 3.3.5 with this code

echo "import httplib
conn = httplib.HTTPConnection('localhost', 8888)
conn.request('GET', 'https://mozillafoundation.org/.well-known/browserid')
req = conn.getresponse()
print req.status, req.reason
for h in req.getheaders():
 print h
print req.read()" | python

I'm getting this error from squid

The system returned: <i>(71) Protocol error
This proxy and the remote host failed to negotiate a mutually acceptable security settings for handling your request. It is possible that the remote host does not support secure connections, or the proxy is not satisfied with the host security credentials.

[gene1wood]

@lmorchard Those pastebin links you posted expired. What were some examples of mozilla sites using SNI, I'd like to test against them.

[gene1wood]

Here's a test site we can use while we work on this : https://sni.velox.ch/

[lmorchard]

Hmm, let's see if I can remember. Basically anything on the generic cluster in phx. For example, this:

curl -3 -vsi -X HEAD https://air.mozilla.org/

Yields this:

* About to connect() to air.mozilla.org port 443 (#0)
*   Trying 63.245.217.86...
* connected
* Connected to air.mozilla.org (63.245.217.86) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
*    subject: serialNumber=I1oZ-yurbFWiPu/xP8HIzW3f9mApQvMv; C=US; ST=California; L=Mountain View; O=Mozilla Corporation; OU=IT; CN=tbpl.mozilla.org
*    start date: 2011-11-27 19:32:01 GMT
*    expire date: 2013-11-29 20:26:17 GMT
*    subjectAltName does not match air.mozilla.org
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
* SSL peer certificate or SSH remote key was not OK

[lmorchard]

From the Mana page, I'm guessing these are some more random examples:

• blog.lizardwrangler.com
• brendaneich.com
• careers.mozilla.org
• dnt.mozilla.org
• firefoxflicks.mozilla.org

[djc]

Any further progress on this? Anything I can do to help?

[gene1wood]

@djc I've gotten squid upgraded but have not figured out how to enable SNI. I've built the new rpm, updated the config to match and tested but am still encountering problems : https://github.com/mozilla/identity-ops/commits/master/chef/cookbooks/persona-proxy

[Lennie]

Hi Gene,

Did you get it working yet ?

Today I just installed a VM with CentOS 6.4 and checked the OpenSSL client and check if it worked: openssl s_client -port 443 -host hacks.mozilla.org -servername hacks.mozilla.org lists the hacks.mozilla.org cert and openssl s_client -port 443 -host hacks.mozilla.org would list the tbpl.mozilla.org cert.

So I installed the RPM for Squid 3.3.5 you linked to on your commit. Then I started a 'tcpdump -lvA -s 16000 -npti eth0 port 443 and dst hacks.mozilla.org | grep --line-buffered hacks.mozilla.org' to check if it would send the servername in cleartext in the TLS-connection and I send the query 'GET https://hacks.mozilla.org/' on port 3128 and that also worked.

So I think it works, what results did you get ?

If it did not work for you are you using CentOS 6.4 or an older version ? Did you use the RPM or did you build your own based on the SRPM ?

[callahad]

@gene1wood ping?

[gene1wood]

! Ok @Lennie you're right it does work. The SSL problems I was seeing must have been related to mozillafoundation.org before they changed their cert. I've tested with this

echo "import httplib
conn = httplib.HTTPConnection('localhost', 8888)
conn.request('GET', 'https://sni.velox.ch/.well-known/browserid')
req = conn.getresponse()
print req.status, req.reason
for h in req.getheaders():
 print h
print req.read()" | python

with great success. I've updated staging and you (and everyone else) can now test this in our staging environment. We should be able to get this out into production with our next production deploy.

[djc]

Tested this against my personal IdP. If I enable SSL on my primary domain [xavamedia.nl](the IdP is configured on ochtman.nl), I get these results:

https://login.persona.org/wsapi/address_info?email=dirkjan%40ochtman.nl (prod) says

{ "type": "secondary", "issuer": "login.persona.org", "disabled": false, "normalizedEmail": "dirkjan@ochtman.nl", "state": "offline" }

https://login.anosrep.org/wsapi/address_info?email=dirkjan%40ochtman.nl (staging) says:

{ "auth": "https://ochtman.nl/persona/authenticate", "prov": "https://ochtman.nl/persona/provision", "type": "primary", "issuer": "ochtman.nl", "state": "unknown" }

So, it seems to work as expected.

[lloyd]

wewt!

On Jul 16, 2013, at 10:44 AM, Dirkjan Ochtman notifications@github.com wrote:

Tested this against my personal IdP. If I enable SSL on my primary domain xavamedia.nl, I get these results:

https://login.persona.org/wsapi/address_info?email=dirkjan%40ochtman.nl (prod) says

{
"type": "secondary",
"issuer": "login.persona.org",
"disabled": false,
"normalizedEmail": "dirkjan@ochtman.nl",
"state": "offline"
}

https://login.anosrep.org/wsapi/address_info?email=dirkjan%40ochtman.nl (staging) says:

{
"auth": "https://ochtman.nl/persona/authenticate",
"prov": "https://ochtman.nl/persona/provision",
"type": "primary",
"issuer": "ochtman.nl",
"state": "unknown"
}

So, it seems to work as expected.

—
Reply to this email directly or view it on GitHub.

[gene1wood]

SNI support is now live in production. Feel free to re-open if you see any problem. Thanks for everyone's patience on this long lived ticket.

[jrgm]

verified current production:

curl https://login.persona.org/wsapi/address_info?email=dirkjan%40ochtman.nl
{
"auth":"https://ochtman.nl/persona/authenticate",
"prov":"https://ochtman.nl/persona/provision",
"type":"primary",
"issuer":"ochtman.nl",
"normalizedEmail":"dirkjan@ochtman.nl",
"state":"known"
}

[lmorchard]

And there was much rejoicing!

[ozten]

@lmorchard thanks for getting this party started!

[djc]

@jrgm that wasn't really a good test, since I'd disabled the SSL on the primary domain again. I've now re-enabled it, and it looks like it still works, so it's still fine. I can finally leave it enabled... Great work by everyone involved here.

[Lennie]

I didn't have time to test it, but I'm glad others did and it worked.

I don't have a domain setup yet for Persona, I just knew this was important, I would also probably need it if I would deploy it.

Maybe someone can answer me this:
If I know their might be users with browsers that don't support SNI, can I use SNI with HTTPS-delegation and point the user-portion to a non-SNI IdP HTTPS-site ?

So the user email address domain is X. Site/domain Y is it's IdP and the HTPS-site for X delegates to Y.

Will the user browser only talk to the Y IdP-site ?

[callahad]

@Lennie That's a really interesting approach. I think it would work.

You're really only talking about WinXP/IE8, which will never have a native implementation of BrowserID. Thus, lookup will be handled by the fallback, which does support SNI, and it will tell the browser to contact site Y.

I should test it, but as far as I know, that'd totally work.

[Lennie]

The Android 2.x default Browser is the other major browser that does NOT support SNI.

Anyway, I do still see some of our customers still using IE7 on WinXP, I know it's sad. But they are less and less frequent. You can ignore that last category. You already are. :-)

[gene1wood]

For the record I'm going to upgrade us to a slightly newer squid rpm because there's a bug in the init script in that 3.3.5 RPM.