This repository has been archived by the owner. It is now read-only.

SNI support for login.{persona,anosrep}.org for multiple SSL virtual hosts on one IP #2583

Closed
lmorchard opened this Issue Oct 11, 2012 · 48 comments

Comments

Projects
None yet
Member

lmorchard commented Oct 11, 2012

It seems that prod and stage Persona do not support the SNI extension for SSL. SNI would allow me to host multiple domains with SSL on the same IP address (ie. my single VM for hobby projects).

Instead, it looks like Persona gets the same default SSL cert from my server for all domains - and that's only correct for one of my domains (i.e. whuru.com), so only one of my domains works with Persona. That includes running a full-on IdP or even for delegating one of my personal domains to another IdP.

If the Mozilla infra can't be changed to support SNI, this would be handy to note in docs.

FWIW, I've been able to detect that my server is using SNI with curl -3. That could also be handy for implementers to know why their IdPs aren't working. It was really confusing to me, because otherwise my copy of cURL supports SNI, thus obscuring the issue.

Member

lmorchard commented Oct 11, 2012

FWIW, RE: docs, I just added this as a tip to the page on MDN.

@ghost ghost assigned fmarier Oct 18, 2012

Contributor

lloyd commented Oct 18, 2012

at the very least we can improve error messaging to help IdP developers isolate the issue.

Member

ozten commented Oct 18, 2012

+1

If we could get check_primary_support running in production and accessible as a webtool, that would be a good bang for debugging buck.

Maybe the go2market crew can take a stab at that?

Member

djc commented Oct 22, 2012

Not sure what version of Node.js is used for Persona, but it looks like this has been fixed upstream:

nodejs/node-v0.x-archive@d07f75a (landed in 0.8)

Member

lmorchard commented Oct 22, 2012

It doesn't seem like this is a node problem in production & staging, though. As I understand it, there's a squid proxy cache in front of the service that takes care of fetching service docs & etc, and I'd guess the SNI support is missing there. Persona running in dev and on my laptop doesn't have this issue.

Member

djc commented Oct 22, 2012

Ah, I guess the proxy could be the problem, as well. Still, Persona on your laptop working doesn't preclude this from being a Node version issue. As for SNI support in Squid, I found the following:

http://wiki.squid-cache.org/Features/BumpSslServerFirst
http://comments.gmane.org/gmane.comp.web.squid.devel/15201

Which seems to indicate that SNI support should be better in the upcoming 3.3 version of Squid.

I think that SNI should be made a MUST to implement for Persona consumers/verification services. I mean, nobody can afford a dedicated IP for each of their domain names. Even large organizations will eventually want to run their own Persona IdP for each domain they serve (think Google Apps for your domain) - and putting like 1000 customers into 1 multi domain cert doesn't sound like a good idea either.

Owner

callahad commented Feb 9, 2013

SNI is unsupported in IE 8 on Windows XP. Allowing SNI would allow IdPs to lock out IE8/XP users. Then again, I don't really see a problem with giving IdP's that ability and making sure that scripts/check_primary_support, checkmyidp.org, and the docs issue warnings about it.

Hm. @gene1wood, any idea what would be involved in making this happen? Should it wait until after / be part of the migration to AWS?

Owner

callahad commented Feb 9, 2013

(If it's simple enough to Just Do It Now, that would be pretty great, too)

I don't think that SNI would even cause a hard lockout for IE8/XP users.
Instead, the server would send them a wrong certificate and the user
would have to accept that. Alternatively, SNI could be added to IE8
(using a plugin/proxy/whatever). Or XP users could use a different
browser.

Mit freundlichen Grüßen / With kind regards
ChO₂

ChO₂ Email: chemistrydioxide AT lodaya DOT im

-------- Weitergeleitete Nachricht --------
Von: Dan Callahan notifications@github.com
Reply-to: mozilla/browserid <reply
+i-7527795-3b34526b3b822f5da31a786afdb4da395b997c1d-3464006@reply.github.com>
An: mozilla/browserid browserid@noreply.github.com
Kopie: chemistrydioxide chemistrydioxide@lodaya.im
Betreff: Re: [browserid] SNI support for login.{persona,anosrep}.org for
multiple SSL virtual hosts on one IP (#2583)
Datum: Sat, 09 Feb 2013 12:48:16 -0800

(If it's simple enough to Just Do It Now, that would be pretty great,
too)


Reply to this email directly or view it on GitHub..

Owner

callahad commented Feb 10, 2013

Training users to accept invalid certificates is the last thing we want to do. ;)

Member

gene1wood commented Feb 11, 2013

It sounds like we're talking about the production and staging persona server side components being able to initiate connections out to and IdP which has its support document, provisioning page and authentication page being served over SSL from a webserver taking advantage of SNI.

The entity that would need to have SNI client support enabled then would be the part of staging or production persona which initiates connections out to interact with these pages.

If these connections traverse our squid proxy servers then the method used is that node.js establishes an http connection to the proxies which in turn establish an SSL connection out to the IdP

https://github.com/mozilla/browserid/blob/fe731c56ea3604ba79566a5da796be4c25ff404f/lib/primary.js#L143

We run squid-3.1.10-1.el6_2.4.x86_64 in staging and production. SNI support appears to have been added to squid in 3.2.0.8

http://bazaar.launchpad.net/~squid/squid/3.2/view/head:/src/forward.cc#L767

Alternatively, browserid could begin using the CONNECT method to tunnel the request through the proxies

http://muffin.doit.org/docs/rfc/tunneling_ssl.html

thereby putting the obligation of handling SNI on node.

Member

lmorchard commented Feb 11, 2013

"It sounds like we're talking about the production and staging persona server side components being able to initiate connections out to and IdP which has its support document, provisioning page and authentication page being served over SSL from a webserver taking advantage of SNI."

Yes, exactly that. This issue is not so much about end-users' browsers as Persona itself using SNI for the HTTP requests it makes.

Owner

callahad commented Feb 11, 2013

This issue is not so much about end-users' browsers as Persona itself using SNI for the HTTP requests it makes.

Which isn't to say that end-user browsers are not also important -- they needs to load both the authentication and provisioning pages -- but that's correct. This issue is about getting SNI working in staging and production, for outbound requests originating on our servers.

In some ways this represents a barrier to a diverse ecosystem of identity providers---it reduces the ability for SAs and hobbyists to experiment with small scale deployments since they'll need to dedicate an IP for it. Certainly it's a big part of why I'm not running one.

What separates Persona from being just another monolithic single point of failure is the ability for completely independent entities to host your id. Important to make that process as inviting as possible!

Member

lmorchard commented Mar 10, 2013

Not to pile on, but it's frustrating that this issue is going on 6 months old now. Needing an IP for each identity provider - even the ones delegating to another provider - is pretty much what stopped my tinkering cold. This issue blocks me from doing much more, because it's not worth been worth the trouble yet.

Is there any chance that this gets fixed in the near future? I'm still waiting for the day when I can start my own IdP.

Is there any problem with Squid 3.2 that forces Persona to stick with Squid 3.1? If no, why is v3.1 still being used?

Contributor

lloyd commented Mar 11, 2013

/cc @gene1wood What are the chances we could conflate a migration to squid 3.2 with our AWS move?

Owner

callahad commented Mar 11, 2013

We need to be careful with enabling SNI heterogeneously across data centers. Wouldn't want to break IdPs during a failover.

gerv commented Mar 22, 2013

Note: it's not "IE8 on Windows XP" which doesn't support Persona, it's any version of IE, or anything else which uses platform crypto (so Chrome?), on Windows XP. http://en.wikipedia.org/wiki/Server_Name_Indication

Given that XP still makes up a scarily large percentage of worldwide machines, and XP is still supported for 1 more year, people relying on this need to be very sure of their target market.

Member

lmorchard commented Mar 22, 2013

FWIW, a few Mozilla sites already use SNI. But, this issue isn't about Persona serving things up with SNI - it's about Persona being liberal in what it accepts and working with sites that have chosen to use SNI. Personally, I don't much care whether IE on XP works with my hobby sites. Others might, and if they do, they can choose not to use SNI. Either way, it would be really nice if Persona supported both.

Contributor

lloyd commented Mar 22, 2013

@callahad I think if we had heterogenous support for a short period, the failure rate would be sufficiently high that no one would rely on it. But we could roll out in a single config change after we're 100% aws.

Either way, bottom line were looking at about two months before this is resolved...

Member

ozten commented Jun 21, 2013

Getting feedback from some Wizards at Open Source Bridge that no SNI makes Persona a non-starter for them.

(plus everyone contributing to this bug :) )

What is our plan? Does AWS use Squid? Did we move to 3.2 yet? Should we switch to CONNECT for a Node based workaround?

Anything @fmarier and I can do this weekend to help?

jcea commented Jun 22, 2013

Trying to deploy Persona Identity Provider and yes, this is a showstoper :-(.

Do we have a target date to get this issue solved?.

Member

gene1wood commented Jun 24, 2013

The impediment at the moment to moving to Squid 3.2 is the absence of a RHEL/CentOS/ScientificLinux blessed RPM. I'll look into building based on this work : http://repo.ngtech.co.il/rpm/centos/6/x86_64/

jcea commented Jun 24, 2013

ETA?. I need SNI support for deploying a 3.000 members Persona verifier.

If I can be of any help, please, let me know.

Member

gene1wood commented Jun 25, 2013

We have a Squid 3.2 RPM now thanks to @mostlygeek and I need to find how to enable it. At the moment on squid 3.3.5 with this code

echo "import httplib
conn = httplib.HTTPConnection('localhost', 8888)
conn.request('GET', 'https://mozillafoundation.org/.well-known/browserid')
req = conn.getresponse()
print req.status, req.reason
for h in req.getheaders():
 print h
print req.read()" | python

I'm getting this error from squid

The system returned: <i>(71) Protocol error
This proxy and the remote host failed to negotiate a mutually acceptable security settings for handling your request. It is possible that the remote host does not support secure connections, or the proxy is not satisfied with the host security credentials.

@ghost ghost assigned mostlygeek Jun 25, 2013

Member

gene1wood commented Jun 25, 2013

@lmorchard Those pastebin links you posted expired. What were some examples of mozilla sites using SNI, I'd like to test against them.

Member

gene1wood commented Jun 25, 2013

Here's a test site we can use while we work on this : https://sni.velox.ch/

Member

lmorchard commented Jun 25, 2013

Hmm, let's see if I can remember. Basically anything on the generic cluster in phx. For example, this:

curl -3 -vsi -X HEAD https://air.mozilla.org/

Yields this:

* About to connect() to air.mozilla.org port 443 (#0)
*   Trying 63.245.217.86...
* connected
* Connected to air.mozilla.org (63.245.217.86) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
*    subject: serialNumber=I1oZ-yurbFWiPu/xP8HIzW3f9mApQvMv; C=US; ST=California; L=Mountain View; O=Mozilla Corporation; OU=IT; CN=tbpl.mozilla.org
*    start date: 2011-11-27 19:32:01 GMT
*    expire date: 2013-11-29 20:26:17 GMT
*    subjectAltName does not match air.mozilla.org
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
* SSL peer certificate or SSH remote key was not OK
Member

lmorchard commented Jun 25, 2013

From the Mana page, I'm guessing these are some more random examples:

  • blog.lizardwrangler.com
  • brendaneich.com
  • careers.mozilla.org
  • dnt.mozilla.org
  • firefoxflicks.mozilla.org
Member

djc commented Jul 8, 2013

Any further progress on this? Anything I can do to help?

Member

gene1wood commented Jul 8, 2013

@djc I've gotten squid upgraded but have not figured out how to enable SNI. I've built the new rpm, updated the config to match and tested but am still encountering problems : https://github.com/mozilla/identity-ops/commits/master/chef/cookbooks/persona-proxy

Lennie commented Jul 11, 2013

Hi Gene,

Did you get it working yet ?

Today I just installed a VM with CentOS 6.4 and checked the OpenSSL client and check if it worked: openssl s_client -port 443 -host hacks.mozilla.org -servername hacks.mozilla.org lists the hacks.mozilla.org cert and openssl s_client -port 443 -host hacks.mozilla.org would list the tbpl.mozilla.org cert.

So I installed the RPM for Squid 3.3.5 you linked to on your commit. Then I started a 'tcpdump -lvA -s 16000 -npti eth0 port 443 and dst hacks.mozilla.org | grep --line-buffered hacks.mozilla.org' to check if it would send the servername in cleartext in the TLS-connection and I send the query 'GET https://hacks.mozilla.org/' on port 3128 and that also worked.

So I think it works, what results did you get ?

If it did not work for you are you using CentOS 6.4 or an older version ? Did you use the RPM or did you build your own based on the SRPM ?

Owner

callahad commented Jul 15, 2013

@gene1wood ping?

Member

gene1wood commented Jul 15, 2013

! Ok @Lennie you're right it does work. The SSL problems I was seeing must have been related to mozillafoundation.org before they changed their cert. I've tested with this

echo "import httplib
conn = httplib.HTTPConnection('localhost', 8888)
conn.request('GET', 'https://sni.velox.ch/.well-known/browserid')
req = conn.getresponse()
print req.status, req.reason
for h in req.getheaders():
 print h
print req.read()" | python

with great success. I've updated staging and you (and everyone else) can now test this in our staging environment. We should be able to get this out into production with our next production deploy.

Member

djc commented Jul 16, 2013

Tested this against my personal IdP. If I enable SSL on my primary domain [xavamedia.nl](the IdP is configured on ochtman.nl), I get these results:

https://login.persona.org/wsapi/address_info?email=dirkjan%40ochtman.nl (prod) says

{ "type": "secondary", "issuer": "login.persona.org", "disabled": false, "normalizedEmail": "dirkjan@ochtman.nl", "state": "offline" }

https://login.anosrep.org/wsapi/address_info?email=dirkjan%40ochtman.nl (staging) says:

{ "auth": "https://ochtman.nl/persona/authenticate", "prov": "https://ochtman.nl/persona/provision", "type": "primary", "issuer": "ochtman.nl", "state": "unknown" }

So, it seems to work as expected.

Contributor

lloyd commented Jul 17, 2013

wewt!

On Jul 16, 2013, at 10:44 AM, Dirkjan Ochtman notifications@github.com wrote:

Tested this against my personal IdP. If I enable SSL on my primary domain xavamedia.nl, I get these results:

https://login.persona.org/wsapi/address_info?email=dirkjan%40ochtman.nl (prod) says

{
"type": "secondary",
"issuer": "login.persona.org",
"disabled": false,
"normalizedEmail": "dirkjan@ochtman.nl",
"state": "offline"
}

https://login.anosrep.org/wsapi/address_info?email=dirkjan%40ochtman.nl (staging) says:

{
"auth": "https://ochtman.nl/persona/authenticate",
"prov": "https://ochtman.nl/persona/provision",
"type": "primary",
"issuer": "ochtman.nl",
"state": "unknown"
}

So, it seems to work as expected.


Reply to this email directly or view it on GitHub.

Member

gene1wood commented Jul 18, 2013

SNI support is now live in production. Feel free to re-open if you see any problem. Thanks for everyone's patience on this long lived ticket.

@gene1wood gene1wood closed this Jul 18, 2013

Member

jrgm commented Jul 18, 2013

verified current production:

curl https://login.persona.org/wsapi/address_info?email=dirkjan%40ochtman.nl
{
"auth":"https://ochtman.nl/persona/authenticate",
"prov":"https://ochtman.nl/persona/provision",
"type":"primary",
"issuer":"ochtman.nl",
"normalizedEmail":"dirkjan@ochtman.nl",
"state":"known"
}

Member

lmorchard commented Jul 19, 2013

And there was much rejoicing!

Member

ozten commented Jul 19, 2013

@lmorchard thanks for getting this party started!

Member

djc commented Jul 19, 2013

@jrgm that wasn't really a good test, since I'd disabled the SSL on the primary domain again. I've now re-enabled it, and it looks like it still works, so it's still fine. I can finally leave it enabled... Great work by everyone involved here.

Lennie commented Jul 19, 2013

I didn't have time to test it, but I'm glad others did and it worked.

I don't have a domain setup yet for Persona, I just knew this was important, I would also probably need it if I would deploy it.

Maybe someone can answer me this:
If I know their might be users with browsers that don't support SNI, can I use SNI with HTTPS-delegation and point the user-portion to a non-SNI IdP HTTPS-site ?

So the user email address domain is X. Site/domain Y is it's IdP and the HTPS-site for X delegates to Y.

Will the user browser only talk to the Y IdP-site ?

Owner

callahad commented Jul 19, 2013

@Lennie That's a really interesting approach. I think it would work.

You're really only talking about WinXP/IE8, which will never have a native implementation of BrowserID. Thus, lookup will be handled by the fallback, which does support SNI, and it will tell the browser to contact site Y.

I should test it, but as far as I know, that'd totally work.

Lennie commented Jul 19, 2013

The Android 2.x default Browser is the other major browser that does NOT support SNI.

Anyway, I do still see some of our customers still using IE7 on WinXP, I know it's sad. But they are less and less frequent. You can ignore that last category. You already are. :-)

Member

gene1wood commented Jul 25, 2013

For the record I'm going to upgrade us to a slightly newer squid rpm because there's a bug in the init script in that 3.3.5 RPM.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.