Feature: Distributed SSL cache

[edevil]

A nice feature for the new https router would be the ability to cache SSL sessions, and share this cache across workers.

The next step would be a way to share this cache across uWSGI instances, which can reside in different machines.

I think this would have a big impact on performance.

[prymitive]

+1
This was discussed a little bit

[unbit]

The problem is in OpenSSL not supporting non-blocking access (or better, callback/evented-based management) to the cache. The problem is better described in the (broken) nginx memcached ssl-cache module. I have checked gnutls and it looks like it has the same problem. Basically the need is in the ability to "suspend" while waiting for infos for the cache. I have looked at Mongrel2 too (using PolarSSL) and it has the same problem (it preferred to not have a cache at all like in uWSGI). If i will not find a better library i will start investigating if the new (1.4) offload-subsystem can be useful to delegate caching to a bunch of threads.

[unbit]

Another solution could be implementing coroutines using setjmp/longjmp, basically the "cache_get" function called by OpenSSL will be 'suspended' to another stack and recovered when datas are ready. Will be ultra-complex but it looks like shared ssl caching is something needed by everyone, but implemented only in apache (as its architecture has no limits on blocking transactions). I will check if someone is interested in sponsoring it.

[prymitive]

Looks like stud did this differently:

bumptech/stud#29
bumptech/stud#50
bumptech/stud#60 <- this one was merged

Long story short (if I got it right) - when new session is created they send session data to other stud nodes via udp, so users sessions are kept in sync across all stud nodes.

[edevil]

The stud approach seems like a good one. That would leave us with only the fetch from a local cache to worry about. Is this still a problem? Maybe with a shared memory cache we can make it so there isn't any blocking.

[unbit]

Yes, cool idea, and in uWSGI we already have all of the pieces to implement it pretty easily. We could simply use the caching subsystem as extremely efficient storage (even in multiprocess mode). The only missing piece is updating the cache via udp, but i suppose a generic plugin will be enough

[prymitive]

How will one declare list of other nodes? Or maybe You are going to use --cluster feature and send those updates to multicast group (it works this way AFAIR)? Or maybe each node will fetch node list from FastRouter (if used)?

What will happen when adds new node? If we have one session per request than client should just negotiate new session without errror instead of quick old session reuse, we just need to disable HTTP keep-alive, do I get it right?

[unbit]

for clustering mode there is no problem as each packet is a multicast one (everytime a new session is created a multicast packet is spread on the network). But clustering mode cannot be used everywhere (network like amazon do not support it). My idea for unicast is in configuring the fastrouter/rawrouter/httpsrouter as a repeater. When a node create a new session it forward it to the repeater that send it to all of the nodes. You can have multiple repeaters (as you can have multiple fastrouter)

[unbit]

Ok, the first step has been made. Using the --ssl-sessions-use-cache you will tell to uWSGI to use its shared cache to store sessions data. The first advantage we get is that we can now use sessions in multiprocessing mode:

./uwsgi --cache 1000 --ssl-sessions-use-cache --master --https 0.0.0.0:8443,foobar.crt,foobar.key -w foobar -M --http-processes 8

will start 8 https processes with a shared ssl cache.

Next steps (help wellcomed):

• adding metrics for ssl cache misses and hits, this will be helpful to check if when we will go in clustering mode things works correctly
• send updates via UDP to cluster nodes in one of the (currently proposed) ways:

1. simply send updates via multicast:

--cache-udp-node 225.1.1.1:3031

2. send to static nodes

--cache-udp-node 192.168.0.1:3031 --cache-udp-node 192.168.0.2:3031

3. send to a repeater (supporting subscription system)

--cache-udp-node addr_of_the_repeater:3031

the repeater is a dumb processes simply redirecting udp packets to the subscribed nodes

[unbit]

A note: --cache and --ssl-sessions-use-cache have to be specified before the --https one. I now it is annoying, i will try to fix that soon or later...

[edevil]

This is great. This week I'll have to try both 1.4.2 and 1.5 dev. :)

As for the cache propagation method, 2) would suffice for me currently as each application has a small number of frontends. When this number increases I would like to try 1) but I don't know if my network supports it.

Thanks!

[unbit]

added a preliminary doc https://github.com/unbit/uwsgi-docs/blob/master/SSLScaling.rst

[unbit]

updated docs to specify what https-session-context is

[edevil]

FYI, I have been using the ssl session cache in development and it has been working very well.

[prymitive]

Does this one needs any additional work? It seems that all the pieces are in place for 1.9?

[unbit]

initial cache sync is broken as the cache remote plugin still need to be ported to the new api (it blocks on a lot of areas) i will work on that tomorrow (it is the last big piece missing)

[edevil]

I can't remember, was this finished?

[prymitive]

AFAIK cache sync works just fine with current master, 1.9.6 also seems to have it working. So I guess it is finished.

I tested it during #221 development and I did not spotted any issues.

[edevil]

Thanks! One less issue then. :)

[edevil]

There seems to be a problem, at least on 1.9.10. There are 2 ways to re-use an SSL session, sessions IDs with server side state and session tickets.

Session tickets appear to be working, but not all clients support it. Session IDs are not being returned in the Server Hello message.

We can test this with:

openssl s_client -no_ticket -connect xxx:443

We will obtain:

Session-ID: 

Whereas if we allow session tickets:

openssl s_client -connect xxx:443
...
Session-ID: C30541EC9B4766FDDC117114FCEB6E4CFD8D79C39C6DF3594AFB359C34F2BCCF

Now, the whole point of caching sessions server side is to return a session ID that can be reused later by a client. Session reuse by tickets does not require any server-side state. So, the logic should be (and it's what nginx does):

1- If the client announces its support for session tickets in the Client Hello message, send a session ticket and no session ID in the Server Hello message.
2- Otherwise, send a session ID and no session ticket to save bandwidth.

With this is mind, distributed ssl sessions are a problem with session tickets, since each node will be encrypting its ticket with a different self-generated key (uWSGI is generating a random key, right?) these tickets will not be valid on other nodes. So I propose a switch to disable session tickets and allow only session reuse through session ids which point to sessions stored in the distributed cache.

[unbit]

tickets should be disabled automatically in distributed cache:

https://github.com/unbit/uwsgi/blob/master/core/ssl.c#L281

[edevil]

You mean I have to recompile uWSGI with the SSL_OP_NO_TICKET option if I plan to have a distributed ssl session cache?

[unbit]

that define (SSL_OP_NO_TICKET) should be automatically available in recent openssl releases.

You may want to add a uwsgi_log("hello"); in it to check if it is called

[edevil]

I have this in my config:

    cache2: name=ssl,items=20000,blocksize=4096
    ssl-sessions-use-cache: ssl
    ssl-sessions-timeout: 300
    https-session-context: discoapi

But still "uwsgi.ssl_sessions_use_cache" is not set, so that code block is not executed.

[unbit]

be sure to set https: option after the cache definition (otherwise it will not be able to find it):

./uwsgi --cache2 name=pippo,items=1000 --ssl-sessions-use-cache=pippo --https :9090,foobar.crt,foobar.key

works, while

./uwsgi --https :9090,foobar.crt,foobar.key --cache2 name=pippo,items=1000 --ssl-sessions-use-cache=pippo

does not work because the cache does not exists when the https server is spawned

[edevil]

Ooops... Today I Learned. :)