Redis SSL support

[antirez]

HQ for ideas / proposals / code about Redis and SSL support.

[bbroerman30]

I'll be looking over the code I wrote over Christmas... I have Dec 18th through Jan 2nd off, so I should have plenty of time finish testing 2.6, and get 2.7 and 2.8 updates merged in. I'll also see if I can create a new branch directly from yours and merge my changes into it. (I'll be working on things in that order probably). Should I start with your 2.6, 2.7 or 2.8?

[antirez]

Thanks @bbroerman30, ideally the best thing is to start from the "unstable" branch. Back porting will not be hard and I can take care of it, but for new stuff it's ideal to hack on the development tree.

[bbroerman30]

I'll fork off of that then, and once I get 2.6 tested (and go through the code and remember what I did and why) I'll start looking at the unstable branch and making changes in there. I'll hold off on 2.7 and 2.8 updates on my side...

[tarcieri]

Just wanted to say thank you for opening this issue!

To anyone who might happen to ask "why not stunnel/stud/spiped/(insert proxy here)/etc" we operate nearly a hundred of these tunnels and have to manage the overhead of provisioning individual certificates, trust stores, adding new certificates to these trust stores for key rotation, ensuring that apps, tunnels, and Redis are started up and taken down in awareness of this fact, and have to implement key rotation within these constraints.

It's very easy to say "just use stunnel". If you have any actual experience maintaining a highly available, secure infrastructure using dozens or hundreds of stunnels, I think you'll quickly become aware the reality of the situation is: easier said than done.

Native SSL support would eliminate all sorts of operational overhead we have around encrypting all of our Redis connections.

Last but not least, I'll leave you the quotes reel from when I originally told my coworkers that there may be a solution on the horizon for native SSL support in Redis:

"This is great."
"YES DIE STUNNELS DIE"
"Nice!"
"DO THIS OH GOD PLEASE DO THIS"
"Very nice!"
"Yes yes yes."

These quotes come from people operationally responsible for maintaining our stunnel-based infrastructure. I want to make these people's lives easier.

[fdr]

Ditto the above, and doubly so for the client drivers. Server operators can (sort of) cope, but getting clients to use stunnel is a considerably bigger chore for all involved.

I'll happily take any implementation over none, but regardless here are some ideas I experimented with.

In a nutshell, I decided SCRAM-SHA1-PLUS with tls-server-end-point channel-binding would let me fix the most Redis issues at once with the least work. This delivers some compelling advantages when the TLS and Authentication mechanisms of an application can interact.

As for implementation, I wrote some prototype code for this (in Go, presuming it'd have to be a proxy anyway) and was getting some traction, but then had to go work on something else. If there's sufficient interest, I may revisit that decision. Regardless, here is why I made these choices:

• Detecting MITM is simplified.
• Doesn't break TLS concentrators.
• Avoids persistent breaks via challenge-response.

From first to last:

Detecting MITM is simplified

This is a feature of "channel-binding". In a nutshell, the pre-shared secret (password) is mixed with both the client and server's view of the TLS state (e.g. presented public key) on the connection. A mismatch (e.g. decrypt-and-re-encrypt with another key) will present as an authentication failure.

This is a very cool way to avoid the complexity of certificate pinning or CAs provided one continues to use the password/pre-shared-key model, as I suspect pre-shared-keys will remain very useful for a long time.

This is known by the lingo SCRAM-SHA1-PLUS. In my prototype, deviating from the standard, to only support this "PLUS" variant, and not the non-channel-bound version.

Doesn't break TLS concentrators

This has to do with the choice of tls-server-end-point.

In my prototype, deviating from the standard, I was going to skip the otherwise mandatory support for tls-unique, as with a proxy design my own program was itself akin to a TLS concentrator. What this amounts to is mixing in the fingerprint of the public key encrypting the channel into the HMACs passed around (see the "algorithm" section).

Avoids persistent breaks via challenge-response

If one manages to get ahold of a decrypted session, salted challenge-response avoids the long-lived password from being compromised. This is a common feature of older methods like DIGEST-MD5.

SCRAM has some useful properties, but the most useful is that it's simpler to implement than even venerable older methods. Otherwise interesting but not as important to me is that it supports holding securely derived keys on the server, so that one does not compromise the password if the server reads out its own passwd-ish file.

The algorithm

It is seen with explanation of the notation in the SCRAM rfc linked above, but here's a taste (and lets you find the section):

     SaltedPassword  := Hi(Normalize(password), salt, i)
     ClientKey       := HMAC(SaltedPassword, "Client Key")
     StoredKey       := H(ClientKey)
     AuthMessage     := client-first-message-bare + "," +
                        server-first-message + "," +
                        client-final-message-without-proof
     ClientSignature := HMAC(StoredKey, AuthMessage)
     ClientProof     := ClientKey XOR ClientSignature
     ServerKey       := HMAC(SaltedPassword, "Server Key")
     ServerSignature := HMAC(ServerKey, AuthMessage)

[bbroerman30]

Status Update: I have a clone of antirez/redis and am working on merging in the changes from my 2.6.17 branch into the trunk. I started with a 3-way diff, and merged everything I could from that, and then started compiling and working out all the compile errors. I have the initial merge done, and checked into my clone. It compiles, and I can connect with redis-cli but there are still critical errors reported by the self test. I'll be working over the next week (while i'm on vacation) to try and sort these out. After my vacation is over, I'll still be able to work some evenings and weekends until it's all working...

[badboy]

@bbroerman30 Is this work already public?

[tarcieri]

@bbroerman30 great to hear!

[bbroerman30]

Was able to generate a self-signed certificate, add it to the root store, connect a client to a server with SSL, verified that it would not connect without, verified it checked the certificate, connected a slave with SSL, and verified it checked the certificate and validated the configured common name... Also, ran the runtest-sentinel and runtest-cluster with minimal errors without SSL (the errors I had were probably due to the tiny box I'm running this on... it's a Athlon XP 2600 with 1G ram) I will run them with SSL over the next couple days.

[bbroerman30]

@badboy It's in my fork of redis (in the trunk). I checked in the changes last night to that repo. I have also been testing my sslredis 2.6.17 to make sure it's working properly as well... I'm using a copy of antirez's trunk for baselines on the unstable, and my sslredis 2.6.17 as the baseline for SSL...

[badboy]

Thanks for your work.

[bbroerman30]

still have work to do... unstable branch is having issues on initial replication to slave. 2.6.17 was working perfectly, so I'm trying to figure out the differences.

[bbroerman30]

ok, got initial replication happening (simple case, full dump) and continual replication between a master and 1 slave. Looking through the code, I think after I get it all working, I want to refactor... creating a structure that will replace the socket file descriptor. The structure should contain the fd, the anetSSLConnection, and a flag that indicates which to look at. Each place the fd (client.fd, server.fd, server.repl_transfer_s, etc.) is used it will be changed to this structure. I'm also thinking of a helper function that encapsulates SSL_write and write, as well we SSL_read and read. This new function will take the new structure.

[bbroerman30]

Having issues getting a cluster running... Same error with the standard (unmodififed) unstable version as with my SSL version. All of the instances of redis server crash with a core dump when I tell the ./redis-trib.rb script to save the configuration. I'm running on a small Athlon XP 2600 box with 1Gb of ram, running stock Ubuntu 14.10 and following the instructions from the Redis cluster tutorial. I only have a couple days left of vacation, and would like to get this initial code change validated.

I don't really want to get into a refactor without knowing that the existing code is working.

[badboy]

@bbroerman30 Can you figure it out yourself or is there anything we can help with?

[bbroerman30]

I'm pretty sure it's related to the small size of the linux box I have available... Do you have a box that has successfully run a small cluster before? If so, I would appreciate someone helping with running a simple cluster test

[badboy]

I've got some DigitalOcean credit left and access to other virtual servers. Just tell me what to do and I spin up some instances.

[bbroerman30]

Can you give it a quick try with regular redis unstable? The instructions i was following are here: http://redis.io/topics/cluster-tutorial

[badboy]

Will do in a moment

[badboy]

Created the cluster with redis-trib. Tested with redis-cli -c. Everything seems to work fine so far.

P.S.: I'm also available on IRC (freenode)

[bbroerman30]

@badboy Awesome. Can you try with the ssl version? Without SSL enabled. You can get it from my repo at https://github.com/bbroerman30/redis. I'm in and out today as it's the last day of my vacation, and I'm trying to get a bunch of other things done before I go back to the normal grind. I really appreciate your help!!!

[badboy]

Any hints for setting up the ssl cert and key?

Update: Sorry, over-read that I should test without SSL, even easier then.

[bbroerman30]

yeah. Setting SSL isn't too bad, except that the Ruby and Tcl clients aren't set up for SSL. I created my own self-signed private key and cert, then installed the cert into the trusted root store on the box. Setting up the trusted root store depends on the OS...

After that, I set up the redis.conf like:
ssl true # turn on SSL (makes all connections require ssl)
ssl_ca_root_dir /usr/share/ca-certificates # The trusted root store for openSSL.
ssl_cert_file /home/brad/redis/cacert.pem # The self-signed cert the server is running with.
ssl_pk_file /home/brad/redis/privkey.pem # The private key I created the cert with.
ssl_cert_pass Pidb95 # Password for the private key/cert.
ssl_dhk_file /home/brad/redis/dHParam.pem # Diffie-Hellman key file .
ssl_cert_common_name bbroerman.net # This is used by the slaves to validate the CN in the cert (as most slaves connect by IP and not name. This is optional)

[badboy]

Ok, got it compiled and running. 6 instances (3 master, 3 slave), but for some reason I can't create a cluster, it hangs on CLUSTER MEET. (but no crash)

[badboy]

Ok, it does crash. Upstart was just fast enough to respawn the process without me noticing.
Log output: https://gist.github.com/badboy/9e16cf070de82c9309be

[badboy]

For some reason this gets called even though I did not enable ssl:
https://github.com/bbroerman30/redis/blob/unstable/src/cluster.c#L1928

[bbroerman30]

That error should be fixed now. I got a bit more picky (as i should have been initially) on the initial setting of link->ssl's attributes on createClusterLink() and setting it's properties when an SSL connection is made. I wish I could run it over here (even the original unmodified unstable won't run cluster on my box)

[badboy]

It works now as expected. redis-trib can create a cluster and I can set keys as expected.

Your local machine must be really underpowered if it doesn't work. I could give you a small virtual machine to test on if you want.

[bbroerman30]

Thanks. I may have time tomorrow to run through tests. The box I have here is one of the kids old computers. Athlon XP 2600 with 1G ram, running Ubuntu 14.10 32 bit.

[chester89]

@itamarhaber what are your thoughts on the matter? Is this feature not high enough on the list of priorities of dev community? I see the paid hosted version doesn't have SSL support either (because using stunnel is just a workaround in my opinion)

[itamarhaber]

@chester89 my thoughts on the matter have not changed - I'd still like to see SSL supported by Redis.

[chester89]

@antirez and yours?

[phenomax]

Would you aim for full SSL support: Securing all connections (even between redis instances), or just client to server?

[geek0x23]

For our use cases, we need full mutual TLS (server and client certificates) for all traffic. This means both normal client traffic and redis-to-redis internal traffic. We also need the ability to configure the OpenSSL engine, if OpenSSL is used. A PKCS#11-based solution would be even better than OpenSSL, though.

[madolson]

@SWAJ
"We also need the ability to configure the OpenSSL engine, if OpenSSL is used" What type of configuration are you looking for? Which ciphers and version are being used?
"A PKCS#11-based solution would be even better than OpenSSL, though." Why is this solution better for your use case?

[geek0x23]

@madolson PKCS#11 is prefered, because we typically store private keys inside hardware security modules. PKCS#11 is the standard that basically every crypto device follows (smart cards, hardware security modules, etc.). PKCS#11 is also used by a few software-based crypto tokens. Mozilla's NSS is probably the most common.

OpenSSL with engine support is an alternative for us, because OpenSSL has the CHIL engine which can interface with our hardware security modules. This engine basically delegates to the hardware security module for many operations. The problem with the CHIL engine, is that it only supports RSA keys right now. We tend to use a lot of EC keys, so CHIL isn't usable in those scenarios.

OpenSSL engine support is relatively straightforward. A good example of how to implement it can be found in Node.js. They expose an API (crypto#setEngine) that's pretty straightforward.

The source implementing this functionality can be found here:

https://github.com/nodejs/node/blob/84e02b178ad14fae0df2a514e8a39bfa50ffdc2d/src/node_crypto.cc#L6241

Edit: To provide further clarification

[collimarco]

I would really like to see SSL supported by Redis: using stunnel is a pain.

Not to mention that using stunnel is so complex that even providers like Heroku suggest partial solutions that are not secure: heroku/heroku-buildpack-redis#15

[madolson]

We have an update here. A group Redis core developers got together to talk about features for Redis and we got agreement from antirez to to get TLS merged in with minor modifications. We are in the process of updating #4855 to address his concerns

[itamarhaber]

What @madolson said - I'm dying to retire this post: https://redislabs.com/blog/secure-redis-ssl-added-to-redsmin-and-clients/ :P

[chester89]

Is there any progress on this @madolson?

[tarcieri]

@chester89 see #4855

[yossigo]

The latest effort for SSL/TLS support is making a lot of progress, see #6236

[chester89]

@yossigo you sure you mentioned the correct issue?

[yossigo]

@chester89 my bad, corrected. thanks!

[moredure]

Is it possible to configure client to use only CA file (redli for example) to connect to Redis with TLS support enabled or Redis with TLS verify client certificate?

9141:M 24 Nov 2019 11:09:56.539 # Error accepting a client connection: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate

Update: Sorry, I found --tls-auth-clients option for server

[yaxing]

@yossigo great progress thanks! Do we have plan on releasing it? (6.x?) @antirez

The latest effort for SSL/TLS support is making a lot of progress, see #6236

[itamarhaber]

@yaxing the plan is to release it as part of v6.

[satheeshaGowda]

Thanks a lot @itamarhaber for the great news and all the great work to putting it together for v6. Greatly appreciate if you may shed some light on v6 release timelines as well.

[itamarhaber]

As far as I know the 1st release candidate for v6 is perhaps a couple of weeks away - Dec/Jan would be my bet. Then, it would take perhaps a couple of iterations to get the final version of of the door, so I'd say around Feb/March for GA... But that's just my speculating... times will tell and soon we shall know for sure :)

…

On Mon, Dec 9, 2019, 04:21 Satheesha CH Gowda ***@***.***> wrote: Thanks a lot @itamarhaber <https://github.com/itamarhaber> for the great news and all the great work to putting it together for v6. Greatly appreciate if you may shed some light on v6 release timelines as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2178>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOHL7BMSAYQUA56IADNHJ3QXWTR3ANCNFSM4AYKXSQQ> .

[satheeshaGowda]

sounds great, thanks @itamarhaber

[clenimar]

great work!

I have built 6.0 with TLS but I'm facing the following error (on redis-cli):

Could not negotiate a TLS connection: Failed to create SSL_CTX
Could not negotiate a TLS connection: Failed to create SSL_CTX

whilst the server shows:

26534:M 28 Jan 2020 14:19:40.313 # Error accepting a client connection: (null)
26534:M 28 Jan 2020 14:19:40.313 # Error accepting a client connection: (null)

the runtest --tls and runtest-cluster --tls suites show a lot of OKs, but also the error I mentioned (Failed to create SSL_CTX). the cluster suite always breaks while trying to reshard. I'm referring to the instructions in [1].

am I missing something?

cheers.

[1] https://github.com/antirez/redis/blob/6.0/TLS.md

[yossigo]

Hi @clenimar, can you provide some more details on what platform you're using? Specifically, what version of OpenSSL you use and whether or not you have some specific configuration or a custom build?

[clenimar]

Hi @yossigo, I'm using Ubuntu 16.04. OpenSSL was 1.0.2g, and upgrading it to 1.1.1 seemed to solve the problem (now I can connect). the problem I'm facing now is regarding cluster creation. the command output keeps "Waiting for the cluster to join .............................." forever.

I've got three servers running in containers (now I'm using upstream redis:6.0-rc image) in the same node and the following command for cluster creation:

$ ./src/redis-cli --cluster create 10.5.0.21:6379 10.5.0.21:6380 10.5.0.21:6381 --tls --cert tests/tls/redis.crt --key tests/tls/redis.key --cacert tests/tls/ca.crt
...
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..........................................

one of the servers:

385:C 29 Jan 2020 17:02:43.031 # Redis version=5.9.101, bits=64, commit=00000000, modified=0, pid=385, just started
385:C 29 Jan 2020 17:02:43.031 # Configuration loaded
385:M 29 Jan 2020 17:02:43.033 * No cluster configuration found, I'm 32566ad8a3ae13aa57593529c80c338e90acdff2
...
385:M 29 Jan 2020 17:02:43.088 * Ready to accept connections
385:M 29 Jan 2020 17:09:03.129 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH

thank you for your time.

cheers.

[yossigo]

@clenimar Apparently redis-cli --tls was broken with older OpenSSL versions, fixed in #6808.

As for the cluster problem, I was not able to reproduce that. What configuration did you use?

[clenimar]

@yossigo thanks!

3 servers are spawned with the following command (each one in a different container):

$ redis-server --cluster-enabled yes --port 0 --tls-port 6379 --tls-cert-file /conf/redis.crt --tls-key-file /conf/redis.key --tls-cluster yes --tls-ca-cert-file /conf/ca.crt

then I trigger cluster creation (the standard configuration) from the host machine, using the redis-cli:

$ ./src/redis-cli --cluster create 10.5.0.60:6379 10.5.0.60:6380 10.5.0.60:6381 --tls --cert tests/tls/redis.crt --key tests/tls/redis.key --cacert tests/tls/ca.crt
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: f465692e5a3ffcc6d4b23a961901c4da55e20908 10.5.0.60:6379
   slots:[0-5460] (5461 slots) master
M: 3db11c6e3064c585bf592ab7af7f807663534d47 10.5.0.60:6380
   slots:[5461-10922] (5462 slots) master
M: 5ccbe645315b4d8c275b10d6843adeeb914efca2 10.5.0.60:6381
   slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
....................................................................

is this the configuration you meant in your comment?

cheers.

[yossigo]

@clenimar I've just double checked this with the official redis:6.0-rc container on Ubuntu 16.04 and it works fine.

Please double check your address and port configuration and validate it works without TLS first. Also, note that depending on how you set up your docker networking you may need to explicitly expose the cluster bus port and/or use the cluster-announce-* settings because Redis may not be aware to the address or port it is exposed on.

[clenimar]

@yossigo you're right. there was a dumb bug in my deployment script preventing the cluster ports to be appropriately exposed. everything works fine now. thank you very much!