Add SSL support to Netty transport layer for Client/Node-to-Node communication

[tlrx]

This pull request modifies the Netty transport layer in order to support SSL for client/node to node communication within an elasticsearch cluster.

Configuration

SSL encryption is activated with a new configuration entry transport.tcp.compress. If set to true, the keystore and trustore of the node must also be configured:

# Enable SSL/TLS encryption for all communication between nodes:
#
# transport.tcp.ssl: true

# Settings for SSL/TLS encryption
#
# transport.tcp.ssl.keystore: /path/to/the/keystore
# transport.tcp.ssl.keystore_password: password
# transport.tcp.ssl.keystore_algorithm: SunX509
#
# transport.tcp.ssl.truststore: /path/to/the/truststore
# transport.tcp.ssl.truststore_password: password
# transport.tcp.ssl.truststore_algorithm: PKIX

If keystores settings are not set in configuration file, it looks for JVM options (-Djavax.net.ssl.keyStore=..., -Djavax.net.ssl.trustStore=...)

Before starting the cluster, make sure that all nodes recognize each other (by importing the certificates in other nodes' trustores)

A generate.sh script can be used to create keystores for testing:

$./generate.sh 3     // will create 3 keystores and 3 certificates

Transport Client

SSL can be configured for a TransportClient with the Java API:

TransportClient client = new TransportClient(settingsBuilder()
                .put("client.transport.nodes_sampler_interval", "30s")
                .put("transport.tcp.ssl", true)
                .put("transport.tcp.ssl.keystore", "/opt/certificates/esnode1.jks")
                .put("transport.tcp.ssl.keystore_password", "mypassword")
                .put("transport.tcp.ssl.truststore", "/opt/certificates/esnode1.jks")
                .put("transport.tcp.ssl.truststore_password", "mypassword")
                .put("client.transport.sniff", false).build());
        
        client.addTransportAddress(new InetSocketTransportAddress("localhost", 9300));

Use cases

1. If a node tries to join a cluster but is not identified (certificate is unknown): discovery/recovery will fail, the node will start but won't join the cluster. The error message in log file is "org.elasticsearch.transport.SSLTransportException: SSL / TLS handshake failed, closing the channel" on cluster side.
2. If a non-SSL node tries to join a cluster configured for SSL: discovery/recovery will fail, the node will start but won't join the cluster. The error message in log file is "StreamCorruptedException: invalid data length".
3. If a transport client tries to connect to a cluster but is not identified: a NoNodeAvailableException is thrown.

Perf

One should pay attention to performance :/

NettyEchoBenchmark:

Warming up...
Warmed up
Ran 50000, TPS 9069.47215672048
Ran 50000, TPS 8939.746111210441
Ran 50000, TPS 10322.047894302228
Ran 50000, TPS 9238.728750923872
Ran 50000, TPS 10181.225819588679
Ran 50000, TPS 13319.126265316996
Ran 50000, TPS 9823.18271119843
Ran 50000, TPS 9529.25481227368
Ran 50000, TPS 9072.763563781527
Ran [500000] iterations, payload [100]: took [51], TPS: 9803.921568627451

NettyEchoSSLBenchmark:

Warming up...
Handshake completed on server side
Handshake completed on client side
Warmed up
Ran 50000, TPS 5730.002292000916
Ran 50000, TPS 6375.924509053813
Ran 50000, TPS 5571.651437486071
Ran 50000, TPS 6209.6373571783415
Ran 50000, TPS 5787.0370370370365
Ran 50000, TPS 5855.486590935707
Ran 50000, TPS 5726.720879624328
Ran 50000, TPS 5262.050094716901
Ran 50000, TPS 5755.726948313572
Ran [500000] iterations, payload [100]: took [87], TPS: 5747.126436781609

Code review and comments are welcome! :)

[dadoonet]

Very nice. I love the idea.

[kimchy]

Thanks for providing this pull request. The story around SSL on the transport layer is a bit more complicated though. If we add this, we need to think about allowing for cases where node to node communication does not use SSL, while clients do.

The overhead of node to node communication over SSL is huge, I don't really think people understand that ... . The value security wise is close to 0, to be honest, if you have proper firewall and other security measures in place.

[tlrx]

Thanks for the feedback. I understand the complexity that represents node to node communication over SSL. This pull request was a quick and easy way to get SSL within a cluster, as requested by many users.

[thejohnfreeman]

This seemed like the most promising proposal, but it was rejected. What does it take to get SSL into ES? Is it enough to have separate options for requiring SSL for connections initiated inside the node versus outside the node? What are your requirements, @kimchy ?

[mbarrien]

Quick note: Even though security value may be 0, having node-to-node communication be encrypted may still be required. Our specific case is that to store HIPAA information in Amazon and still be considered in compliance for auditing purposes, all patient data must be encrypted when sent over the wire. It's not strictly required by pure HIPAA regulations, but a HIPAA related contract (BAA) signed with Amazon does require it. Because this pull request was rejected, Elasticsearch fails to meet this requirement and thus can't be used.

HIPAA won't be the last place we'll see this requirement (I suspect a lot of government related Amazon work will be similarly covered). And since it's a bunch of legalese, no amount of pointing out how little actual security is gained or how firewalls and port isolation are enough will suffice to pass the lawyers and auditors scrutiny.

Please consider resurrecting this pull request!

[robottaway]

@mbarrien in the same boat here. Firewall rules alone won't cut it. We've been audited by expensive HIPAA experts and I can tell you the actual price of not having SSL transport layer, and it's definitely not 0.

@kimchy is the pull request going to hurt anyone if they actually implement it? At this point I may just have to clone ES and roll my own solution even if it's ugly. BTW the overhead looks completely acceptable for our setup if the figures @tlrx provided are in the ballpark.

[fatlotus]

@robottaway Yeah, I'd second that — seems silly to throw away @tlrx's efforts just because we think it was difficult to do.

[danielberlinger]

@kimchy I'd like to add a vote for this feature (despite the age of the issue). It would be incredibly helpful from a HIPPA/HITECH compliance perspective.

[MartinHatas]

+1000 for this feature

[anandsomani]

+1000 for this one!

[Mrc0113]

+100000....Also looking to have node to node encryption for HIPAA information in Amazon

[robottaway]

We have a repo containing the SSL enabled version of the 1.1.0 code here:

https://gitlab.devero.com/public/projects

there is also a SSL and QOS enabled RabbitMQ river plugin. On my radar to
cut releases in our Nexus so they can simply be dowloaded but for now you
have to "mvn package" to get the built product.

We plan to keep SSL branches as we move up versions of Elasticsearch or
until SSL transport becomes part if the project proper.

-rob

On Saturday, May 3, 2014, Marc DiPasquale notifications@github.com wrote:

+100000....Also looking to have node to node encryption for HIPAA
information in Amazon

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2105#issuecomment-42123510
.

[RobbieHer]

@robottaway 👍

Also, could you please confirm if the v1.1.0_ssl_work branch contains functionality to SSL protect inter-node communications on the transport layer (port 9300)? If so, could you please share some configuration settings, java client API usage and expected performance metrics?

Thanks!

[robottaway]

@RobbieHer encryption on 9300 is what this branch of ours provides. If you want TLS/SSL on the 9200 you will likely setup NGINX, Apache or another proxy in front which will allow setting up encryption. Note if you are going to use the Java API in your app you will need to configure encryption for 9300 there also since you cannot have both secured and unsecured.

[RobbieHer]

@robottaway Thanks!

Any idea on when this change can/will make it into the official ElasticSearch branch?

[robottaway]

You are very welcome! I would hope this could get into the official branch(es) some day! Past conversations being an indicator it doesn't seem likely though. It def is a bit of work to have to patch every branch we want to use, and it certainly keeps us from upgrading as quickly as we could.

[skurfuerst]

👍 for this feature! Checking it out right now, it would be awesome if this could be implemented in ES or an "official" module.