You can clone with
This would help with those that are hosting elasticsearch and need security when making calls from another machine which cannot be used with the TCP transport.
HTTPS support should allow for configuration of the SSL certificate from a path, along with any other easily discoverable certificate sources.
I second this. I have legal requirements to protect private patient information in transit and at rest. Native HTTPS, or SSL support would help solve a good piece of this problem.
+1 - an encrypted transport is required at my client site.
Related to this would be locking down ES node to node communication over SSL as well.
Yep - I'd like both an HTTPS entry point and SSL inter-node comms.
I agree +1 for me too.
The content you are editing has changed. Reload the page and try again.
Attach images by dragging & dropping or
Uploading your images…
Unfortunately, we don't support that file type.
Try again with a PNG, GIF, or JPG.
Yowza, that's a big file.
Try again with an image file smaller than 10MB.
This browser doesn't support image attachments.
We recommend updating to the latest
Google Chrome, or
Something went really wrong, and we can't process that image.
I don't think there's a real need for supporting this in ElasticSearch itself? Most installations where this is neccessary could easily use a reverse proxy such as Nginx?
I think data is very sensitive and shouldn't rely on validation from third parties.
The data is in ES, ES should do the auth since it knows it's own data structure the most.
The simplicity of native encrypted transport is highly attractive. The complexity and fragility of wrapping all data in transit through an SSL proxy would be a barrier.
SSL at least ... I mean it's an http transport layer not some abstract peice of code.
Well since it seems people want this but dons't sound like its something you want to build in directly ... maybe a plugin would be good? I looked at Nginx and while it looks supper light weight and pretty clean it would add another virtual hop and, when you already have the extra weight of SSL it would be nice to have the ability to access ES directly with the least amount of extra overhead. I wonder if http://www.openssl.org/ would be helpful in integrating.
i guess a simple tutorial on how to set up nginx would suffice. nginx is not that hard to setup.
I was wondering if anyone was working on this issue, if not I would like to take a shot.
I don't believe so (and I think Shay has a lot of other things going on smiles). I'd contribute, but I'm a .NET programmer by trade, and don't have the skills in Java to make an effective contribution on this issue.
If you're going to take a shot, note, that they really have to go hand-in-hand; HTTPS is nice, but without some kind of authentication, you don't gain much for it. Additionally, you'll have to come up with a way to specify the credentials (username/password) of the users that can access the cluster.
I don't have any use-case for HTTPS (i use Nginx as SSL offloader to great effect), but needed HTTP Basic. You can find my plugin here:
My two cents on SSL: Supporting SSL does not only mean implementing all the nuts and bolts (certificate management, etc.), it also has to be efficient and safe. There are other projects (stunnel, nginx, your favourite hardware load balancer) that are much better at doing all that. If you want your elasticsearch to speak ssl externally without configuring nginx - bind it to a local port and put stunnel in front of it. This is a common and tested solution.
@skade: That isn't always feasible; especially when you are on a cloud infrastructure. As an example, Azure, which I've been able to get it to run on, won't let me install any of those.
It might be common and tested, but it's not always applicable.
@skade Your suggestion for using nginx or stunnel is fine from a whiteboard perspective, but nginx isn't going to deal with cluster communications, and stunnel can be a bit of an operational mess.
@BenHall I'd be interested in lending a hand on the issue as well I would see it as touching two core components, the transport module and the http module.
@prb: That's interesting -- could you elaborate why an Nginx-based proxy (for example) isn't enough? (I don't understand the nginx isn't going to deal with cluster communications part. You can restrict the in-cluster communication based on IPs.)
Elasticsearch uses the network for both external communications (e.g., over HTTP via the http module or via the transport module) and internal communications (e.g., shuttling data between nodes, cluster membership, etc.). All of that communication should be secured, and not all of it is over HTTP.
@prb: You can disable http on nodes entirely, and they will communicate via transport, which is not open, or? You can restrict how nodes communicate. At EC2, you can further restrict access to 9200/9300 to certain IPs, etc. So I still don't get why is auth needed here. Of course, something like Nginx-based proxy is only meant for authorizing access from outside the cluster.
+1 This is a major deficiency from a government security requirements perspective, and may be a blocker for our project moving from Solr to ElasticSearch, since we have our Solr shards locked down via tomcat.
@asanderson you can deploy elasticsearch as a war file within a wen container if you want using the wares plugin (check the transport-wares repo). Questions in the mailing list.
@kimchy Excellent! Other than performance, are there any other disadvantages to deploying via war?
@asanderson not really, same API, quite simply wrapping as well. If you are up to it, would love to get async support to the servlet based on the servlet 3.0 async feature :)
@kimchy good to know.
One of these days, I'd love to contribute, but right now I don't have the bandwidth.
Sorry, but the best I can do is to continue to evangelize ElasticSearch. ;-)
FWIW, I've spent the last 5+ years replacing expensive COTS products with Solr, and I must say that ElasticSearch out-of-the-box seems to address all of Solr's shortcomings, IMHO.
Now, I get to spend the next year or so replacing Solr with ElasticSearch. ;-)
We just released Jetty plugin for elasticsearch. It is a drop-in HTTP transport replacement that exposes full power of embedded Jetty including support for SSL, logging, authentication and access control. It is similar to transport-wares plugin that Shay mentioned above, but instead of running elasticsearch inside a web server, it embeds Jetty web server into elasticsearch. See https://github.com/sonian/elasticsearch-jetty for more information.
Outstanding! Great job! It just keeps getting better and better. ;-)
So I can set up a reverse proxy or use the Jetty plugin to secure access to elasticsearch, but now I can no longer use the Java API...
There's been some good progress on this issue from plugin contributors (thanks imotov and kimchy) but I've not yet seen anything that addresses the inter node cluster communications. HTTP proxies like Apache and Nginx won't cut it there, as the communications between nodes are lower level.
So, any form of movement on that front?
For those using the jetty plugin for SSL:https://github.com/sonian/elasticsearch-jetty
You can also utilize the Chef cookbook to speed-up your AWS deployments:https://github.com/pulkitsinghal/cookbook-elasticsearch
I've made a pull request for SSL support in transport client and node-to-node communications:#2105
(This does not concern the HTTPS authentication)
This is a weird coincidence, I wrote something similar during the last week. Its much worse in the configuration department, but it uses a different approach: it makes NettyTransport fit for implementing the SSL functionality as a plugin.
Its a bit different in some regards, but the broad direction is the same.
Has anybody tried running the elasticsearch transport over ssh forwarding or a VPN? Or, for that matter, the http transport. If so, any comments about performance impact?
This is search engine, there is no point to have HTTPS support, you can organize it at proxy-webserver level.
Absolutely disagree. It is a DISTRIBUTED search engine and there are legitimate security requirements to support SSL across the distributed nodes even if the entire node set is behind a firewall. Trust me. We have clients that require it. Considering that Netty supports SSL and ElasticSearch is built upon Netty, why is this a big deal?
... especially in the world of multi-tenant cloud environments. ;-)
Perhaps more of a philosophical question, but why is it important to have these features baked in, if they're already provided by the elasticsearch-jetty project? Works great for us...
Many new "big data" systems are built under the assumption that security is handled at a higher-level. MongoDB does not support authentication between nodes (correct me if I am wrong, it has been years since I used MongoDB) and even the earliest versions of Hadoop did not have security between nodes/HDFS.
Clients requiring a feature does not equate it to being an essential feature. If that were the case, I have a few essential features that ElasticSearch needs to support as well. :) I have used embedded Jetty app security before, and it works well. Security could/should be expanded on, but I rather see other features worked on first.
If you only require https or very simple authentication then elasticsearch-jetty solves the issue.
If you want true authorization and authentication then you need something more. Probably hits those of us using ES as a db and/or multi-tenant more then others. Not that it matters but Mongo and others now support this, but I'm sure its a large ES change under the covers to do at the index level and stop data leakage.
@devoncrouse It's about having to support YET ANOTHER software configuration item which needs to be patched and/or upgraded for this one feature whereas we use Apache Tomcat for everything else. Again, ElasticSearch is built on Netty, and Netty supports SSL, so why wouldn't ElasticSearch leverage that support out-of-the-box.
@brusic Not in the government sector where public key infrastructure (PKI) is required and security certification & accreditation requires encryption between every node. So, it's not about a single customer requiring a feature; it's about a barrier-to-entry for an entire potential customer vertical sector. ;-)
@awick It's about supporting public key infrastructure (PKI) between nodes. My impression was that it would not be a big ES change, since Netty already supports SSL. So, my impression was that it would be another ES optional set of configuration properties that expose the Netty-supported configuration. Of course, YMMV. ;-)
I'm having a devil of a time getting zen transport to work with forwarded ports. If I'm interpreting the tea leaves correctly - I have remotenode connecting to masternode via a forwarded port 9300 - which then immediately tells remotenode that the masternode is to be reached by a completely different IP address.
Master binds to: 192.168.1.30:9300
Client forwards the above to 127.0.2.2:9300 via an ssh tunnel.
Client's elasticsearch has conf file set to find unicast master at ["127.0.2.2:9300]
Client's elasticsearch succeeds in connecting to that port and talks to master.
Master then says master is 192.168.1.20, which is unreachable, client dumps stacktrace.
I'm with @asanderson here. My installation needs encryption between nodes because users on the internal network cannot be trusted. It's not just a nice feature to have, it is prohibiting adoption of ES at all.
I second what thejohnfreeman is saying. I cannot implement ES into our environment until I can send and receive from it over an encrypted connect.
How does network.publish_host help?
Sorry @thejohnfreeman, that was directed to @dangarthwaite. publish_host allows you to set the host/port that ES publishes to other nodes, instead of the host/port that it is bound to. Which should fix his issue with port forwarding.
I see, thank you for the clarification.
The Jetty plugin doesn't have the proper CORS response headers. Specifically when it comes to an OPTIONS request. It would be really awesome to have this feature. Especially when we also have Kibana at play here. It only makes sense.
You maybe want have a look here: https://github.com/salyh/elasticsearch-security-plugin (Do not really offer basic authentication but goes beyond it). Supports yet PKI and Kerberos authentication (and authorization through LDAP) for the REST layer.
+100 for this feature.