Support for AWS managed elasticsearch

[quintonm]

Does StageMonitor support using AWS managed ElasticSearch? The AWS version does not work with the normal java client. Only the REST API is exposed. I didn't really see anything in the docs to indicate if it would or would not work.

[felixbarny]

As stagemonitor only uses Elasticsearch's HTTP API I don't see a problem although I've never tried the AWS ES version. What is the version number of underlying ES server?

[quintonm]

2.3

[felixbarny]

Stagemonitor is compatible with 2.x as listed in the wiki: https://github.com/stagemonitor/stagemonitor/wiki/Installation#requirements

[quintonm]

Cool. I will try it out and update it if works as expected.

[ryanrupp]

When the AWS ES service originally was released I looked into it and the problem was the security mechanisms were pretty limited (at that time at least) in terms of making API requests to Elasticsearch. Seemed the access options were either whitelisting by IP or you have to sign the requests. If signing the requests is still a requirement Stagemonitor would have to provide some hookpoint do to this.

For reference, here's the issue where Zipkin added support for AWS Elasticsearch where they use an interceptor to do the signing as an example of a project doing this.

[felixbarny]

Have you already tried this out? Is signing the request still required or are there other options now?

[quintonm]

I haven't gotten to it yet. However, it will have some problems with the AWS managed elasticsearch. 1) The only way to implement security are ip whitelisting and request signing. Some people get around this by adding a proxy in front of elasticsearch to perform basic auth. This allows for the proxy to perform the request signing. It also give people a way to secure Kibana with basic auth. However, I think most would prefer to have support for request signing. 2) It is not possible to change the value of `threadpool.bulk.queue_size`. That means that you can't have a large number of JVMs all sending metrics directly to elasticsearch. Some options around this might be to add support for shipping metrics to logstash, writing to a logfile to be picked up by filebeats, kafka, etc.

…

On Mon, Dec 12, 2016 at 1:12 AM, Felix ***@***.***> wrote: Have you already tried this out? Is signing the request still required or are there other options now? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA8gtaYV_g3SJnlm9qK9xxLf1LywlIQsks5rHPPdgaJpZM4LFmaE> .