AWS ElasticSearch service

[mathom]

With the new AWS ElasticSearch service they have the HTTP requests authenticated via IAM. With a couple tweaks I got a requests sigv4 plugin to work for this: tedder/requests-aws4auth#2

The resulting code is pretty succinct (see the PR for how I'm connecting to ES) so maybe this would be a good addition to the documentation? Is anyone working on anything related to this?

[ipartola]

+1. For whatever silly reason Amazon rolled out their ES service without supporting most of their IAM authentication mechanisms. Authorizing servers by public IP is silly, and the only real option is to sign requests to the ElasticSearch endpoint. I think ideally elasticsearch-py should provide some way to augment requests with additional headers so that the AWS request signing authentication can be tacked on without making it a part of this project directly.

[mathom]

Yep I think IAM signing is the way to go. We're in VPC so IP auth is not an option.

[honzakral]

Thank you for this, I will mention it in the documentation with other environment considerations.

[honzakral]

Is there a canonnical example I can use for the documentation I could use?

[ipartola]

Here's more or less what I got working:

Step 1: Install requests-aws4auth from @mathom's fix branch (https://github.com/mathom/requests-aws4auth/tree/es_fix) at least until tedder/requests-aws4auth#2 is merged.

Step 2: Initialize your connection to AWS ES like so:

import elasticsearch

host = 'YOURHOST.us-east-1.es.amazonaws.com'
awsauth = AWS4Auth(your_access_key, your_secret_key, region, 'es')

es = elasticsearch.Elasticsearch(
    hosts=[{'host': host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=elasticsearch.connection.RequestsHttpConnection
)
print(es.cluster.health())

[honzakral]

Thanks! I will have to wait for the patch to be merged then before referring to it, I will leave this ticket here open as a reminder for both me and anyone looking for a solution.

[mathom]

Just a heads up, the required changes have been merged and released to pypi in aws4auth 0.7

[streeter]

Has anyone else had any issues with Amazon's ES service? It seems like when Amazon changes the underlying IP for the hostname, my connections stop working. I'm connecting with the hostname (not IP), and every once in a while the connections stop returning any results until I restart the webserver. However, if I connect from the same box, via a python shell (with the same settings), the connection returns results.

My hunch is that somewhere the IP is getting cached so when Amazon moves the ES cluster by pointing the DNS to a new IP, we're still using the old connection. Any thoughts on how best to fix this?

[mathom]

@streeter we restart our web workers after so many requests (gunicorn option, I believe) so we haven't encountered this.

[streeter]

@mathom Thanks. Was just investigating that option for us.

[honzakral]

Fixed via a7a56b1

Thank you!