New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making signed requests to Elasticsearch #853

Open
AlexRudd opened this Issue Oct 14, 2016 · 4 comments

Comments

Projects
None yet
4 participants
@AlexRudd

AlexRudd commented Oct 14, 2016

Hi,

I'm writing a python lambda function to process incoming ELB logs and put them into an Elasticsearch domain. So far the boto3 library has been great, but I'm struggling to find out how to either make my bulk index post through the client or how to sign a direct rest api call.

I thought the generate_presigned_url call might be what I want, but I can't find out what is a valid ClientMethod

Is there an easy way to sign an outgoing request through the boto3 library using assumed iam role credentials?

Thanks,
Alex

@kyleknap

This comment has been minimized.

Member

kyleknap commented Oct 14, 2016

There is not really such a functionality. We would like to in the future expose a generate_presigned_request for such purposes, especially since the generate_presigned_url is limited in that it can only return a url and does not include the request body. Marking as a feature request.

@AlexRudd

This comment has been minimized.

AlexRudd commented Oct 17, 2016

Thanks!

For anyone else who comes across this problem, here's what I ended up doing:

import boto3
from aws_requests_auth.aws_auth import AWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection

session = boto3.session.Session()
credentials = session.get_credentials().get_frozen_credentials()

es_host = 'search-my-es-domain.eu-west-1.es.amazonaws.com'
awsauth = AWSRequestsAuth(
    aws_access_key=credentials.access_key,
    aws_secret_access_key=credentials.secret_key,
    aws_token=credentials.token,
    aws_host=es_host,
    aws_region=session.region_name,
    aws_service='es'
)

# use the requests connection_class and pass in our custom auth class
es = Elasticsearch(
    hosts=[{'host': es_host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(es.info())
@tylersmith34

This comment has been minimized.

tylersmith34 commented Jan 17, 2018

@AlexRudd The script you posted saved the day! I'd been searching for a week and many different scripts before I came across yours. I wanted to migrate from one domain to another and the Boto2 based script AWS provided didn't work. Thank you!!!!!!

@AnishKhobragade

This comment has been minimized.

AnishKhobragade commented Jan 23, 2018

@AlexRudd Awesome

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment