Making signed requests to Elasticsearch

[alexrudd]

Hi,

I'm writing a python lambda function to process incoming ELB logs and put them into an Elasticsearch domain. So far the boto3 library has been great, but I'm struggling to find out how to either make my bulk index post through the client or how to sign a direct rest api call.

I thought the generate_presigned_url call might be what I want, but I can't find out what is a valid ClientMethod

Is there an easy way to sign an outgoing request through the boto3 library using assumed iam role credentials?

Thanks,
Alex

[kyleknap]

There is not really such a functionality. We would like to in the future expose a generate_presigned_request for such purposes, especially since the generate_presigned_url is limited in that it can only return a url and does not include the request body. Marking as a feature request.

[alexrudd]

Thanks!

For anyone else who comes across this problem, here's what I ended up doing:

import boto3
from aws_requests_auth.aws_auth import AWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection

session = boto3.session.Session()
credentials = session.get_credentials().get_frozen_credentials()

es_host = 'search-my-es-domain.eu-west-1.es.amazonaws.com'
awsauth = AWSRequestsAuth(
    aws_access_key=credentials.access_key,
    aws_secret_access_key=credentials.secret_key,
    aws_token=credentials.token,
    aws_host=es_host,
    aws_region=session.region_name,
    aws_service='es'
)

# use the requests connection_class and pass in our custom auth class
es = Elasticsearch(
    hosts=[{'host': es_host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(es.info())

[tylersmith34]

@alexrudd The script you posted saved the day! I'd been searching for a week and many different scripts before I came across yours. I wanted to migrate from one domain to another and the Boto2 based script AWS provided didn't work. Thank you!!!!!!

[AnishKhobragade]

@alexrudd Awesome

[karthikeyansundararajan-agi]

I copy pasted the exact same code but got this

Traceback (most recent call last):
  File "/Users/karthik/Documents/Github/devops-tf-generate/connect-to-es.py", line 27, in <module>
    print(es.info())
  File "/Users/karthik/.pyenv/versions/3.9.1/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/Users/karthik/.pyenv/versions/3.9.1/lib/python3.9/site-packages/elasticsearch/client/__init__.py", line 294, in info
    return self.transport.perform_request(
  File "/Users/karthik/.pyenv/versions/3.9.1/lib/python3.9/site-packages/elasticsearch/transport.py", line 413, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch)
  File "/Users/karthik/.pyenv/versions/3.9.1/lib/python3.9/site-packages/elasticsearch/transport.py", line 630, in raise_error
    raise UnsupportedProductError(message)
elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch

[tedder]

It was intentionally broken by Elastic:
https://www.theregister.com/2021/08/09/elasticsearch_python_client_change/
elastic/elasticsearch-py#1623