Connection handler for Amazon ES

[TheDeveloper]

Here's a connector I created for using the client with Amazon ES: https://github.com/thedeveloper/http-aws-es

The connector handles request signing needed for Amazon ES. It is configurable through the client config (see the header comments in the code for instructions).

Perhaps you might find this useful for a reference implementation or for putting in the docs / examples for how to use the client with Amazon ES 😄

[tgroshon]

👍 I have been looking for something like this all day! Thanks for implementing it.

Now that ElasticSearch is a managed service in AWS, this would be a very useful process to document.

I also raised an issue on aws-sdk-js to see if they will be adding an ElasticSearch client that does request signing: aws/aws-sdk-js#754

[nackjicholson]

👍 This is great, I've also been looking for this all day. I'm still running into problems when trying to connect to my AWS ES instance from my ec2 machines though. How are you guys setting up your access policies and such in order to be able to hit your AWS ES cluster with the elasticsearch client?

I'm using AWS.Ec2MetadataCredentials to pick off the credentials of my IAM Role, and I'm still getting a 403 back.

  {
    "message": "The security token included in the request is invalid."
  }

http://stackoverflow.com/questions/33204997/aws-elasticsearch-service-iam-role-based-access-policy

I made this stackoverflow which goes into more detail about my setup if you're curious. Thanks for this module, and any help!

[TheDeveloper]

@nackjicholson are you getting this with v1.1 of the connector? I have just published the latest to npm so there is a chance you were getting an old version.

[spalger]

Thanks for putting this together @TheDeveloper. I would be happy to mention it in the documentation. Where do you think would be best? Perhaps a new extensions page that can talk about this plugin and potentially others in the future?

[TheDeveloper]

@spalger yup that sounds good. And perhaps a link to it on https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/extending_core_components.html for easy access to existing plugins.

[spalger]

Added a page about extensions in 3aefbbd, it should show up on the official docs within the hour (they're built automatically). Let me know if you want to make some changes to the short blurb I copied from the readme.

[nackjicholson]

@TheDeveloper Yes! Version 1.1.1 did fix my issue. Thanks a ton sir! I was starting to worry the Amazon ES service just didn't work with IAM Roles!

Here's how I tested it out, any way to get rid of that refresh call? Am I doing this right?

var AwsEsConnector = require('http-aws-es');
var AWS = require('aws-sdk');
var elasticsearch = require('elasticsearch');

var creds = new AWS.EC2MetadataCredentials();

creds.refresh(function(err) {
  if (err) throw err;

  var es = new elasticsearch.Client({
    host: 'search-my-domain.us-west-2.es.amazonaws.com',
    log: 'trace',
    connectionClass: AwsEsConnector,
    amazonES: {
      region: 'us-west-2',
      credentials: creds
    }
  });

  es
    .search({
      index: 'my-index',
      type: 'some-type'
    })
    .then(console.log); // logs the response
});

Also, I can relocate further questioning to an issue in the http-aws-es github if it's out of place here.

[TheDeveloper]

@spalger splendid. That looks just fine. Thank you!

@nackjicholson feel free to bring further discussion over to http-aws-es. For this particular case, if you want to avoid initialising the client in a callback, try to provide creds to your app synchronously, such as via environment variables.

[yellowandy]

Thank you, was looking for this all day!