how to connect with amazon ES service

[yodaiken]

Hi—

I'm trying to use mongo-connector with Amazon's new hosted Elasticsearch (http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/what-is-amazon-elasticsearch-service.html).

I believe that it's not using normal Elasticsearch Shield etc, but rather is using Amazon's rather complex authentication scheme.

Any tips for getting mongo-connector to work?

Best,
Aaron

[llvtt]

There are two ways of passing arguments to the internal Elasticsearch client:

1. Pass options encoded in the url (-t option)
2. Pass options in the configuration file. Using clientOptions in this dictionary allows you to pass in individual options. For example, "args": {"clientOptions": {"foo": XXX}} sets foo=XXX in the constructor for the Elasticsearch client.

[yodaiken]

I think the issue is that you need a lot of special headers (which change throughout the session?) to use Amazon's authentication service -- ideally there would be some way to get Elasticsearch to connect via Amazon's SDK.

[consulthys]

As commented here, the only real option is to allow the es_doc_manager to talk to an IAM-secured Amazon Elasticsearch cluster and the only way to go is that the embedded Elasticsearch Python client be empowered to do so by means of some additional configuration option or simply with another doc manager dedicated to the Amazon Elasticsearch service.

[llvtt]

You can pass arbitrary arguments to the Elasticsearch client's constructor
with the following option:
https://github.com/mongodb-labs/mongo-connector/wiki/Usage-with-ElasticSearch#customizing-behavior-of-the-elastic-client

You should be able to accomplish what was described on elasticsearch-py's
Github page with that option using the mongo-connector configuration file.

On Sat, Oct 31, 2015 at 11:07 PM, Valentin Crettaz <notifications@github.com

wrote:

As commented here http://stackoverflow.com/a/33459628/4604579, the only
real option is to allow the es_doc_manager to talk to an IAM-secured
Amazon Elasticsearch cluster and the only way to go is that the embedded
Elasticsearch Python client be empowered to do so
elastic/elasticsearch-py#280 by means of some
additional configuration option or simply with another doc manager
dedicated to the Amazon Elasticsearch service.

—
Reply to this email directly or view it on GitHub
#337 (comment)
.

[yodaiken]

Hi @llvtt -- as far as I can tell, mongo-connector only allows passing arbitrary primitives as arguments. As you can see in @consulthys' link (elastic/elasticsearch-py#280 (comment)) to support AWS we need to a) import a new package and b) pass elasticsearch a reference to an instance of a class from that package.

Am I missing something?

[llvtt]

@yodaiken You're totally right; this doesn't look currently possible with mongo-connector. This could be included in a future Elastic DocManager if there's enough interest in having this, but right now I'd like to keep mongo-connector itself as lean as possible and encourage others to write their own DocManagers as plugins, rather than adding the kitchen sink to the project.

Adding and setting up requests_aws4auth looks relatively straightforward, though. I'd recommend creating an AWS4Auth instance and grabbing the required parameters out of environment variables. Then, modify the call to the constructor of Elasticsearch to take in the AWS4Auth object in the http_auth parameter, following the code snippet that @consulthys linked to.

Let me know if you need any further help working on this.

[svbrunov]

Hello, @llvtt!

I'd like to keep mongo-connector itself as lean as possible and encourage others to write their own DocManagers as plugins, rather than adding the kitchen sink to the project.

This sounds reasonable. Please consider extracting the existing «built-in» document managers from the Mongo Connector as separate plugins. A plugin — an extension of a system, not «a system itself».

Best regards,
Sergey Brunov.

[aherlihy]

I'm closing this for now, but feel free to file a new issue or re-open if you like. We're currently working on pulling out the Elastic doc manager into a separate package (to handle supporting both 1.x and 2.x) so it may be easier to extend the elastic search doc managers in the future. Thank you!