Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customizable Elasticsearch configuration #532

afaulconbridge opened this issue Apr 1, 2019 · 2 comments


None yet
2 participants
Copy link

commented Apr 1, 2019

Currently there is a bunch of configuration for Elasticsearch that is baked into the pipeline.

This should be externalized into a separate file, similar to the ops and data configuration.

This is related to #367 (Document elasticsearch index configuration required)

@afaulconbridge afaulconbridge self-assigned this Apr 1, 2019

@afaulconbridge afaulconbridge changed the title Configurable Elasticsearch configuration Customizable Elasticsearch configuration Apr 1, 2019


This comment has been minimized.

Copy link

commented Apr 1, 2019

I had a look at AWS' Elasticsearch Service (where they manage it for you). It seemed pretty promising, however I ran in to an error in the HPA step of the pipeline:

elasticsearch.exceptions.AuthenticationException: TransportError(401, u'{"Message":"Your request: \'/_cluster/settings\' payload is not allowed."}')

This was triggered by the following line in

/return/self.transport.perform_request('GET', '/_cluster/settings',params=/params/)

after some digging it turns out that "cluster/settings" not supported:

This may not be the only issue with this AWS service, but I stopped at this point. Running a DIY Elasticsearch instance on AWS (tuned with the script I got from Miguel) seems to work just fine.


This comment has been minimized.

Copy link

commented May 9, 2019

This is done with PR opentargets/data_pipeline#486

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.