Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customizable Elasticsearch configuration #532

Closed
afaulconbridge opened this issue Apr 1, 2019 · 2 comments

Comments

Projects
None yet
2 participants
@afaulconbridge
Copy link

commented Apr 1, 2019

Currently there is a bunch of configuration for Elasticsearch that is baked into the pipeline.

This should be externalized into a separate file, similar to the ops and data configuration.

This is related to #367 (Document elasticsearch index configuration required)

@afaulconbridge afaulconbridge self-assigned this Apr 1, 2019

@afaulconbridge afaulconbridge changed the title Configurable Elasticsearch configuration Customizable Elasticsearch configuration Apr 1, 2019

@Docproc

This comment has been minimized.

Copy link
Collaborator

commented Apr 1, 2019

I had a look at AWS' Elasticsearch Service (where they manage it for you). It seemed pretty promising, however I ran in to an error in the HPA step of the pipeline:

elasticsearch.exceptions.AuthenticationException: TransportError(401, u'{"Message":"Your request: \'/_cluster/settings\' payload is not allowed."}')

This was triggered by the following line in cluster.py/get_settings

/return/self.transport.perform_request('GET', '/_cluster/settings',params=/params/)

after some digging it turns out that "cluster/settings" not supported: https://forums.aws.amazon.com/message.jspa?messageID=683415

This may not be the only issue with this AWS service, but I stopped at this point. Running a DIY Elasticsearch instance on AWS (tuned with the script I got from Miguel) seems to work just fine.

@afaulconbridge

This comment has been minimized.

Copy link
Author

commented May 9, 2019

This is done with PR opentargets/data_pipeline#486

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.