-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Index data to AWS elastic search from AWS cloud search #9
Index data to AWS elastic search from AWS cloud search #9
Conversation
Fix tests to run with elastic search Refactor tests to remove redundant tests
Add tests for the health facilities scraper
59c8cf4
to
993c7c4
Compare
993c7c4
to
4e703be
Compare
Hey @Gathondu. Thanks for the PR. Could you also add the documentation to the README? |
@@ -17,8 +17,13 @@ | |||
# Clinical document endpoint | |||
"cloudsearch_cos_endpoint": "http://doc-cfa-healthtools-ke-cos-nhxtw3w5goufkzram4er7sciz4.eu-west-1.cloudsearch.amazonaws.com/", | |||
# Health facilities endpoint | |||
"cloudsearch_health_faciities_endpoint":"https://doc-health-facilities-ke-65ftd7ksxazyatw5fiv5uyaiqi.eu-west-1.cloudsearch.amazonaws.com", | |||
"cloudsearch_health_faciities_endpoint": "https://doc-health-facilities-ke-65ftd7ksxazyatw5fiv5uyaiqi.eu-west-1.cloudsearch.amazonaws.com", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's remove all the cloudsearch endpoints.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@DavidLemayian this too is handled in the other branch
# Bucket="cfa-healthtools-ke", | ||
# Key=self.health_facilities_scraper.s3_key | ||
# )['Body'].read() | ||
# self.assertEqual(uploaded_data, data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why commented?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried writing tests for health facilities and test that they are uploading but it was failing because of the key. Funny enough I removed it too in the other branch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@DavidLemayian funny. I did this on the other branch..I merged to develop on my local repo and continued with the slack integration. This was handled
What does this PR do?
Shifts the indexing of the scraped data from AWS cloud search to elastic search.
Fixes the health-facilities scrapper
Adds tests for health facilities
Refactors tests that were already there
Add continuous integration with CircleCi
Description of Task to be completed?
Replace CloudSearch with ElasticSearch
How should this be manually tested?
Run nostests --nocapture in a terminal pointing to the projects directory and virtual environment