Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generation and indexing of enriched data to test the reports and other functions #81

Closed
aswanipranjal opened this issue Jul 24, 2018 · 2 comments

Comments

@aswanipranjal
Copy link
Contributor

aswanipranjal commented Jul 24, 2018

The reports generated (and other files/functions) need to be tested using data fetched from elasticsearch.
Right now, the functions in manuscripts2 are tested on git data from grimoirelab-perceval repository which is fetched at the time of testing. This is causing the tests to fail after some time when more commits are added in grimoirelab-perceval. Example

The other approach that is being used is to take the raw data for the data sources --> upload it to ES --> enrich that raw data --> upload that data again to ES.
This approach works but it is slow and using raw data is unnecessary as the enriched data can be directly used.

This issue is to solve the problem regarding the tests. We have to find out a good way to get frozen (which is stored locally) enriched data --> upload it to elasticsearch --> query the data for results.

We are looking for a solution which provides us with the enriched data easily and which is capable of generating that enriched data again if something goes wrong. It should also be able to upload the data into elasticsearch with proper mappings for the data.

@aswanipranjal
Copy link
Contributor Author

check pull #82

@aswanipranjal
Copy link
Contributor Author

This is closed by #84

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant