GP2-1398_comtrade_data_copy #838
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ctory-api into GP2-1398_comtrade_data_copy
Codecov Report
@@ Coverage Diff @@
## develop #838 +/- ##
===========================================
+ Coverage 97.35% 97.43% +0.08%
===========================================
Files 112 113 +1
Lines 3818 3942 +124
===========================================
+ Hits 3717 3841 +124
Misses 101 101
Continue to review full report at Codecov.
|
depsiatwal
reviewed
Feb 23, 2021
CHANGELOG.md
Outdated
| @@ -22,7 +23,7 @@ | |||
| - GP2-1382 - getting paid structure | |||
| - GP2-1180 - travel bus BE | |||
| - NOTICKET - Exportplan make dict default for JSON fields. | |||
| - GP2-1181 - business risk | |||
| - GP2-1181 - 1398business risk | |||
There was a problem hiding this comment.
Thanks for the spot.
depsiatwal
approved these changes
Feb 23, 2021
webbyfox
reviewed
Feb 24, 2021
dataservices/tests/conftest.py
Outdated
| trade_value=91.97, | ||
| ) | ||
| yield | ||
| models.InternetUsage.objects.all().delete() |
There was a problem hiding this comment.
I guess you need to delete all records from ComtradeReport here
There was a problem hiding this comment.
Hehe - well spotted - thanks
webbyfox
reviewed
Feb 24, 2021
dataservices/tests/conftest.py
Outdated
| year_2019=13, | ||
| ) | ||
| yield | ||
| models.InternetUsage.objects.all().delete() |
There was a problem hiding this comment.
similarly here EaseOfDoingBusiness
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Moves a copy of filtered comtrade data into the directory-api database.
The new request to get comtrade data supports multiple countries, single commodity code and does no processing, so all the data are returned for the FE to compute changes and trends.
The country list is always a list of iso2 codes.
There is also a new request for country data that allows multiple country's data to be requested in one go, again using iso3. The model names required can be passed in so this can become a generic request to get any data for a country.
A compacted comtrade data extract is in S3. run
make manage import_comtrade_datato load it.This process can take an hour or more I'm afraid.
The extract is generated as follows:
Comtrade data are downloaded as vast csv files (about 6GB), one for each year. There is a management command
make manage import_comtrade_data -- --raw <filename>That reads through such files and loads your db with only the data that are needed.
Then, a sql dump of the table to generate the single data file that gets uploaded to S3.