Skip to content
This repository has been archived by the owner on Dec 22, 2022. It is now read-only.

Commit

Permalink
Data collection - Peru automated #52 #71
Browse files Browse the repository at this point in the history
  • Loading branch information
@pablodz
pablodz committed May 17, 2020
1 parent 40521c7 commit 3261812
Show file tree
Hide file tree
Showing 86 changed files with 2,328 additions and 60 deletions.
58 changes: 29 additions & 29 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,35 +96,35 @@ WARNING: Some countries are not reporting their death and recovery figures to th

#### Tabla 1. Data resources for each country

| # | Country | User | Data Sources |
| --- | ------------------------ | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| 1 | ARGENTINA | [martingra](https://bit.ly/2UA5JjF) [pablorea](https://bit.ly/3dHNlwG) | [D1](https://bit.ly/3aabv0y) [D2](https://bit.ly/394NsPy) |
| 2 | BOLIVIA | [mamanipatricia](https://bit.ly/2UzdbLU) | [D1](https://bit.ly/3bh1qz6) |
| 3 | BRAZIL | [Automated - dfuribez](https://bit.ly/3aN3xLc) | [S1](https://bit.ly/2xMr5kR) [D1](https://bit.ly/2WuChNd) |
| 4 | CHILE | [ivanMSC](https://bit.ly/2UZBUb6) | [D1](https://bit.ly/2xWXhlH) [D2](https://bit.ly/02Jg6JDf) |
| 5 | COLOMBIA | [Automated - dfuribez](https://bit.ly/3aN3xLc) | [S1](https://bit.ly/39LPi8n) [D1](https://bit.ly/2xkYD9k) [D2](https://bit.ly/2UsSu2U) |
| 6 | COSTA RICA | [Automated - dfuribez](https://bit.ly/3aN3xLc) | [API](https://bit.ly/2V34zfz) |
| 7 | CUBA | [yudivian](https://bit.ly/2wVSYqL) | [API](https://bit.ly/2JGSA1Z) |
| 8 | DOMINICAN REPUBLIC | [ZurMaD](https://bit.ly/2wZwntr) | [D1](https://bit.ly/2UKoUY9) [D2](https://bit.ly/2J2aBHM) |
| 9 | ECUADOR | [pablora19](https://bit.ly/2UB6rgl) | [D1](https://bit.ly/2J3ompB) [D2](https://bit.ly/2UsK2R7) |
| 10 | EL SALVADOR | [Automated - ZurMaD](https://bit.ly/2wZwntr) | [D1](https://bit.ly/2U7N7Hm) [D2](https://bit.ly/39JhgBn) |
| 11 | FRENCH GUIANA | WITHOUT MAINTENANCE | [D1](https://bit.ly/2UUZsxU) |
| 12 | GUADELOUPE | WITHOUT MAINTENANCE | [D1](https://bit.ly/2V0aQsj) |
| 13 | GUATEMALA | [ncovgt2020](https://bit.ly/3aHpECQ) | |
| 14 | HAITI | WITHOUT MAINTENANCE | |
| 15 | HONDURAS | [Automated - ivanMSC](https://bit.ly/2UZBUb6) | [D1](https://bit.ly/2UQBDs5) |
| 16 | MARTINIQUE | WITHOUT MAINTENANCE | [D1](https://bit.ly/34fbNRW) |
| 17 | MEXICO | [carranco-sga](https://bit.ly/2UAAdSw) | [D1](https://bit.ly/3brQ7nY) |
| 18 | NICARAGUA | WITHOUT MAINTENANCE | [D1](https://bit.ly/2QQNfJB) |
| 19 | PANAMA | [josetup123](https://github.com/josetup123) | [D1](https://bit.ly/2UpH8he) |
| 20 | PARAGUAY | WITHOUT MAINTENANCE | |
| 21 | PERU | [DSRP](https://bit.ly/2UzdciW) [ZurMaD](https://github.com/zurmad) [diegocl02](https://bit.ly/2wNIlGt) | [D1](https://bit.ly/2J5Wnpj) [D2](https://bit.ly/3dSKwZO) [D3](https://bit.ly/2StGIoL) |
| 22 | PUERTO RICO | WITHOUT MAINTENANCE | |
| 23 | SAIN PIERRE AND MIQUELON | WITHOUT MAINTENANCE | |
| 24 | SAINT BARTHELEMY | WITHOUT MAINTENANCE | |
| 25 | SAINT MARTIN | WITHOUT MAINTENANCE | |
| 26 | URUGUAY | WITHOUT MAINTENANCE | |
| 27 | VENEZUELA | [rendergraf](https://bit.ly/345Z2Jg) | [D1](https://bit.ly/2J3E0Br) [D2](https://bit.ly/3acdykY) |
| # | Country | User | Data Sources |
| --- | ------------------------ | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| 1 | ARGENTINA | [martingra](https://bit.ly/2UA5JjF) [pablorea](https://bit.ly/3dHNlwG) | [D1](https://bit.ly/3aabv0y) [D2](https://bit.ly/394NsPy) |
| 2 | BOLIVIA | [mamanipatricia](https://bit.ly/2UzdbLU) | [D1](https://bit.ly/3bh1qz6) |
| 3 | BRAZIL | [Automated - dfuribez](https://bit.ly/3aN3xLc) | [S1](https://bit.ly/2xMr5kR) [D1](https://bit.ly/2WuChNd) |
| 4 | CHILE | [ivanMSC](https://bit.ly/2UZBUb6) | [D1](https://bit.ly/2xWXhlH) [D2](https://bit.ly/02Jg6JDf) |
| 5 | COLOMBIA | [Automated - dfuribez](https://bit.ly/3aN3xLc) | [S1](https://bit.ly/39LPi8n) [D1](https://bit.ly/2xkYD9k) [D2](https://bit.ly/2UsSu2U) |
| 6 | COSTA RICA | [Automated - dfuribez](https://bit.ly/3aN3xLc) | [API](https://bit.ly/2V34zfz) |
| 7 | CUBA | [yudivian](https://bit.ly/2wVSYqL) | [API](https://bit.ly/2JGSA1Z) |
| 8 | DOMINICAN REPUBLIC | [ZurMaD](https://bit.ly/2wZwntr) | [D1](https://bit.ly/2UKoUY9) [D2](https://bit.ly/2J2aBHM) |
| 9 | ECUADOR | [pablora19](https://bit.ly/2UB6rgl) | [D1](https://bit.ly/2J3ompB) [D2](https://bit.ly/2UsK2R7) |
| 10 | EL SALVADOR | [Automated - ZurMaD](https://bit.ly/2wZwntr) | [D1](https://bit.ly/2U7N7Hm) [D2](https://bit.ly/39JhgBn) |
| 11 | FRENCH GUIANA | WITHOUT MAINTENANCE | [D1](https://bit.ly/2UUZsxU) |
| 12 | GUADELOUPE | WITHOUT MAINTENANCE | [D1](https://bit.ly/2V0aQsj) |
| 13 | GUATEMALA | [ncovgt2020](https://bit.ly/3aHpECQ) | |
| 14 | HAITI | WITHOUT MAINTENANCE | |
| 15 | HONDURAS | [Automated - ivanMSC](https://bit.ly/2UZBUb6) | [D1](https://bit.ly/2UQBDs5) |
| 16 | MARTINIQUE | WITHOUT MAINTENANCE | [D1](https://bit.ly/34fbNRW) |
| 17 | MEXICO | [carranco-sga](https://bit.ly/2UAAdSw) | [D1](https://bit.ly/3brQ7nY) |
| 18 | NICARAGUA | WITHOUT MAINTENANCE | [D1](https://bit.ly/2QQNfJB) |
| 19 | PANAMA | [josetup123](https://github.com/josetup123) | [D1](https://bit.ly/2UpH8he) |
| 20 | PARAGUAY | WITHOUT MAINTENANCE | |
| 21 | PERU | [DSRP](https://bit.ly/2UzdciW) [ZurMaD](https://github.com/zurmad) [diegocl02](https://bit.ly/2wNIlGt) | [D1](https://bit.ly/2J5Wnpj) [D2](https://bit.ly/3dSKwZO) [D3](https://bit.ly/2StGIoL) [D4](https://bit.ly/3cCg5Gc) |
| 22 | PUERTO RICO | WITHOUT MAINTENANCE | |
| 23 | SAIN PIERRE AND MIQUELON | WITHOUT MAINTENANCE | |
| 24 | SAINT BARTHELEMY | WITHOUT MAINTENANCE | |
| 25 | SAINT MARTIN | WITHOUT MAINTENANCE | |
| 26 | URUGUAY | WITHOUT MAINTENANCE | |
| 27 | VENEZUELA | [rendergraf](https://bit.ly/345Z2Jg) | [D1](https://bit.ly/2J3E0Br) [D2](https://bit.ly/3acdykY) |

\*Automated by scripts in [this folder](https://github.com/DataScienceResearchPeru/covid-19_latinoamerica/tree/master/utils/scripts)

Expand Down
38 changes: 7 additions & 31 deletions utils/scripts/data_collection/data/brazil_data.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,18 @@
import sys
import os

if 1 < len(sys.argv) < 3:
init_date = sys.argv[1]
try:
init_date = pd.to_datetime(init_date, format="%Y/%m/%d")
except ValueError:
print("incorrect date format")
sys.exit(0)
else:
print("Usage brazil_data.py <initial_date>\n date format: %Y/%m/%d example: 2020/03/25")
sys.exit(0)

init_date = pd.to_datetime("2020/03/25", format="%Y/%m/%d")

confirmed_url = "https://raw.githubusercontent.com/elhenrico/covid19-Brazil-timeseries/master/confirmed-cases.csv"
deaths_url = "https://raw.githubusercontent.com/elhenrico/covid19-Brazil-timeseries/master/deaths.csv"
dsrp_github="https://raw.githubusercontent.com/DataScienceResearchPeru/covid-19_latinoamerica/master/latam_covid_19_data/daily_reports/2020-03-08.csv"


confirmed = pd.read_csv(confirmed_url)
deaths = pd.read_csv(deaths_url)
compare = pd.read_csv(dsrp_github)

###
compare = pd.read_csv("https://raw.githubusercontent.com/DataScienceResearchPeru/covid-19_latinoamerica/master/latam_covid_19_data/daily_reports/2020-03-08.csv")
brazil_compare = compare[compare.Country=="Brazil"]
###

brazil_compare = compare[compare['Country']=="Brazil"]

string_a = "áéíóúäëïöüâêîôûã" # character to be replaced
string_b = "aeiouaeiouaeioua" # character to replace with
Expand All @@ -40,18 +29,11 @@ def remove_tildes(string):

confirmed = confirmed.rename(columns={"Unnamed: 0": "Subdivision", "Unnamed: 1": "Code"})
deaths = deaths.rename(columns={"Unnamed: 0": "Subdivision", "Unnamed: 1": "Code"})
#print(confirmed)

sub_brazil = sorted(brazil_compare.Subdivision.unique())
sub_repo = sorted(confirmed.iloc[:,0].unique())

#print(sub_brazil)

#print("Subdivision brazil:", len(sub_brazil))
#print("Subdivisions:", len(sub_repo))

other_subdivisions = list(set(sub_repo) - (set(sub_brazil))) # Subdivisiones other than those listed in the main repo
#print(other_subdivisions)

confirmed = confirmed[~confirmed.Subdivision.isin(other_subdivisions)].sort_values("Subdivision")
deaths = deaths[~deaths.Subdivision.isin(other_subdivisions)].sort_values("Subdivision")
Expand All @@ -62,9 +44,6 @@ def remove_tildes(string):
confirmed_columns = confirmed.columns[1:]
deaths_columns = deaths.columns[1:]

#print(" ".join(confirmed_columns))
#print(" ".join(deaths_columns))

columns_order = ["ISO 3166-2 Code", "Country", "Subdivision", "Last Update", "Confirmed", "Deaths", "Recovered"]

for column in confirmed_columns:
Expand Down Expand Up @@ -106,9 +85,6 @@ def remove_tildes(string):
daily_report.Confirmed = daily_report.Confirmed.astype("int64")
daily_report.Recovered = daily_report.Recovered.astype("int64")


path_data=f"utils/scripts/data_collection/data/brazil_temporal/{date}.csv"
print(daily_report[daily_report.Country=="Brazil"])
# os.system("git pull")
daily_report.to_csv(f"utils/scripts/data_collection/data/brazil_temporal/{date}.csv", index=False)
# os.system(f"git add {daily_report_path}")
# os.system("git commit -m 'Update Brazil'")
daily_report.to_csv(path_data, index=False)
Loading

0 comments on commit 3261812

Please sign in to comment.