[DataChallenge] Check and Improve the Data availability. #7
Description
This is part of a DatChallenge
Description
While uploading HOBO data into the database, you will run into error messages like this:
Uploading Metdata
Some entries are missing metadata:
----------------------------------
name hobo_id data_available region latitude longitude exposition altitude influence description
27 S*******, Leon 10350064 no Freiburg NaN NaN NaN NaN NaN NaN
Done.
Downloading data repository. This may take a minute.
Done.
Start uploading. You can grab a coffee...
File ./data-master/hobo/2021/raw/10350071.csv references HOBO ID=10350071, which is not found.
Parsing file './data-master/hobo/2021/hourly/10801132_Th.csv' was not successfull.
Do not edit the files by hand!
Parsing file './data-master/hobo/2021/hourly/10347394_Th.csv' was not successfull.
Do not edit the files by hand!
Parsing file './data-master/hobo/2021/hourly/10350068_Th.csv' was not successfull.
Do not edit the files by hand!
Done.
100% (36 of 36) |#############################################| Elapsed Time: 0:00:34 ETA: 00:00:00
There are several things that can go wrong, when you load data from a remote location into a database with a script. Luckily, the hydenv CLI catched most of the errors and gave you (hopefully) expressive error messages. For this data challenge you need to run the cli over all HOBO data and identify all catched (and maybe uncatched) error. Open issues for each kind of error to describe what is going wrong and discuss possible solutions.
Assignment
Before you take action, invite @mmaelicke and @modche into the discussion. The data challenge is solved, when all problems in the HOBO data files from all the past years are discussed and solved wherever possible. That means, all issues opened in the context of this data challenge need to be verified by and closed by an instructor.