Some issues of the daily datasets

[halccw]

Record some issues to be solved.

1. Missing cities
   From the current 7 daily datasets (Rice, Wheat, Onion...), There are1308 cities(or towns or markets) in daily datasets that are not covered by regions.csv. I have not found an efficient way to solve it. The whole list please check: https://github.com/fabbrix/humanitas/blob/master/data/india/csv_daily/agmarknet.nic.in/missing_cities_daily.csv
2. Duplicate dates and abnormal spikes problem

Rice

[mstefanro]

Duplicate dates exist for daily data as well?
On 04/28/2014 07:05 PM, chingchia wrote:

Record some issues to be solved.

Missing cities
From the 7 current daily datasets (Rice, Wheat, Onion...), There
are1308 cities in daily datasets that are not covered by
regions.csv. I have not found an efficient way to solve it. The
whole list please check:
https://github.com/fabbrix/humanitas/blob/master/data/india/csv_daily/agmarknet.nic.in/missing_cities_daily.csv

Duplicate dates and abnormal spikes problem

Rice

figure_1
https://cloud.githubusercontent.com/assets/4166714/2820142/af3b8d78-cef6-11e3-8ff8-c916a7bd3eb1.png

figure_2
https://cloud.githubusercontent.com/assets/4166714/2820141/af3b28c4-cef6-11e3-9e7c-fbfb7ee159aa.png

—
Reply to this email directly or view it on GitHub
#20.

[halccw]

Yes, similar to the weekly ones.

The following is from daily Rice:

https://github.com/fabbrix/humanitas/blob/master/analysis/preproc/dup_daily.txt

[tonyo]

Duplicates arise from weird tabular data.
See, for example, http://agmarknet.nic.in/cmm2_home.asp?comm=Rice&dt=28/01/2010, for Gajapathinagaram. There are two rows with empty subproducts, but the data is duplicated from previous rows.
I made some additional checks, and it looks like all rows with missing subproduct are redundant.
@ChingChia Could you please try to ignore products with empty subproduct field and see what happens?

[halccw]

This is the duplication-record of excluding empty subproduct: (daily Rice)

https://github.com/fabbrix/humanitas/blob/master/analysis/preproc/dup_daily_rice_exclude_empty_subproduct.txt

The result seems nice, 2 identical dates to 1 identical price. I can eliminate them by taking of one of the duplicated 2-dates. (taking the non-zero tonnes one)