Data collection

[sayantikabanik]

Sources

• IDP portal

[nitinjethwani7]

References:
Report:1
Crop_Price_Report1.pdf

Report: 2

Machine_Learning_for_Price_Prediction_for_Agricult.pdf

[nitinjethwani7]

Raw Data:
Rawdata.xls

Sample Dataset Features:

Sample_DF.xlsx

[nitinjethwani7]

Hackathon Gdrive Link:Farm yield forecasting

https://drive.google.com/drive/folders/17kI6TMFvd4ehfXCPjBNo6-zjC5BoB01J?usp=sharing

[sayantikabanik]

@sahithi02 references are here
@sayantikabanik go through the data

[sayantikabanik]

The excel version of sample data looks broad and a story behind can be proposed
while the hackathon one has masked location data for farms (which would be harder to present in evaluation)

[sayantikabanik]

https://agmarknet.gov.in/

https://fcainfoweb.nic.in/Reports/Report_Menu_Web.aspx
(select the dates and a report is generated)
Daily Retail_Wholesale Report.pdf
@nitinjethwani7 @sahithi02 @anuraagbhavaraju take a look into this website and the data

[sayantikabanik]

http://www.nhb.gov.in/OnlineClient/MonthlyPriceAndArrivalReport.aspx

we have to parse the data or pick the json which APi is throwing
(we can pick a few products and set data collection frequency)

[anuraagbhavaraju]

The excel version of sample data looks broad and a story behind can be proposed while the hackathon one has masked location data for farms (which would be harder to present in evaluation)

@sayantikabanik by hackathon data do you mean Rawdata.xlsx?

[anuraagbhavaraju]

http://www.nhb.gov.in/OnlineClient/MonthlyPriceAndArrivalReport.aspx

we have to parse the data or pick the json which APi is throwing (we can pick a few products and set data collection frequency)

@sayantikabanik this has an export to excel option also if that helps. But I am not sure why rates from this sources are weird. I checked a couple of vegetables, all the rates are in thousands

[sayantikabanik]

will be working with sample df from data.gov.in and another model based on individual commodity
@nitinjethwani7 will be sharing details on data collection allocation

[sahithi02]

Adding data source for the rainfall column

http://www.indiaenvironmentportal.org.in/media/iep/infographics/Rainfall%20in%20India/112%20years%20of%20rainfall.html

[nitinjethwani7]

Sample DF: V1 (2 YEARS COMPLETE DATA)
Sample_DF.xlsx

@anuraagbhavaraju
@sahithi02
@kullayappagithub
@krishnasastry37
Please help me extract data in the above format for any 1 location each . For example ,you can download regional data for hyderabad/Gujarat(Region wise) and prepare the format accordingly as given in the sample DF file.

[sayantikabanik]

Data collection discussion

• summary dropping the idea for location-based data for now
• vlookup or any other process from raw data to get the required data

2014-2016 @sahithi02
2017-2019 @anuraagbhavaraju
2021 + 2013 @nitinjethwani7

[sayantikabanik]

Raw data - #6 (comment)

[sayantikabanik]

https://github.com/sayantikabanik/FP2/tree/main/forecasting_framework/data
(add all the datasets here, I will be transferring it to cloud soon )

Please keep in mind that we will be using API based approach in building out data pipelines so coding in python including data cleaning etc is helpful in long run

[anuraagbhavaraju]

@nitinjethwani7 @sayantikabanik @sahithi02 I uploaded a first version of Rawdata.xls converted into the format that we discussed this morning in the below link with name 'FPData v1.0.xlsx'. I created a short python script to do this job

https://github.com/sayantikabanik/FP2/tree/main/forecasting_framework/data

@nitinjethwani7 I did it for the entire file (all time periods), please have a look and let me know if it needs any modifications

[nitinjethwani7]

@anuraagbhavaraju -Please help me integrate rainfall data in the same file which you added.Here is the rainfall data till 2015
Rainfall data till 2015.xls
!

[anuraagbhavaraju]

@anuraagbhavaraju -Please help me integrate rainfall data in the same file which you added.Here is the rainfall data till 2015 Rainfall data till 2015.xls !

Hi Nitin, this is done. I uploaded FPData v1.1.xlsx

[sayantikabanik]

#Need the logic you have used for the extraction from Raw format
Also please add one block of comment with this #info:

• Final raw sources used
• Logic used
• Any potential changes