Integration of sensor data from Texas Mesonet soil moisture network

[fearghalodonncha]

Texas mesonet network provides soil moisture (and other pertinent variables) at 95 stations across Texas.

Integration of these data can help develop machine learning models since they provide long-term measurements of soil moisture.

Details on the Mesonet API provided here. The basic structure is something along the lines:

• request metadata on all stations (site id, station name, latitude, longitude) from the GET stations Call: https://www.texmesonet.org/api/Stations
• For a specific site id, request past data for significant time period (e.g. 5 yrs == 6,570,000 minutes). This probably has to be batched for functionality. Example to download for site_id 4 for past 200 minutes: https://www.texmesonet.org/api/AllChartingFieldsById/4/200

These will serve as the basis for time series forecasting models

[Nachiket18]

https://www.texmesonet.org/DataProducts/CustomDownloads - To keep download times under 2 minutes, there are established limits on date range and data type combinations. I think this is the first roadblock in getting data for 5 years via API. We area able to download the data for 15 minutes interval for 15 days . There are other options like downloading the data using the custom download section (https://www.texmesonet.org/DataProducts/CustomDownloads) which would give us 1 year data. There's a way that I know using Mechanical Soup to do that programmatically. @fearghalodonncha

[charitarthchugh]

I think we would have to script it with multiple requests, and doing that asynchronously will help.

[Nachiket18]

The problem is API is designed in such a way that it's not retrieving the data for more than 15 days. No matter how many asynchronous call you do it won't go beyond that. Hence the alternate route is using their Data Products and they way we can programmatically do it is using Mechanical Soup.

[charitarthchugh]

As a sidenote, I found this: https://www.skymetweather.com/corporate/skymet-APIs.php as another potential data source (Geared towards India).

[Nachiket18]

I ,@kartikibhargava and @abeetath were able to develop a code for extracting the data from Texas Mesonet and it runs in 35 minutes. @fearghalodonncha @Gaurav-Ramakrishna. The code is placed on AI club fork of the LiquidPrep repository

[fearghalodonncha]

This is fantastic @Nachiket18 and team! Great progress. Can you push the code to the LiquidPrep-ML repo. Thanks!

[charitarthchugh]

The code is currently pushed to UConnAI/LiquidPrep-ML. Should we open a pull request?

[fearghalodonncha]

Cool. No, this is fine for time being. Thanks all