Weather data - use 3rd party API to populate DB table for every point

[tnigon]

"point": the location of a response variable we are trying to predict via supervised regression

It looks like Open Weather Map is a clear leader in accessing hyperlocal weather data. Who knows what accuracy they have (or what accuracy anyone has for that matter), but I think it's worth saving in our tables to be used as a potential feature for prediction. They have estimated soil temp/moisture, an option for getting weather for a polygon, and have scalable options for historical and forecasted weather.

It sounds like Descartes has their own weather data API (through GSF Weather?), so maybe we table this until we get a chance to see what kind of weather info Descartes has. Also worth considering is the cost of retrieving weather vs the cost of a platform like Descartes - if we're going to have to spend that money anyway, it may be worth it to keep it all on one account (but have to consider timeline as well). My thoughts are that it should be relatively easy to port from one weather provider to another (at least early on), and we should get the data populated so we can do some R&D to see which weather variables are most important.

Alternative providers

• OpenWeatherMap
• Visual Crossing
• aWhere
• SolCast

To Do

1. #XX - Decide if we want to go forward with Open Weather Map for accessing weather data (1000 free calls per day without a paid account, but no historical weather)
2. Weather DB Schema Plans (long term storage v. API call) #24 - Develop a plan for what data we want to retrieve and store ourselves vs what we can simply call when needed, and develop the DB schema for doing table joins, etc. for customers. Initial thoughts - EPIC weather requirements, including: T_min (ºC), T_max (ºC), relative humidity (%), precipitation (mm), wind speed (m/s @ 2m above ground level), and solar radiation (MJ/m^2).
3. #XX - Develop our weather application to retrieve json weather for our locations and save to the DB.
4. #XX - A bit higher level, but as it pertains to weather, decide what data Insight should "own"/maintain vs what the customer should "own" (whose DB should hold the data)? Initial thoughts are whoever bought or created the data (e.g., using their own weather station).
5. #XX - [Long term] - Develop a redundancy plan around API integrations for multiple weather providers. Given the volatility in the weather API industry, we don't want to depend on a single provider unless there is a clear benefit in accuracy from one provider over another. This could also include direct integration from public open-source weather forecast models such as GFS rather than via commercial provider.
6. #XX - [Medium term] - (Phase II objective?) Have the ability to "override" modeled weather from an API with an actual weather station nearby our point of interest. This is something to work on with Troy Schmidtke from EarthScout.
7. #XX - Assess forecasted prediction accuracy (if/when forecast weather data are used). Decide if weather forecast data will be saved in our DB, and if so, decide how it will be saved. I think we will always want to save and predictions (real-time or forecasted), but it is another story to store the forecast features (at least as long as they will never be used in a training dataset). With weather forecasts, we would just use "observed" weather or "modeled" weather in our training because we know it will be more accurate. If we want to save forecast weather, explicitly determine the value as it relates to our business.