USGS STN Flood Event Data

[fernando-aristizabal]

@cheginit

The USGS maintains a short-term network (STN) flood event service to store and serve data from flood events including rapid deployment sensors and high water marks (HWMs). Their website is here which includes background, viewers, data portals, RESTFUL documentation, and data dictionaries.

While their RESTFUL service is well documented and easy to use, it does require in-depth knowledge of how to use REST in general and specifically this service as well. There are some processing considerations to parse, organize, and geo-reference the data where applicable to conform with Python standards.

Currently, fernando-aristizabal:stn_flood_event_viewer contains a proposed Python client for the this RESTFUL service. This work is WIP as it still requires testing, documentation, review, improved geo-referencing, and parameter filtering.

These changes found in pygeohydro/stnfloodevents.py contain several advantages to the RESTFUL service:

• The user does not need to know the details of RESTFUL in general and of this API specifically.
• Parses the data and returns Python objects (e.g., pandas.DataFrame, geopandas.GeoDataFrame) instead of JSON.
• Convenience functions are offered for data dictionaries.
• Geo-references the data where applicable.

We believe pygeohydro was the best location for this but we are open for feedback and willing to change things around if necessary. Again the work is WIP so more to follow to ensure testing, documentation, and standards are met.

Edit: I realized pygeoogc has the RESTfulURLs class which might be a good home for some of the proposed URLs used.

Best,
Fernando Aristizabal

[cheginit]

Thanks for working on this, sounds like a very nice addition! I think, adding it to PyGeoHydro makes sense.

I understand that it's a work in progress. I have few suggestions:

1. For making web calls, HyRiver offers two options: AsyncRetriever and RetrySession from PyGeoOGC. I suggest to use AsyncRetriever. It's efficient and takes care of many technical details for sending requests and receiving responses. So instead of using requests, requests.get, requests.text, requests.json, and _check_response in your code, you just simply use ar.retrieve_text or ar.retrieve_json. You can check it out here and let me know if you have any questions.
2. Regarding the return_type idea in your code, HyRiver's philosophy is to make sensible default choices, so I recommend to return the data as a dataframe by default and provide a flag-type argument called, for example, as_dict so when it's set to true, the function returns it as a dict.
3. About filter_param type arguments which are strings, it's much better if you have only one dict-type argument called, for example, query_params and ask user to pass filters as key-values. Before making requests, you can check the keys of this dict to ensure they match the service's keys.
4. For converting a pandas dataframe to geopandas you can use points_from_xy so you don't have to worry about the efficiency of your code for large datasets.

[fernando-aristizabal]

@cheginit I'm continuing this discussion in an open, draft PR over in pygeohydro. Thanks for the feedback.