Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BiG-CZ: Fetch CUAHSI Values using WaterML and Ulmo #2353

Merged
merged 3 commits into from
Oct 10, 2017
Merged

BiG-CZ: Fetch CUAHSI Values using WaterML and Ulmo #2353

merged 3 commits into from
Oct 10, 2017

Commits on Oct 9, 2017

  1. Upgrade pip and add Ulmo 

    Ulmo (https://github.com/ulmo-dev/ulmo/) is a data access
library designed to fetch information from various hydrology
and climatology sources. It works on the WaterML standard,
which is a protocol built on top of SOAP. We will use it
to fetch detailed values for sensor data from CUAHSI.

Also upgrade pip which speeds up our Python dependency
installation step significantly.
    @rajadain
    rajadain committed Oct 9, 2017
    Configuration menu
    Copy the full SHA
    b9055f4 View commit details
    Browse the repository at this point in the history

  2. Return more details for Variables 

    Previously, a CUAHSI search would have a `concept_keywords` array
of strings, corresponding to each variable. Now that we want to
fetch more detailed values, we switch to having an array of
`variables`, each of which in addition to `id`, `name` (that
tends to be longer, more like a description than name), and
`concept_keyword`, has a `site` and `wsdl` which will be used
to fetch values for that variable in a given timespan.

This array is sorted by `concept_keyword`, which will also be
used as a display label in the UI.
    @rajadain
    rajadain committed Oct 9, 2017
    Configuration menu
    Copy the full SHA
    b3cb9fb View commit details
    Browse the repository at this point in the history

  3. Add details and values endpoints 

    For fetching CUAHSI variable values, we add two endpoints:

`details` expects a `wsdl` and `site` parameter, and uses it to
fetch the WaterML siteinfo object for the given site from the
given wsdl url. This siteinfo object contains details for the
variables in the site, including units, and the time ranges for
which each variable has values. This information is important,
because if we query for `values` with a time outside this range,
the endpoint crashes.

`values` expects a `wsdl`, `site`, `variable`, `from_date` and
`to_date`, and uses it to fetch the values of that variable at
that site in the given time range, and returns the WaterML value
object.

The expected use is one call to `details` for a site, to fetch
units and time ranges for each variable, and then a call to
`values` for each variable in the site, fetching its values for
the given time range.

---

Notes:

1. The WaterML objects themselves are complex and elaborate, with
   many nested keys and arrays. Fortunately, we don't need to
   write our own serializers for them, because that is all taken
   care of within Ulmo. Furthermore, because of WaterML being an
   underlying standard, we can reasonably expect the data to
   follow a certain shape, and exercise this assumption all the
   way on the client side, without having to verify it on the
   server.

2. Ulmo uses suds under the hood to make SOAP requests, and by
   default uses the suds cache, which is distinct from the redis
   cache used elsewhere in the app (for Geoprocessing, etc). I
   have not looked into changing this default behavior, leaving it
   to a future investigation should the need arise.
    @rajadain
    rajadain committed Oct 9, 2017
    Configuration menu
    Copy the full SHA
    2c2d41f View commit details
    Browse the repository at this point in the history