BiG-CZ: CUAHSI Variable Values using Ulmo and WaterML #2328

rajadain · 2017-10-05T17:03:01Z

Overview

Fetches values from CUAHSI / WDC using Ulmo for each variable at a site. We're currently only showing the latest value in the table, with charts coming in a follow-up PR.

Connects #2243
Connects #2238

Demo

Notes

This PR adds two endpoints: /details and /values. Currently we are only using /values in the UI, but may use /details in the future. There is a dummy placeholder for the charts, but no values.

Known Issues

The popover for the headline for WDC Details View doesn't work.

Testing Instructions

Check out this branch and reprovision app and worker. Run the bundle script.
Go to :8000/?bigcz, pick a shape and search for something
Go to the WDC tab. Click on an item to go to the details view.
Ensure you see a spinner fetching the values.
As values succeed, ensure you see them being filled in the table one by one.
Ensure if fetching succeeds for all values, the "Last Updated" value is updated to be more recent. This happens because the API call used for the initial load has older values than the new ones that come via this second API.
Ensure that if any fetch fails, there is an error icon where the spinner goes.

arottersman

Just took a look through the code — pulling it down now to test. I like the way you've set this up, and thanks for the helpful commit messages!

Comments are either clarification or thoughts on alternative ways to implement some of the pieces, all pretty optional.

arottersman · 2017-10-06T17:11:43Z

src/mmw/apps/bigcz/views.py

+    params = request.query_params
+    catalog = params.get('catalog')
+
+    if not catalog:


I wonder if there should be some sort of request serializer that would handle all the validation here and in here. Validation errors off a serializer would also be a list, instead of the way it's done here, returning only one missing field error at a time.

This isn't that necessary at the moment, but could be something we add when we want to expose the API.

arottersman · 2017-10-06T17:20:50Z

src/mmw/js/src/data_catalog/models.js

+            var values = this.get('values');
+
+            values.reset(response.values);
+            mrv = response.values[response.values.length - 1].value


Is response.values guaranteed to have at least 1 value?

Also, this line is missing a semi-colon. When the build works again, you may see some bad lint output.

arottersman · 2017-10-06T17:20:53Z

src/mmw/js/src/data_catalog/models.js

+            values.reset(response.values);
+            mrv = response.values[response.values.length - 1].value
+
+            delete response.values;


Why do we delete it?

arottersman · 2017-10-06T17:22:19Z

src/mmw/js/src/data_catalog/views.js

+                last_dates = variables.map(function(v) {
+                        var values = v.get('values');
+
+                        return new Date(values.at(values.length - 1).get('datetime'));


Consider using values.last() if possible.

arottersman · 2017-10-06T17:38:28Z

src/mmw/js/src/data_catalog/views.js

+        var mode = this.model.get('mode'),
+            variables = this.model.get('variables'),
+            view = mode === 'table' ?
+                   new CuahsiTableView({ collection: variables }) :


For some of our other toggles, we render all views onShow (or wherever) and then do the switching via display: none; styling on the non-active view (bootstrap might take care of this?).

I find the way you implemented it here a lot easier to follow, but I think the other way reduces the amount of rendering needed.

arottersman · 2017-10-06T17:40:47Z

src/mmw/js/src/data_catalog/views.js

+    },
+
+    fetchComplete: function() {
+        this.model.set({ fetching: false, error: false });


Consider moving these (fetchComplete, fetchError) to the model.

rajadain · 2017-10-06T18:29:55Z

Thanks for taking a look. I'm going to take this feedback into account, and add some more features, and will let you know when this is ready for another look.

Instead of list of concept_keyword strings, we now return a list of objects with keys: code, name, and concept_keyword. This will allow us to continue using concept_keyword rather than the very verbose name as the display field, and adding it under the hood to the code necessary for fetching detail values. We still serve the name, in case the full form is needed. Some sample values: { "code": "NWISUV:63680", "concept_keyword": "Turbidity", "name": "Turbidity, water, unfiltered, monochrome near infra-red LED light, 780-900 nm, detection angle 90 +-2.5 degrees, formazin nephelometric units (FNU)" } { "code": "NWISUV:00301", "concept_keyword": "Oxygen, dissolved percent of saturation", "name": "Dissolved oxygen, water, unfiltered, percent of saturation" } { "code": "NWISUV:99988", "concept_keyword": "Radiation, incoming PAR", "name": "Photosynthetically active radiation (average flux density on a horizontal surface during measurement interval), micromoles of photons per square meter per second" }

Ulmo (https://github.com/ulmo-dev/ulmo/) is a data access library designed to fetch information from various hydrology and climatology sources. It works on the WaterML standard, which is a protocol built on top of SOAP. We will use it to fetch detailed values for sensor data from CUAHSI. Also upgrade pip which speeds up our Python dependency installation step significantly.

This fetches detailed information for any given site. This should be used for the detail page in the app.

This fetches values for any given site and variable. The date range can be optionally specified, by default it is one week up to today. These values can be used for the chart and table in the detail view.

Previously we were using the same Marionette view with differing templates for each catalog. Now that we are trying to add custom events and sub-views, this approach no longer suffices. We make a base view which has open / close functionality built in, and extend it for each catalog, supplying the template in the child view. Instead of a dictionary of templates, CATALOG_RESULT_DETAILS_VIEWS now holds the corresponding view.

These describe the individual variable calls, and the returned values

Different sites and variables have values only for certain ranges. Querying for ranges outside of those lead to errors. Now we first query the /details endpoint to get the units and begin and end date for each variable, then validate the given search dates to be within those (and use sensible defaults of 1 week or entire date range, whichever is shorter, if no from and to dates are specified). Furthermore, we only search each site once. If the values are successfully retrieved, they will be shown immediately the next time the user opens the details page for that site. If the value retrieval errors out for some reason, we retry that every time the details page is opened.

rajadain · 2017-10-09T14:17:10Z

I'm going to split this work into back-end and front-end pieces, and open two separate PRs for each. Closing this in the meantime. Thanks for your feedback!

rajadain · 2017-10-09T18:45:59Z

Superseded by #2353 and #2354.

rajadain added BigCZ in progress labels Oct 5, 2017

arottersman reviewed Oct 6, 2017

View reviewed changes

rajadain added 19 commits October 9, 2017 08:55

Transmit service WSDL url used for timeseries

25e879c

Add details endpoint for CUAHSI

539cbfa

This fetches detailed information for any given site. This should be used for the detail page in the app.

Add values endpoint for CUAHSI

12902a2

This fetches values for any given site and variable. The date range can be optionally specified, by default it is one week up to today. These values can be used for the chart and table in the detail view.

Add Value and Variable models for CUAHSI

1bd458e

These describe the individual variable calls, and the returned values

Fetch values for some CUAHSI values

b0a904a

Add loading spinner for showing state

03a0883

Switch between Chart and Table views

f5d8d79

Show error if any calls fail

75513c1

Rearchitect

70cd0d8

Abstract out DATE_FORMAT for reuse

303b008

Handle edge cases

e2d615c

Polish

9e8b42b

Formatting

2d7e42d

Variable Row Error Messages

cc1b14e

Popover cleanup

137373e

rajadain force-pushed the tt/bigcz-cuahsi-detail-values branch from d225262 to 137373e Compare October 9, 2017 14:15

rajadain closed this Oct 9, 2017

hectcastro removed the in progress label Oct 9, 2017

rajadain deleted the tt/bigcz-cuahsi-detail-values branch October 9, 2017 18:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BiG-CZ: CUAHSI Variable Values using Ulmo and WaterML #2328

BiG-CZ: CUAHSI Variable Values using Ulmo and WaterML #2328

rajadain commented Oct 5, 2017 •

edited

Loading

arottersman left a comment •

edited

Loading

arottersman Oct 6, 2017

arottersman Oct 6, 2017

arottersman Oct 6, 2017

arottersman Oct 6, 2017

arottersman Oct 6, 2017

arottersman Oct 6, 2017

arottersman Oct 6, 2017

rajadain commented Oct 6, 2017

rajadain commented Oct 9, 2017

rajadain commented Oct 9, 2017

BiG-CZ: CUAHSI Variable Values using Ulmo and WaterML #2328

BiG-CZ: CUAHSI Variable Values using Ulmo and WaterML #2328

Conversation

rajadain commented Oct 5, 2017 • edited Loading

Overview

Demo

Notes

Known Issues

Testing Instructions

arottersman left a comment • edited Loading

Choose a reason for hiding this comment

arottersman Oct 6, 2017

Choose a reason for hiding this comment

arottersman Oct 6, 2017

Choose a reason for hiding this comment

arottersman Oct 6, 2017

Choose a reason for hiding this comment

arottersman Oct 6, 2017

Choose a reason for hiding this comment

arottersman Oct 6, 2017

Choose a reason for hiding this comment

arottersman Oct 6, 2017

Choose a reason for hiding this comment

arottersman Oct 6, 2017

Choose a reason for hiding this comment

rajadain commented Oct 6, 2017

rajadain commented Oct 9, 2017

rajadain commented Oct 9, 2017

rajadain commented Oct 5, 2017 •

edited

Loading

arottersman left a comment •

edited

Loading