Refactors templates and adds QC variable attributes #129

benjwadams · 2017-08-15T18:30:29Z

Refactors template by moving template boilerplate into a loop. Also
adds default attributes for a number of QC variables. Requires updated
logic from glider-dac repo.

Refactors template by moving template boilerplate into a loop. Also adds default attributes for a number of QC variables. Requires updated logic from glider-dac repo.

benjwadams · 2017-08-15T18:30:44Z

Required by ioos/glider-dac#105

benjwadams · 2017-08-15T18:31:39Z

erddap/templates/private/dataset.deployment.xml

-    {% for var in qartod_variables %}
+    {# if there are QARTOD variables defined, populate them #}
+    {% if qc_var_types['qartod'] | length > 0 %}
+    {% for qartod_var in (qc_var_types['qartod']) %}


Dunno why I put parentheses here.

benjwadams · 2017-08-15T18:34:47Z

erddap/templates/private/dataset.deployment.xml

-        <sourceName>time_qc</sourceName>
-        <destinationName>precise_time_qc</destinationName>
+        <sourceName>{{req_var}}</sourceName>
+        <destinationName>{{dest_var_remaps.get(req_var, req_var)}}</destinationName>


Perhaps I should have added a comment here. This tries to get any name remappings and defaults to the original variable name if not found. defaultdict didn't seem to cover this use case elegantly, but maybe I was missing something.

Bobfrat · 2017-08-15T18:35:15Z

erddap/templates/private/dataset.deployment.xml

-            <att name="long_name">v Variable Quality Flag</att>
+            <att name="long_name">{{dest_var_remaps.get(req_var, req_var)}} Variable Quality Flag</att>
+            {% if '_FillValue' not in qc_var_types['gen_qc'][req_var] %}
+            <att name='_FillValue'>-127</att>


Do you need to set a type here?

<att name='_FillValue' type='int'>-127</att>

https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#addAttributes

Bobfrat · 2017-08-15T18:39:09Z

erddap/templates/private/dataset.deployment.xml

-        <addAttributes>
-            <att name="ioos_category">Quality</att>
-            <att name="long_name">v Variable Quality Flag</att>
+            <att name="long_name">{{dest_var_remaps.get(req_var, req_var)}} Variable Quality Flag</att>


Does the output from {{dest_var_remaps.get(req_var, req_var)}} result in the variable name + '_qc'? If so I think we need to drop the _qc in the long name.

E.G.

<destinationName>u_qc</destinationName> <att name="long_name">u Variable Quality Flag</att>

benjwadams · 2017-08-15T20:42:26Z

erddap/templates/private/dataset.deployment.xml

-        <addAttributes>
-            <att name="ioos_category">Quality</att>
-            <att name="long_name">v Variable Quality Flag</att>
+            <att name="long_name">{{dest_var_remaps.get(req_var, req_var).replace('_qc', '')}} Variable Quality Flag</att>


Ew. May refactor later; too much logic in the templates.

kwilcox · 2017-08-16T15:02:27Z

I would recommend adding both _FillValue and missing_value attributes to maintain backwards and forward compatibility.

I'm also not a fan of assigning the QARTOD flag of 2 for the missing_value. Accessing the data with (most) client libraries is going to mask the data and it will be an extra step before analysis to retrieve the actual flag value of 2.. @rsignell-usgs @ocefpaf @shane-axiom any thoughts?

Bobfrat · 2017-08-17T12:20:21Z

erddap/templates/private/dataset.deployment.xml

+               flag_values, and flag_meanings defined #}
+           <att name="flag_values">1B, 2B, 3B, 4B, 9B</att>
+           <att name="flag_meanings">GOOD NOT_EVALUATED SUSPECT BAD MISSING</att>
+           <att name="_FillValue" type='byte'>2</att>


@kerfoot what's your opinion on setting the _FillValue to a 2 (NOT_EVALUATED) flag?

kerfoot · 2017-08-17T12:55:07Z

I agree with @kwilcox. The _FillValue should be a value not listed in variable:flag_values

Adds flag_values, flag_meanings, _FillValue attributes to *_qc variables if they are not present in the original dataset.

kwilcox · 2017-08-17T22:13:25Z

erddap/templates/private/dataset.deployment.xml

-            {% if '_FillValue' not in qc_var_types['gen_qc'][req_var] %}
-            <att name='_FillValue' type='byte'>-127</att>
+            {% if req_var not in qc_var_types['gen_qc'] %}
+            <att name="flag_values">0,1,2,3,4,5,6,7,8,9</att>


Why do the gen_qc variables have a different syntax for flag_values when compared to the qartod variables?

<att name="flag_values">0,1,2,3,4,5,6,7,8,9</att>
vs
<att name="flag_values">1B, 2B, 3B, 4B, 9B</att>

Fixed in latest to comply with CF.

Uses same types in flag_attributes values, per CF standard section 3.5: "The flag_masks and flag_values attributes are both the same type as the variable to which they are attached, and contains a list of values matching unique bit fields.""

kwilcox · 2017-08-18T17:48:39Z

erddap/templates/private/dataset.deployment.xml

@@ -310,7 +310,7 @@
            <att name="ioos_category">Quality</att>
            <att name="long_name">{{dest_var_remaps.get(req_var, req_var).replace('_qc', '')}} Variable Quality Flag</att>
            {% if req_var not in qc_var_types['gen_qc'] %}
-            <att name="flag_values">0,1,2,3,4,5,6,7,8,9</att>
+            <att name="flag_values">0b, 1b, 2b, 3b, 4b, 5b, 6b, 7b, 8b, 9b</att>


I'm mobile and can't test this in ERDDAP or the NCML syntax at the moment... but I believe this should be:

<att name="flag_values" type="byte">0 1 2 3 4 5 6 7 8 9</att>

I don't know where the b is coming from and spaces are used as the separator in NcML/NetCDF array attributes.

Can you add any tests for these templates? Maybe loading one into NetCDF Java with some fill data and testing the attributes?

I agree, but I would think it needs to be a byteList, right?

<att name="flag_values" type="byteList">0 1 2 3 4 5 6 7 8 9</att>

It should be byte. byteList isn't a thing in the NcML Schema. See: http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/ncml/AnnotatedSchema4.html, the "attribute Element" section and the "DataType Type" attribute.

Oh I thought this was an ERDDAP datasets.xml template so I was looking at the following:
https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#addAttributes

My mistake, you are right, I thought this was an NcML file! byteList is correct.

kwilcox · 2017-08-18T17:48:53Z

erddap/templates/private/dataset.deployment.xml

@@ -331,7 +331,7 @@
            <att name="ioos_category">Quality</att>
            {# all qartod variables should have _FillValue, missing_value,
               flag_values, and flag_meanings defined #}
-           <att name="flag_values">1B, 2B, 3B, 4B, 9B</att>
+           <att name="flag_values">1b, 2b, 3b, 4b, 9b</att>


Same as other comment

Refactors templates and adds QC variable attributes

ef7c6a7

Refactors template by moving template boilerplate into a loop. Also adds default attributes for a number of QC variables. Requires updated logic from glider-dac repo.

benjwadams commented Aug 15, 2017

View reviewed changes

Bobfrat reviewed Aug 15, 2017

View reviewed changes

benjwadams added 2 commits August 15, 2017 16:39

Add type info to _FillValue attr

39c6fcc

Remove '_qc' from long_name attribute

8b800eb

benjwadams commented Aug 15, 2017

View reviewed changes

Bobfrat reviewed Aug 17, 2017

View reviewed changes

benjwadams added 3 commits August 17, 2017 16:51

Add flag_values, flag_meanings, _FillValue to *_qc

c3e25b0

Adds flag_values, flag_meanings, _FillValue attributes to *_qc variables if they are not present in the original dataset.

Adds _FillValue to QARTOD vars if not present

8712e94

Adds valid_min and valid_max attrs to QC and QARTOD vars

7964f7b

kwilcox reviewed Aug 17, 2017

View reviewed changes

Fix flag_values attribute to use same types

4e0749b

Uses same types in flag_attributes values, per CF standard section 3.5: "The flag_masks and flag_values attributes are both the same type as the variable to which they are attached, and contains a list of values matching unique bit fields.""

kwilcox reviewed Aug 18, 2017

View reviewed changes

benjwadams mentioned this pull request Aug 18, 2017

DAC ERDDAP and QC outstanding issue #127

Closed

Set flag_meanings to byteList type

0e0a062

Bobfrat approved these changes Aug 18, 2017

View reviewed changes

Bobfrat merged commit c23e83e into ioos:master Aug 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactors templates and adds QC variable attributes #129

Refactors templates and adds QC variable attributes #129

benjwadams commented Aug 15, 2017

benjwadams commented Aug 15, 2017

benjwadams Aug 15, 2017

benjwadams Aug 15, 2017

Bobfrat Aug 15, 2017 •

edited

Loading

Bobfrat Aug 15, 2017

benjwadams Aug 15, 2017

kwilcox commented Aug 16, 2017 •

edited

Loading

Bobfrat Aug 17, 2017

kerfoot commented Aug 17, 2017

kwilcox Aug 17, 2017

benjwadams Aug 18, 2017

kwilcox Aug 18, 2017

Bobfrat Aug 18, 2017

kwilcox Aug 18, 2017

Bobfrat Aug 18, 2017

kwilcox Aug 18, 2017

kwilcox Aug 18, 2017

Refactors templates and adds QC variable attributes #129

Refactors templates and adds QC variable attributes #129

Conversation

benjwadams commented Aug 15, 2017

benjwadams commented Aug 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bobfrat Aug 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kwilcox commented Aug 16, 2017 • edited Loading

Choose a reason for hiding this comment

kerfoot commented Aug 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bobfrat Aug 15, 2017 •

edited

Loading

kwilcox commented Aug 16, 2017 •

edited

Loading