necdf metadata causing high and low values along cost.

[Jeffreydaw]

GIOPS monthly for August 2018

link

GIOPS monthly for May 2018 EDIT: this is a cashed image there are problems this May too, see posts below.

link

[Jeffreydaw]

this seems to be a result of the metadata in the NetCDF files having changed.

August 1st 2018

float votemper(time_counter, depth, y, x) ;
	votemper:units = "Kelvins" ;
	votemper:_FillValue = 1.e+20f ;
	votemper:missing_value = 1.e+30f ;
	votemper:valid_min = 173.f ;
	votemper:valid_max = 373.f ;
	votemper:long_name = "Sea water potential temperature" ;
	votemper:short_name = "votemper" ;
	votemper:standard_name = "sea_water_potential_temperature" ;

August 31st 2018

float votemper(time_counter, depth, y, x) ;
	votemper:units = "Kelvins" ;
	votemper:_FillValue = 1.e+20f ;
	votemper:missing_value = 1.e+30f ;
	votemper:valid_min = 173.f ;
	votemper:valid_max = 373.f ;
	votemper:long_name = "Sea water potential temperature" ;
	votemper:short_name = "votemper" ;
	votemper:standard_name = "sea_water_potential_temperature" ;

July 1st 2018

float votemper(time_counter, deptht, y, x) ;
	votemper:units = "Kelvins" ;
	votemper:missing_value = -99.f ;
	votemper:valid_min = 173.f ;
	votemper:valid_max = 373.f ;
	votemper:long_name = "Water temperature at CMC" ;
	votemper:short_name = "votemper" ;
	votemper:online_operation = "inst(only(x))" ;
	votemper:axis = "TZYX" ;
	votemper:scale_factor = 1.f ;
	votemper:add_offset = 0.f ;
	votemper:savelog10 = 0.f 

[Jeffreydaw]

this may have something to do with the fact that _FillValue and missing_value have different values this is confusing though as they have this issues in GIOPS day too which does not seem to have the problem.

below is an exsert from an email I sent on my findings

I've tried to open the dataset with xarray (a python library) and it tells me that it can't open it because it Discovered conflicting _FillValue and missing_value. as they are different in this file I'm not surprised it is complaining. If you have returned to the EEEC standard whey is there a _FillValue and a missing_value? according to the GIOPS documentation that I can find there should only be one of them, the _fillValue
http://collaboration.cmc.ec.gc.ca/cmc/CMOI/product_guide/docs/lib/product_manual_giops-111_20160208_e.pdf
from what I can tell they are supposed to be treated as the same according to NOAA, (though some groups don't). Also, it looks like missing_value is deprecated. this info comes from https://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2013/msg00291.html
I was able to open the file using ds = xr.open_dataset(f, decode_cf=False) and as far as I can tell the only thing that seems off it the above-mentioned mismatch of"missing_value" and _FillValue

[Jeffreydaw]

the fact that the mont of MAy 2018 looks ok above is because of cashed tiles

blow you can see the tiles near Alaska are bad but others are ok (the ok ones were cashed before the problem started. the values here are blue because they are -99C the missing_value before the change

this may be becayse of how thredds handels files or sonthing in our code but it is till a result of the change in the meta data

[Jeffreydaw]

Here is a sample log message when the page loades GIOPS daily for October 2018.

/opt/tools/miniconda3/lib/python3.6/site-packages/xarray/conventions.py:537: SerializationWarning: variable 'votemper' has multiple fill values {1e+30, 1e+20}, decoding all values to NaN

and here is the message form the log then loading a view from before June 14th. (June 1st 2018)

/opt/tools/miniconda3/lib/python3.6/site-packages/xarray/conventions.py:537: SerializationWarning: variable 'votemper' has multiple fill values {1e+30, 1e+20}, decoding all values to NaN.

They appair to be the same despite the fact that they each file only have 1 missing_value. this is most lily because of the fact that thredds aggregates the files togther and that is causing xarry problems.

so this issue is the same/the-cause-of issue #304

[Jeffreydaw]

proof that the they that THREDDS is talking only the last value for the metadata.
currend sample of missing vale and fill value for 3 different dates

June 1, 2018
votemper:missing_value = -99.f ;

aug 1, 2018:
votemper:_FillValue = 1.e+20f ;
votemper:missing_value = 1.e+30f ;

oct 10, 2018
votemper:_FillValue = 1.e+30f ;
votemper:missing_value = 1.e+30f ;

And below shows the output if the full datasets is read by xarray.

nav_GIOPS_d_fillset_directopen = xr.open_dataset("http://navigator.oceansdata.ca/thredds/dodsC/giops/daily/aggregated.ncml", decode_cf = False)
nav_GIOPS_d_fillset_directopen.variables['votemper']
<xarray.Variable (time_counter: 947, depth: 50, y: 1021, x: 1442)>
[69712552700 values with dtype=float32]
Attributes:
    units:          Kelvins
    _FillValue:     1e+30
    missing_value:  1e+30
    valid_min:      173.0
    valid_max:      373.0
    long_name:      Sea water potential temperature
    short_name:     votemper
    standard_name:  sea_water_potential_temperature
    coordinates:    nav_lon nav_lat time_counter depth  

as can be seen missing_value and _FillValue match the vales for Oct 10, 2018 (the end of the dataset)

[Jeffreydaw]

plan, rewright metea data and values in all old files.

[Jeffreydaw]

updated metea data and fill values, code is in gitlab repo

[NoahGallant-MUN]

Is this fixed?