Skip to content

Commit

Permalink
Oxygen calculations are removed from parsers to be performed by a ded…
Browse files Browse the repository at this point in the history 
…icated pre-processing routine using GibbsSeaWater toolbox. This new oxygenPP routine has ben added to the list of default PP routines that needs to be performed in the toolboxProperties.txt config file. A unit test has been added to check oxygen calculations. This unit test is ran before trying to compile a binary.

This change moves all the oxygen calcs from the instrument read routines to a oxygen pre-processor.
Also test for checking the output from the pre-processing against Sea-Bird calculation.
  • Loading branch information
@petejan @ggalibert
petejan authored and ggalibert committed Aug 29, 2017
1 parent 9831396 commit dc1a63b
Show file tree
Hide file tree
Showing 31 changed files with 1,046 additions and 1,256 deletions.
12 changes: 6 additions & 6 deletions AutomaticQC/CTDSurfaceSoakQC.m
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
function [data flags paramsLog] = CTDSurfaceSoakQC( sample_data, data, k, type, auto )
function [data, flags, paramsLog] = CTDSurfaceSoakQC( sample_data, data, k, type, auto )
%IMOSINOUTWATERQC Flags samples which were taken before and after the instrument was placed
% in the water.
%
Expand Down Expand Up @@ -92,7 +92,7 @@

% only concerned here with pumped sensors or variables derived from pumped
% observations
pumpedVar = {'TEMP', 'CNDC', 'DOX1', 'DOX2', 'PSAL', 'DENS'};
pumpedVar = {'TEMP', 'CNDC', 'DOX', 'DOXY', 'DOX1', 'DOX2', 'DOXS', 'PSAL', 'DENS'};
ignoreVar = {'TIME', 'PROFILE', 'DIRECTION', 'LATITUDE', 'LONGITUDE', 'BOT_DEPTH', 'ETIME'};

qcSet = str2double(readProperty('toolbox.qc_set'));
Expand All @@ -107,16 +107,16 @@
flags = ones(lenData, 1, 'int8')*rawFlag;
if any(iPumped)
% initially all data is raw
switch find(iPumped);
case 1 % temperature
switch pumpedVar{iPumped};
case 'TEMP' % temperature
if ~isempty(iTSS)
flags = sample_data.variables{iTSS}.data;
end
case {2, 5, 6} % dependent on conductivity
case {'CNDC', 'PSAL', 'DENS'} % dependent on conductivity
if ~isempty(iCSS)
flags = sample_data.variables{iCSS}.data;
end
case {3, 4} % dependent on oxygen
case {'DOX', 'DOXY', 'DOX1', 'DOX2', 'DOXS'} % dependent on oxygen
if ~isempty(iOSS)
flags = sample_data.variables{iOSS}.data;
end
Expand Down
2 changes: 2 additions & 0 deletions AutomaticQC/imosGlobalRangeQC.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ DEPTH
CPHL
CHLU
CHLF
DOX
DOXY
DOXS
DOX1
DOX2
UCUR
Expand Down
2 changes: 2 additions & 0 deletions AutomaticQC/imosRateOfChangeQC.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@ PSAL = 2*stdDev
PRES = 2*stdDev
PRES_REL = 2*stdDev
DEPTH = 2*stdDev
DOX = 2*stdDev
DOXY = 2*stdDev
DOXS = 2*stdDev
DOX1 = 2*stdDev
DOX2 = 2*stdDev
UCUR = 2*stdDev
Expand Down
2 changes: 2 additions & 0 deletions AutomaticQC/imosVerticalSpikeQC.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ DEPTH = 3
CPHL = PABIM
CHLU = PABIM
CHLF = PABIM
DOX = PABIM
DOXY = PABIM
DOXS = PABIM
DOX1 = PABIM
DOX2 = PABIM
9 changes: 5 additions & 4 deletions IMOS/imosParameters.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,11 @@ DIR_MAG, 0, from_direction,
DIRECTION, 0, direction_of_the_profile, , , , , , , , char
DIRT, 0, to_direction, degree, clockwise, true north, E, 999999.0, 0.0, 360.0, float
DIST_ALONG_BEAMS, 0, distance_from_sensor_along_beams, m, , sensor, Z, 999999.0, -12000.0, 12000.0, float
DOX1, 1, mole_concentration_of_dissolved_molecular_oxygen_in_sea_water, umol l-1, , , O, 999999.0, 0.0, 900000.0, float
DOX2, 1, moles_of_oxygen_per_unit_mass_in_sea_water, umol kg-1, , , O, 999999.0, 0.0, 880000.0, float
DOX, 0, volume_concentration_of_dissolved_molecular_oxygen_in_sea_water, ml l-1, , , O, 999999.0, 0.0, 200.0, float
DOX1, 1, mole_concentration_of_dissolved_molecular_oxygen_in_sea_water, umol l-1, , , O, 999999.0, 0.0, 1000.0, float
DOX2, 1, moles_of_oxygen_per_unit_mass_in_sea_water, umol kg-1, , , O, 999999.0, 0.0, 1000.0, float
DOXS, 1, fractional_saturation_of_oxygen_in_sea_water, percent, , , O, 999999.0, , , float
DOXY, 1, mass_concentration_of_oxygen_in_sea_water, kg m-3, , , O, 999999.0, 0.0, 29.0, float
DOXY, 1, mass_concentration_of_oxygen_in_sea_water, mg l-1, , , O, 999999.0, 0.0, 29.0, float
DOXY_TEMP, 1, temperature_of_sensor_for_oxygen_in_sea_water, degrees_Celsius,, , T, 999999.0, 0.0, 50.0, float
DRYT, 0, dry_bulb_temperature, degrees_Celsius,, , M, 999999.0, , , float
DYNHT, 0, dynamic_height, m, , , E, 999999.0, , , float
Expand Down Expand Up @@ -124,7 +125,7 @@ SSWVT, 0, sea_surface_wave_to_directional_variance_spectral_densit
Sv, 0, mean_volume_backscatter_coefficient, m-1, , , A, 9999.0, 0.0, 1.0, float
SV_mean, 0, mean_volume_backscatter, decibel, , , A, 999999.0, -128.0, 0.0, float
Sv_kurt, 0, kurtosis_volume_backscatter, m-1, , , A, 9999.0, 0.0, 1.0, float
Sv_pcnt_good, 0, percent_Sv_samples_included, percent, , , A, 9999.0, 0.0, 1.0, float
Sv_pcnt_good, 0, percent_Sv_samples_included, percent, , , A, 9999.0, 0.0, 100.0, float
Sv_sd, 0, standard_deviation_volume_backscatter, m-1, , , A, 9999.0, 0.0, 1.0, float

This comment has been minimized.

Sign in to view

Copy link
@petejan

petejan Aug 29, 2017

Author Contributor

Must have slipped through, hardly to do with oxygen pre-processor.
Sv_skew, 0, skewness_volume_backscatter, m-1, , , A, 9999.0, 0.0, 1.0, float
Sv_unfilt, 0, mean_volume_backscatter_including_bad_data, m-1, , , A, 9999.0, 0.0, 1.0, float
Expand Down
137 changes: 4 additions & 133 deletions Parser/YSI6SeriesParse.m
View file
Original file line number Diff line number Diff line change
Expand Up @@ -222,145 +222,16 @@
sample_data.variables{end}.comment = ...
'Dissolved oxygen saturation from ROX optical sensor.';

% mg/l => umol/l
% mg/l
case 'odo2'
sample_data.variables{end}.name = 'DOX1';
sample_data.variables{end}.name = 'DOXY';
sample_data.variables{end}.typeCastFunc = str2func(netcdf3ToMatlabType(imosParameters(sample_data.variables{end}.name, 'type')));
sample_data.variables{end}.data = sample_data.variables{end}.typeCastFunc(field' * 44.660/1.429); % O2 density = 1.429kg/m3
sample_data.variables{end}.data = sample_data.variables{end}.typeCastFunc(field');
sample_data.variables{end}.comment = ...
['Dissolved oxygen from ROX optical sensor originally expressed '...
'in mg/l, O2 density = 1.429kg/m3 and 1ml/l = 44.660umol/l were assumed.'];
'Dissolved oxygen from ROX optical sensor.';
end
sample_data.variables{end}.coordinates = coordinates;
end

% Let's add DOX1/DOX2 if PSAL/CNDC, TEMP and DOXS are present and DOX1 not
% already present
doxs = getVar(sample_data.variables, 'DOXS');
dox1 = getVar(sample_data.variables, 'DOX1');
if doxs ~= 0 && dox1 == 0
doxs = sample_data.variables{doxs};
name = 'DOX1';

% to perform this conversion, we need temperature,
% and salinity/conductivity+pressure data to be present
temp = getVar(sample_data.variables, 'TEMP');
psal = getVar(sample_data.variables, 'PSAL');
cndc = getVar(sample_data.variables, 'CNDC');
pres = getVar(sample_data.variables, 'PRES');

% if any of this data isn't present,
% we can't perform the conversion
if temp ~= 0 && (psal ~= 0 || (cndc ~= 0 && pres ~= 0))
temp = sample_data.variables{temp};
if psal ~= 0
psal = sample_data.variables{psal};
else
cndc = sample_data.variables{cndc};
pres = sample_data.variables{pres};
% conductivity is in S/m and gsw_C3515 in mS/cm
crat = 10*cndc.data ./ gsw_C3515;

% we need to use relative pressure using gsw_P0 = 101325 Pa
psal.data = gsw_SP_from_R(crat, temp.data, pres.data - gsw_P0/10^4);
end

% O2 solubility (Garcia and Gordon, 1992-1993)
%
solubility = O2sol(psal.data, temp.data, 'ml/l');

% O2 saturation to O2 concentration measured
% O2 saturation (per cent) = 100* [O2/O2sol]
%
% that is to say : O2 = O2sol * O2sat / 100
data = solubility .* doxs.data / 100;

% conversion from ml/l to umol/l
data = data * 44.660;
comment = ['Originally expressed in % of saturation, using Garcia '...
'and Gordon equations (1992-1993) and ml/l coefficients, assuming 1ml/l = 44.660umol/l.'];

sample_data.variables{end+1}.dimensions = 1;
sample_data.variables{end}.comment = comment;
sample_data.variables{end}.name = name;
sample_data.variables{end}.typeCastFunc = str2func(netcdf3ToMatlabType(imosParameters(sample_data.variables{end}.name, 'type')));
sample_data.variables{end}.data = sample_data.variables{end}.typeCastFunc(data);
sample_data.variables{end}.coordinates = coordinates;

% Let's add DOX2
name = 'DOX2';

% O2 solubility (Garcia and Gordon, 1992-1993)
%
solubility = O2sol(psal.data, temp.data, 'umol/kg');

% O2 saturation to O2 concentration measured
% O2 saturation (per cent) = 100* [O2/O2sol]
%
% that is to say : O2 = O2sol * O2sat / 100
data = solubility .* doxs.data / 100;
comment = ['Originally expressed in % of saturation, using Garcia '...
'and Gordon equations (1992-1993) and umol/kg coefficients.'];

sample_data.variables{end+1}.dimensions = 1;
sample_data.variables{end}.comment = comment;
sample_data.variables{end}.name = name;
sample_data.variables{end}.typeCastFunc = str2func(netcdf3ToMatlabType(imosParameters(sample_data.variables{end}.name, 'type')));
sample_data.variables{end}.data = sample_data.variables{end}.typeCastFunc(data);
sample_data.variables{end}.coordinates = coordinates;
end
end

% Let's add a new parameter if DOX1, PSAL/CNDC, TEMP and PRES are present
dox1 = getVar(sample_data.variables, 'DOX1');
dox2 = getVar(sample_data.variables, 'DOX2');
if dox1 ~= 0 && dox2 == 0
dox1 = sample_data.variables{dox1};
name = 'DOX2';

% umol/l -> umol/kg
%
% to perform this conversion, we need to calculate the
% density of sea water; for this, we need temperature,
% salinity, and pressure data to be present
temp = getVar(sample_data.variables, 'TEMP');
pres = getVar(sample_data.variables, 'PRES');
psal = getVar(sample_data.variables, 'PSAL');
cndc = getVar(sample_data.variables, 'CNDC');

% if any of this data isn't present,
% we can't perform the conversion to umol/kg
if temp ~= 0 && pres ~= 0 && (psal ~= 0 || cndc ~= 0)
temp = sample_data.variables{temp};
pres = sample_data.variables{pres};
if psal ~= 0
psal = sample_data.variables{psal};
else
cndc = sample_data.variables{cndc};
% conductivity is in S/m and gsw_C3515 in mS/cm
crat = 10*cndc.data ./ gsw_C3515;

% we need to use relative pressure using gsw_P0 = 101325 Pa
psal.data = gsw_SP_from_R(crat, temp.data, pres.data - gsw_P0/10^4);
end

% calculate density from salinity, temperature and pressure
dens = sw_dens(psal.data, temp.data, pres.data - gsw_P0/10^4); % cannot use the GSW SeaWater library TEOS-10 as we don't know yet the position

% umol/l -> umol/kg (dens in kg/m3 and 1 m3 = 1000 l)
data = dox1.data .* 1000.0 ./ dens;
comment = ['Originally expressed in mg/l, assuming O2 density = 1.429kg/m3, 1ml/l = 44.660umol/l '...
'and using density computed from Temperature, Salinity and Pressure '...
'with the CSIRO SeaWater library (EOS-80) v1.1.'];

sample_data.variables{end+1}.dimensions = 1;
sample_data.variables{end}.comment = comment;
sample_data.variables{end}.name = name;
sample_data.variables{end}.typeCastFunc = str2func(netcdf3ToMatlabType(imosParameters(sample_data.variables{end}.name, 'type')));
sample_data.variables{end}.data = sample_data.variables{end}.typeCastFunc(data);
sample_data.variables{end}.coordinates = coordinates;
end
end
end


Expand Down
24 changes: 14 additions & 10 deletions Parser/convertSBEcnvVar.m
View file
Original file line number Diff line number Diff line change
Expand Up @@ -141,27 +141,31 @@
comment = 'Artificial chlorophyll data computed from bio-optical sensor raw counts measurements.';

% oxygen (mg/l)
% mg/l => umol/l
% mg/l
case {'oxsolMg0x2FL', 'oxsatMg0x2FL', 'sbeox0Mg0x2FL'}
name = 'DOX1';
data = data .* 44.660/1.429; % O2 density = 1.429kg/m3
comment = 'Originally expressed in mg/l, O2 density = 1.429kg/m3 and 1ml/l = 44.660umol/l were assumed.';
name = 'DOXY';
comment = '';

% oxygen (ml/l)
% ml/l => umol/l
% ml/l
case 'sbeox0ML0x2FL'
name = 'DOX1';
data = data .* 44.660;
comment = 'Originally expressed in ml/l, 1ml/l = 44.660umol/l was assumed.';
name = 'DOX';
comment = '';

% oxygen (umol/L)
% umol/L
case 'sbeox0Mm0x2FL'
name = 'DOX1';
comment = '';

% oxygen (umol/Kg)
% umol/Kg
case {'oxsolMm0x2FKg', 'oxsatMm0x2FKg', 'sbeox0Mm0x2FKg', 'sbeopoxMm0x2FKg'}
name = 'DOX2';
comment = '';

% Oxygen, SBE 63 [% saturation]
case 'sbeopoxPS'
% Oxygen [% saturation]
case {'sbeopoxPS', 'sbeox0PS'}
name = 'DOXS';
comment = '';

Expand Down
39 changes: 0 additions & 39 deletions Parser/readSBE19cnv.m
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,45 +96,6 @@
comment.(nn) = c;
end

% Let's add a new parameter if DOX1, PSAL/CNDC, TEMP and PRES are present and
% not DOX2
if isfield(data, 'DOX1') && ~isfield(data, 'DOX2')

% umol/l -> umol/kg
%
% to perform this conversion, we need to calculate the
% density of sea water; for this, we need temperature,
% salinity, and pressure data to be present
temp = isfield(data, 'TEMP');
pres = isfield(data, 'PRES_REL');
psal = isfield(data, 'PSAL');
cndc = isfield(data, 'CNDC');

% if any of this data isn't present,
% we can't perform the conversion to umol/kg
if temp && pres && (psal || cndc)
temp = data.TEMP;
pres = data.PRES_REL;
if psal
psal = data.PSAL;
else
cndc = data.CNDC;
% conductivity is in S/m and gsw_C3515 in mS/cm
crat = 10*cndc ./ gsw_C3515;

psal = gsw_SP_from_R(crat, temp, pres);
end

% calculate density from salinity, temperature and pressure
dens = sw_dens(psal, temp, pres); % cannot use the GSW SeaWater library TEOS-10 as we don't know yet the position

% umol/l -> umol/kg (dens in kg/m3 and 1 m3 = 1000 l)
data.DOX2 = data.DOX1 .* 1000.0 ./ dens;
comment.DOX2 = ['Originally expressed in mg/l, assuming O2 density = 1.429kg/m3, 1ml/l = 44.660umol/l '...
'and using density computed from Temperature, Salinity and Pressure '...
'with the CSIRO SeaWater library (EOS-80) v1.1.'];
end
end
end

function [name, data, comment] = convertData(name, data, instHeader, procHeader, mode)
Expand Down
38 changes: 0 additions & 38 deletions Parser/readSBE19hex.m
View file
Original file line number Diff line number Diff line change
Expand Up @@ -192,44 +192,6 @@
end
end

% Let's add a new parameter if DOX1, PSAL/CNDC, TEMP and PRES are present
if isfield(newData, 'DOX1')
% umol/l -> umol/kg
%
% to perform this conversion, we need to calculate the
% density of sea water; for this, we need temperature,
% salinity, and pressure data to be present
temp = isfield(newData, 'TEMP');
pres = isfield(newData, 'PRES');
psal = isfield(newData, 'PSAL');
cndc = isfield(newData, 'CNDC');

% if any of this data isn't present,
% we can't perform the conversion to umol/kg
if temp && pres && (psal || cndc)
temp = newData.TEMP;
pres = newData.PRES;
if psal
psal = newData.PSAL;
else
cndc = newData.CNDC;
% conductivity is in S/m and gsw_C3515 in mS/cm
crat = 10*cndc ./ gsw_C3515;

% we need to use relative pressure using gsw_P0 = 101325 Pa
psal = gsw_SP_from_R(crat, temp, pres - gsw_P0/10^4);
end

% calculate density from salinity, temperature and pressure
dens = sw_dens(psal, temp, pres - gsw_P0/10^4); % cannot use the GSW SeaWater library TEOS-10 as we don't know yet the position

% umol/l -> umol/kg (dens in kg/m3 and 1 m3 = 1000 l)
newData.DOX2 = newData.DOX1 .* 1000.0 ./ dens;
comment.DOX2 = ['Originally expressed in umol/l, assuming 1l = 0.001m3 '...
'and using density computed from Temperature, Salinity and Pressure '...
'with the CSIRO SeaWater library (EOS-80) v1.1.'];
end
end
end

function check = checkField(struc, fieldName, fieldValue)
Expand Down
Loading

0 comments on commit dc1a63b

Please sign in to comment.