Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
World Ozone and Ultraviolet Radiation Data Centre (WOUDC)
Quality Assessment Guide
Table of Contents
- 6.1 Ozonesonde * 6.1.1 Pump flow rate * 6.1.2 Response time * 6.1.3 Pump temperature * 6.1.4 Background current * 6.1.5 Pump motor current (mA) * 6.1.6 Correction factor * 6.1.7 Integrated ozone * 6.1.8 Sonde total ozone * 6.1.9 Ozone partial pressure profile
- 6.2 TotalOzone * 6.2.1 Total column ozone
- 6.3 Spectral * 6.3.1 Spectral Irradiance Native Flag
Quality assessment (Qa) of ozone and ultra violet radiation observations is a crucial component of data processing and management in WOUDC. It benefits both the data provider and user:
- Data providers are informed when their observing instruments and/or methods are functioning at or outside expected/normal tolerance.
- Data users who would like to be aware of the quality of the data before consuming it for their research and product generation needs.
The scope of this document is to provide an inventory of quality assessment rules applied to data processed by WOUDC and serve as a guide to the overall quality assessment done by WOUDC. In here, one will find:
- Qa flagging scheme
- Inventory of Qa tests per dataset, element, etc.
- For each test:
- Qa parameters/thresholds
This document will be reflect all data quality related activities conducted by WOUDC.
Acronyms used in this document:
Table 1: Acronyms found in this documents.
3 Qualty Assessment Test Categories
The following categories of test are or will be covered by WOUDC quality assessment. Note, not all assessment will involve all categories mentioned below. Minimally, the first three categories: presence, integrity and range will be covered during most assessments.
Table 2: Test categories
|Presence (P)||Checks to see if there's a value to apply tests for.|
|Integrity (I)||Tests for the data type or format to determine whether it complies with the expected data (i.e. integer, real number, etc.) or within the specified set of allowable values.|
|Range (r)||Tests if value is within a specified range (e.g. defined by instrumentation limits, historical climate extremes, statistically, etc.)|
|Inter-variable (IV)||Tests consistency between the values of elements with closely connected parameters within the same observation.|
|Tempoal (T)||Tests for anomalies within a time series: step test, persistence, unrealistic spikes & dips.|
|Spatial (S)||Tests for spatial consistency by comparison to neighboring observations.|
4 Quality Assessment Test Organization
The following depiction illustrates the components of a Qa test and how it relates to the element under assessment.
Figure 1: Qa test organization
A Qa test category can contain any number of tests. Each test can contain any number pre-conditions that must be met prior to executing this Qa test. A test is defined by a function depicting an inequality with parameters. For example, a range test can be of the form: a <= x <= b, where x is the element value under assessment and a, b are parameters/thresholds. Each element can have any number of tests. Each test has a single result. Various roll-ups/summaries of the results are also attainable from the detailed result set.
5 Flagging Scheme
The following flagging scheme is applied to the Qa results. This is a subset of WMO’s data quality flag.
Table 3: Data quality flags
|Flag Value||Flag Name||Definition|
|-10||Suppressed||The data provider has indicated that the value is not to be used or published due to instrumentation or environmental issues which have compromised the data.|
|-1||Missing||There is no value available.|
|0||Error||The value is physically impossible, beyond the capability of the sensor to detect, or identified as incorrect.|
|10||Doubtful||The value is physically possible but is statistically improbable: at or exceeding expected extreme values.|
|15||Suspect / Warning||The value is physically possible but is identified as uncertain.|
|20||Inconsistent||The value is inconsistent when compared to an independently measured predictor variable. It is not possible to determine if either the value or the predictor variable is suspect.|
|100||Accepted / Passed||The value passed all applicable quality assessment test(s) or has been verified as acceptable.|
To each Qa test executed on an element, a flag denoting the result will be attached to it.
6 Quality Assessment Test
This section will contain the inventory of Qa tests performed on elements per dataset
Ozonesonde reports vertical profile of ozone as integrated ozone, sonde total ozone and partial pressure of ozone vs. height along with some auxiliary data. Qa checks will be applied to both sonde vertical profile data and auxiliary data. Auxiliary data checks are only conducted on Ozonesonde reported using ECC (electrochemical cell) or Brewer Mast instruments.
6.1.1 Pump flow rate
If pump flow rate is reported, perform range test otherwise assign “
M” missing flag.
0.25 - 0.35 (s/ml)
6.1.2 Response time
If response time is reported, perform range test otherwise assign “
M” missing flag.
10 - 40 (1/e)(s)
6.1.3 Pump temperature
If pump temperature is reported, perform range test otherwise assign “
M” missing flag.
273 - 329 (K)
6.1.4 Background current
(#PREFLIGHT_SUMMARY,Ib0,Ib1,Ib2) There are three background currents (1B1, 1B0, 1B2) recommended in the specifications. Check each one separately for presence and value.
If background current is reported, perform range test otherwise assign “
M” missing flag.
0 - 0.17 uA
6.1.5 Pump motor current (mA)
If pump motor current is reported, perform range test otherwise assign “
M” missing flag.
x < 100 mA
6.1.6 Correction factor
If correction factor is present, and is from ECC or Brewer Mas, move to range test, otherwise do not perform the test, assign “
0.6 - 1.8 for ECC 0.4 - 2.0 for Brewer Mast
6.1.7 Integrated ozone
If integrated ozone is reported, perform range test otherwise assign “
M” missing flag.
0 - 600 DU
6.1.8 Sonde total ozone
Sonde total ozone is the sum of integrated ozone and residual ozone. There is large variation in sonde total ozone. The amount varies on earth with respect to the position of the sun (latitude). Therefore, three latitude ranges are used for the assessment of sonde total ozone. The check applies to sonde total ozone from all ozonesondes.
If sonde total ozone is reported, perform range test otherwise assign “M” missing flag.
Accepted values of Sonde total ozone for each latitude range are given in the table below.
|60 °N - 90 ° N||200 - 600 DU|
|60 ° N - 60 ° S||200 - 500 DU|
|60 ° S - 90 ° S||80 - 600 DU|
6.1.9 Ozone partial pressure profile
During ozonesonde ascent, ozone partial pressure changes continuously with ascend. Variation in partial pressure can pinpoint some of the instruments failure errors. The two test categories are sharp change in ozone partial pressure and pressure within layers. The check applies to sonde total ozone from all ozonesondes.
18.104.22.168 Sharp change in ozone partial pressure
- If ΔPPO3 ≥ 0.5 mPa within 10 meters above 20 hPa, then flag
- During 10 meters of ascent for atmosphere pressure greater than 20hPa, if the change in partial pressure is ≥ 0.5 mPa, then flag the data as an error (otherwise accept it).
- If ΔPPO3 = 0 within 1 observation, then flag (likely instrument failure)
- If during ascent partial pressure of ozone changes to zero for the observations afterwards, then flag the value as an error and all the values observed after that point (most likely caused by an instrument failure).
22.214.171.124 Ozone partial pressure limits within layers
- Ozonesonde profile report atmospheric pressure and ozone partial pressure. During ascent from ground (atmospheric pressure) to 250 hPa, if the ozone partial pressure is 0 ≤ PPO3 < 20 mPa, then accept the data (otherwise flag as error).
- For atmospheric pressure 20 hPa to 250 hPa, if the ozone partial pressure is 0 ≤ PPO3 < 35 mPa, then accept the data (otherwise flag as error).
- For atmospheric pressure 20 hPA and higher, if the volume mixing ratio is 3 ≤ VMRO3 ≤ 9, then accept the data (otherwise flag as error).
Note: The check applies to ozone data from all ozonesonde types. Volume mixing ratio is not reported in the ozone profile but calculated as shown below:
Volume mixing ratio of ozone = (Partial pressure of ozone * 10)/atmospheric pressure (hPa)
126.96.36.199 Pump Failure
If the last ozone partial pressure of the sonde data is below 20 hPa, atmospheric pressure is very low (0), and stays the same, then flag as an error. This is most likely caused by battery failure. Check applies to ozone data from all ozonesonde types.
6.2.1 Total column ozone
100≤OT≥700 DU In exceptional circumstances, OT may show readings below 100 DU over the Antarctic stations in the southern hemisphere spring. These cases may be indicative of conditions that correspond to severely depleted total ozone, and needs close scrutiny. Where permissible, these readings should be verified against ozonesondes.
6.3.1 Spectral Irradiance Native Flag
Range check: 0 <= GLOBAL_SUMMARY.Flag <= 100 (Acceptable)
Apply a higher level summary to the incoming/native quality assessment flag found in each GLOBAL_SUMMARY table. This check will immediately identify if the Spectral file is valid or not, as well as identify the offending GLOBAL_SUMMARY tables in the file.