Skip to content

Collection of manuals and scripts to assist in automated quality assurance and quality control for oceanographic data collected via fishing gear as a platform.

Notifications You must be signed in to change notification settings

BerringDC/BDC_qc

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

50 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

BDC_QC

Collection of manuals and scripts to assist in automated quality assurance and quality control for oceanographic data collected via fishing gear as a platform.


Manual

Fishing gear as a platform for sensors: Implementation of real-time quality control of in-situ temperature and salinity data collected via fishing gear

Version 1

November 2020

Authors: Berthe Vastenhoud, Carles Castro Muniain & Cooper Van Vranken


Table of contents


Introduction

This document describes the implementation of the automated checks that are performed on CTD (Conductivity, Temperature, Depth) data that are collected via fishing gear as a platform for sensors. Trajectory data describe the positions and time of the fishing vessel.

Figure 1. A fishing vessel as a data collection platform.


Flags

The data collected by fishing vessels of opportunity, i.e. with sensors attached to fishing gear, is aimed to be interoperable and used by different users with different requirements. In order to maximize (re)usability, the data is quality controlled and flagged to characterize data. Flags are always included in the data delivery, to optimize data reliability and consistency.

Quality checks are mainly based on the tests described by IOOS (U.S. Integrated Ocean Observing System, 2020), EuroGOOS (EuroGOOS DATA-/MEQ working group, 2010), the Glider network (U.S. Integrated Ocean Observing System, 2016) and the Argo network (Wong et al., 2020).

The flags used by BDC to indicate QC status are based on existing standards defined by other programs and datasets for oceanographic observations. Flags are indicated in Table 1.

Code Meaning
0/NA No QC was performed
1 Good data
3 Suspect data
4 Bad data
5 Corrected data
9 Missing value

Table 1. Quality flags.

  • Data flagged as (0) are not quality controlled, and therefore recommended not to be used without QC performed by the user.
  • Data flagged as (1) have been quality controlled, and can be used safely.
  • Data flagged as (3) have been quality controlled, and marked as suspect. These data can't be used directly, but have the potential to be corrected in delayed mode.
  • Data flagged as (4) have been quality controlled and should be rejected.
  • Data flagged as (5) have been corrected.
  • Data flagged as (9) are missing.

Real-time Quality control

Figure 2. Schematic of the data flow applied to oceanographic data from fishing vessels.


Fisheries quality control tests

Platform identification (under development)

Check if there is an unknown sensor ID/Vessel ID


Vessel ID control test

Check if the vessel is operating in an expected region, defined by the operator. Action: Values that fail the test should be flagged as suspect data (3).

Region Longitude min (º) Longitude max(º) Latitude min(º) Latitude max(º)
Greenland -60 -15 55 90
North Sea and Baltic -15 30 45 60
Atlantic -75 30 55 90
New Zealand 160 175 -50 -30
Alaska -180 -125 45 90

Table 2. Region location.


Gear type control test

Check if the gear type assigned is correct.

Gear types are either defined as fixed gear (Fixed) or mobile gear (Mobile). A profile measured by a sensor on fixed gear is assigned to one single GPS point. A profile by a sensor on mobile gear follows the trajectory from the fishing vessel.

The distance travelled by the sensor during the profile is calculated between the first and last data locations. Action: Values that fail the test should be flagged as suspect data (3).

Flags Description
Suspect (3) Gear type:

Fixed: distance > 200 meters
Mobile: distance <= 200 meters
Pass (1) Applies for test pass condition.

Table 3. Gear type flags.


Quality control tests CTD

Impossible date test

The date of the profile can be no earlier than 01/01/2010 and no later than current date in UTC. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) Impossible date:

01/01/2010 < Date > UTC
Pass (1) Applies for test pass condition.

Tablel 4. Impossible date flags.


Impossible location test

This test controls whether the geographic location is reasonable, based on the global limits for longitude and latitude. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) Impossible location:

-180º < longitude > 180º
-90º < latitude > 90º
Pass (1) Applies for test pass condition.

Table 5. Impossible location flags.


Position on land test

This test requires that the observation latitude and longitude from a float profile be located in an ocean. Here we use a 5 minutes bathymetry file (ETOPO5/TerrainBase) downloaded from http://www.ngdc.noaa.gov/mgg/global/etopo5.html. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) Measurement is on land.
Pass (1) Applies for test pass condition.

Table 6. Position on land flags.


Impossible speed test

This test is only applied to profiles coming from mobile gear. It controls whether there are no erroneous locations provided. The speed of the vessels are generated given the positions and times of the vessel. Vessel speed is expected not to exceed 3 ms−1. Otherwise, it means either the positions or times are bad data, or a vessel is sailing full speed rather than fishing.

This test is helpful for determining if there is an error in merging the sensor and GPS data, often due to setting a sensor to a time zone other than UTC. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) Speed is too high for mobile gear fishing.

Vessel speed > 4.12 ms−1 (8 knots)
Pass (1) Applies for test pass condition.

Table 7. Impossible speed flags.


Global range test

Gross filter on the observed values of pressure, temperature and salinity based on the sensor ranges (NKE TD, NKE CTD and ZebraTech Moana TD).

This test applies a gross filter on the observed values of pressure, temperature and salinity. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) Measurement outside sensor operating range

NKE sensors
-5 dbar < Pressure
-2 < Temperature > 35 °C
2 < Salinity > 42 PSU

ZebraTech sensors
-5 dbar < Pressure
-2 < Temperature > 35 °C

Lowell sensors
-5 dbar < Pressure
-5 < Temperature > 50 °C

Marport sensors
-5 dbar < Pressure
-5 < Temperature > 25 °C

Hobo/Vemco sensors
-5 dbar < Pressure
-20 < Temperature > 50 °C
Suspect (3) -5 <= Pressure < 0 dbar
Pressure > Max sensor depth + 10% or 50% (ZebraTech)
Pass (1) Applies for test pass condition.

Table 8. Global range flags.


Spike test

The spike tests checks whether there is a significant difference between sequential measurements, by comparing a measurement to its adjacent ones. The test does not consider differences in pressure, and rather assumes measurements that adequately reproduce changes in temperature and salinity with pressure.

Here, V2 is the tested value, and V1 and V3 are the values before and after. Spikes consisting of more than one data point are difficult to capture, but their onset may be flagged by the rate of change test.

Cut-off values are based on (Wong et al., 2020), and V2 will be flagged based on the following values. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) Measurement differs significantly from its neighbours

Pressure < 500 dbar:
Test value T > 6.0 °C
Test value S > 0.9 PSU

Pressure > = 500 dbar:
Test value T > 2.0°C
Test value S > 0.3 PSU
Pass (1) Applies for test pass condition.

Table 9. Spike flags.


Digit rollover test

Here we compare the measurements at the end of the profile to the adjacent measurement. Temperature at the bottom should not differ from the adjacent measurement by more than 1°C. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) Measurement differs significantly from its neighbours

T2 - T1 > 1.0 °C
Pass (1) Applies for test pass condition.

Table 10. Digit rollover flags.


Stuck value/ flat line test

It is possible that, when sensors fail, continuously repeated observations of the same value are produced. In this test, the present observation is compared to several previous observations. The present observation is flagged if the present observation is the same as all previous observations, calculating in a tolerance value. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) The five most recent observations are equal

Tolerance values:
Temperature: 0.05 °C
Salinity: 0.05 PSU
Pressure: 0.5 dbar
Suspect (3) The three most recent observations are equal
Pass (1) Applies for test pass condition.

Table 11. Flat line flags.


Rate of change test

This test is applied per segment (Up-Down-Bottom), and inspects the segments on a rate of change exceeding a threshold defined by the operator. In this case the thresholds are based on the IOOS examples (U.S. Integrated Ocean Observing System, 2020), where the rate of change between measurement Tn-1 and Tn must be less than three standard deviations (3*SD). The SD of the T time series is computed over the full segment.

This test needs to find a balance between setting a threshold too low, triggering too many false alarms, and setting a threshold too high, triggering too little alarms. Action: Values that fail the test should be flagged as suspect data (3).

Flags Description
Suspect (3) The rate of change exceeds the selected threshold.
Pass (1) Applies for test pass condition.

Table 12. Rate of change flags.


Mud test

This test is applied only in the Up segment. Controls whether the temperature sensor has some mud biasing the measurements once hauled. Values are rolled by 10 measurements and observations are flagged if rolled temperature is lower than 0.005ºC. Action: Values that fail the test should be flagged as bad data (4).

Flags Description
Fail (4) Invariant measurement
Rolled temperature < 0.005 °C
Pass (1) Applies for test pass condition.

Table 13. Mud flags.


Timing/ gap test

This test controls whether the most recent measurement has been received within the expected time period. Action: Values that fail the test should be flagged as suspect data (3).

Flags Description
Suspect (3) Check for the arrival of data

Data didn't come in as expected: NOW – TIM_STMP > TIM_INC
Pass (1) Applies for test pass condition.

Table 14. Timing flags.


Climatology test

Test that data point falls within seasonal expectations according to different regions.

This test is a variation on the gross range check, where the thresholds T_Season_MAX and T_Season_MIN are adjusted monthly, seasonally, or at some other operator-selected time period (TIM_TST) in a specific region. Because of the dynamic nature of T and S in some locations, no fail flag is identified for this test and measurements will only be marked as 'suspect' (3).

Regional ranges are defined based on (EuroGOOS DATA-MEQ working group, 2010) and (Wong et al., 2020), see also Figure 3.

clima

Figure 3. Regions used for the climatology test.

Red Sea (from 10 N to 32 N and 30 E to 45 E)

  • Temperature in range 21.7°C to 40.0°C
  • Salinity in range 2.0 to 41.0

Mediterranean Sea (from 30 N to 45 N and 6 W to 30 E)

  • Temperature in range 10.0°C to 40.0°C
  • Salinity in range 2.0 to 40.0

North Western Shelves (from 50 N to 60 N and 20 W to 10 E)

  • Temperature in range –2.0°C to 24.0°C
  • Salinity in range 0.0 to 37.0

South West Shelves (From 25 N to 50 N and 30 W to 0 W)

  • Temperature in range –2.0°C to 30.0°C
  • Salinity in range 0.0 to 38.0

Arctic Sea (above 60N)

  • Temperature in range –1.92°C to 25.0°C
  • Salinity in range 2.0 to 40.0

Seasonal limits per area still have to be defined, in the meantime we take min and max of temp and sal of all measurements from our vessels in DB.

Flags Description
Suspect (3) Measurement outside climatology range

Seas_min_T < Temperature> Seas_max_T
Seas_min_S < Salinity > Seas_max_S
Pass (1) Applies for test pass condition.

Table 15. Climatology flags.


Drift test (under development)

Quality control tests oxygen/ turbidity (under development)

Delayed-mode quality control (under development)

Quality control tests (under development)


References

Annie Wong, Robert Keeley, Thierry Carval and the Argo Data Management Team (2020).

Argo Quality Control Manual for CTD and Trajectory Data. http://dx.doi.org/10.13155/33951

U.S. Integrated Ocean Observing System, 2020. Manual for Real-Time Quality Control of In-situ Temperature and Salinity Data Version 2.1: A Guide to Quality Control and Quality Assurance of In-situ Temperature and Salinity Observations. 50 pp. https://doi.org/10.25923/x02m-m555

U.S. Integrated Ocean Observing System (2016) Manual for Quality Control of Temperature and Salinity Data Observations from Gliders. Version 1.0. Silver Spring, MD, U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Ocean Service, Integrated Ocean Observing System, 23pp. & Appendices. http://hdl.handle.net/11329/289

EuroGOOS DATA-MEQ working group (2010). Recommendations for in-situ data Near Real Time Quality Control. https://doi.org/10.13155/36230

About

Collection of manuals and scripts to assist in automated quality assurance and quality control for oceanographic data collected via fishing gear as a platform.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages