Quality Control of Temperature and Salinity profiles
Python Makefile
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin Saving pressure. May 19, 2015
cotede
docs Using conda to build documentation (readthedocs). Aug 6, 2016
tests list(dict.keys()) for Python-3 compatibility. May 28, 2017
.gitattributes Setting up LFS: *.gz|nc|cdf Nov 26, 2015
.gitignore Git now ignores .swp (vim swap files) Jul 21, 2015
.travis.yml Travis testing Python-3 May 28, 2017
.zenodo.json Typo. May 29, 2017
AUTHORS.rst Adding authors reference file. Jul 8, 2015
CONTRIBUTING.rst Adding CONTRIBUTING instructions. Jul 6, 2015
HISTORY.rst
LICENSE.rst
MANIFEST.in Moving installing requirements to requirements.txt Mar 30, 2016
Makefile Release new versions to pypi using twine. May 29, 2017
README.rst Zenodo badge pointing to latest DOI. May 29, 2017
readthedocs.yml
requirements.txt Temporarily adding requirement on pyArgo. May 28, 2017
setup.cfg Bump version: 0.19.0 → 0.19.1 May 29, 2017
setup.py Bump version: 0.19.0 → 0.19.1 May 29, 2017
test-requirements.txt
tox.ini Improving TOX setup, including different combinations. May 28, 2017

README.rst

CoTe De l'eau

Documentation Status

`CoTeDe<http://cotede.castelao.net>`_ is an Open Source Python package to quality control (QC) hydrographic data such as temperature and salinity. It was designed to attend individual scientists as well as operational systems with large databases, reading the inputs from different formats and types of sensors, and processing those in parallel for high performance. To achieve that, CoTeDe is highly customizable, allowing the user to compose the desired set of tests, as well as the specific parameters of each test. Otherwise there are preset QC procedures conforming with GTSPP, EuroGOOS and ARGO recommendations. It is also implemented innovating approaches to QC like the Fuzzy Logic (Timms 2011, Morello 2014) and Anomaly Detection (Castelão 2015).

At this point it is operational for profiles (CTD, XBT and Argo) and tracks (TSG). For CTD profiles and TSG time series it uses PySeabird package to interpret directly the SeaBird's .cnv output file, and for argo it uses PyARGO package to interpret the netCDF files.

This is the result from several generations of quality control systems, which started in 2006, when I developed from scratch an automatic quality control system for realtime evaluation of thermosalinographs at AOML-NOAA, USA. Later I was advising the quality control of the brazilian hydrography of PIRATA.

My vision is that we can do better than we do today with more flexible classification techniques, which includes machine learning, to minimize the burden on manual expert QC improving the consistency, performance and reliability of the QC procedure for oceanographic data, especially for realtime operations.

Why use CoTeDe

CoTeDe can apply different quality control procedures:
  • The default GTSPP, EGOOS or Argo procedures;
  • A custom set of tests, including user defined thresholds;
  • A novel approach based on Anomaly Detection, described by Castelao 2015;
  • Two different fuzzy logic approaches: as proposed by Timms 2011 & Morello 2014, and using usual defuzification by the bisector.

Process multiple files in parallel, ideal for large datasets.

Export output, original data plus flags, into netCDF files following OCEANSites data structure.

Quick howto

To evaluate the records of a profile:

import cotede.qc

pqc = cotede.qc.fProfileQC('example.cnv')

To see the temperature records of the primary sensor:

pqc['temperature']

To see the flags of all tests applied on the secondary sensor of salinity:

pqc.flags['salinity2']

To evaluate a full set of profiles at once, like all profiles from a cruise, use the class ProfileQCCollection, like:

dataset = ProfileQCCollection('/path/to/data/', inputpattern=".*.cnv")

dataset.flags['temperature'].keys()

Check the notebooks gallery for more examples and functionalities: http://nbviewer.ipython.org/github/castelao/CoTeDe/tree/master/docs/notebooks/

Documentation

http://cotede.readthedocs.org