https://github.com/britairving/UVP_submission_formatting/
This repository provides scripts to reformat UVP data downloaded from Ecotaxa for submission to data archival websites SeaBASS and BCO-DMO.
The Underwater Video Profiler (UVP) is designed for the quantification of particles and of large zooplankton in the water column. The UVP5 software acquires and processes images in real time. During the downcast, The UVP5 captures images of a known volume of water. Particles are detected and analyzed to get size and grey level for each region of interest. Image post processing and metadata creation is accomplished with the Zooprocess software. Images and their associated metadata have been uploaded to the Ecotaxa website (http://ecotaxa.obs-vlfr.fr/) which serves as a tool for taxonomic annotation of zooplankton and classification of non-living particles, using machine learning and human verification, as well as a repository for all globally collected UVP data.
See Setup wiki page for details on how to organize your data and configuration scripts to utilize this toolbox, and how to export data in the appropriate format from Ecotaxa's Image and Particle Modules.
https://github.com/britairving/UVP_submission_formatting/wiki/Setup
- User defines cruiseid, project directory (projectdir), and UVP PAR file (odv_rfile)
- Reads in metadata for the project from fullfile(projectdir, [cruiseid '_UVP_metadata.m'])
- Reads in detailed ODV PAR file exported in ODV format from Ecotaxa's Particle Module
- Format fields for BCO-DMO (since BCO-DMO does not have specific formatting requirements, this just entails renaming variables for clarity)
- Writes parameter descriptions to excel file UVP_PAR_ParameterDescriptions.xlsx
- Format header text (hdr_text) based on project metadata from {cruiseid}_UVP_metadata.m
- Write formatted data with header to a file in projectdir
- User defines cruiseid, project directory (projectdir), and UVP ZOO file (odv_rfile)
- Optionally, the user can define validated_profiles, a text file listing fully validated profiles. If defined, only those profiles will be written to the file.
- Reads in metadata for the project from fullfile(projectdir, [cruiseid '_UVP_metadata.m'])
- Reads in detailed ODV ZOO file exported in ODV format from Ecotaxa's Particle Module
- Format fields for BCO-DMO and remove unnecessary columns
- Query WoRMs to match taxonomic names to scientificNames and scientificNameIDs
- Writes parameter descriptions to excel file UVP_ZOO_ParameterDescriptions.xlsx
- Format header text (hdr_text) based on project metadata from {cruiseid}_UVP_metadata.m
- Write formatted data with header to a file in projectdir
Level 1b definition: Individual level counts with automatic (including interpretation of class scores or probabilities ) and manual classifications, and biovolume and size parameters for each region of interest (ROI). A ROI is defined as a rectangular subset of pixels in a given image. The submission of a Level 1b data table for plankton and other particle observations to SeaBASS must include morphological information for each ROI and must be accompanied by documents that include relevant metadata and processing information.
- Read individual particle and plankton level manual classifications exported from the Ecotaxa Image module in D.O.I. export (tsv) format
- Read in metadata from [cruiseid]_UVP_metadata.m
- Read sb fields from SeaBASS_define_taxonomic_Level1b_fields.m
- Query WoRMS database for taxa scientificNames and scientificNameIDs
- Write YAML-formatted namespace file defining non-conforming ROI's
- Calculate sb fields from definitions
- Build SeaBASS formatted metadata header
- write level1 sb file
- Write Assessed_id_list_[cruiseid].csv file
- User defines cruiseid, project directory (projectdir), and UVP PAR file to read in (odv_rfile), SeaBASS filename to write to (sb_wfile)
- Optionally, the user can define r2r_elog, an excel sheet that lists R2R events with casts
- Reads in metadata for the project from fullfile(projectdir, [cruiseid '_UVP_metadata.m'])
- Reads in detailed ODV PAR file exported in ODV format from Ecotaxa's Particle Module
- Format fields and units for SeaBASS and remove unnecessary columns
- Write formatted SeaBASS header based on project metadata from {cruiseid}_UVP_metadata.m
- Writes a differential file, with fields PSD_DNSD, PSD_DVSD, and a non-differential file with files PSD_NSD and PSD_VSD.
Level 2 definition: abundance and biovolumes
- User defines cruiseid, project directory (projectdir), and UVP ZOO file to read in (odv_rfile), SeaBASS filename to write to (sb_wfile)
- Optionally, the user can define r2r_elog, an excel sheet that lists R2R events with casts
- Optionally, set cfg.limit_to_taxa to a cell array of taxonomic names you're interested in
- Reads in metadata for the project from fullfile(projectdir, [cruiseid '_UVP_metadata.m'])
- Reads in detailed ODV ZOO file exported in ODV format from Ecotaxa's Particle Module
- Format fields and units for SeaBASS and remove unnecessary columns
- Query WoRMs to match taxonomic names to scientificNames and scientificNameIDs
- Write formatted SeaBASS header based on project metadata from {cruiseid}_UVP_metadata.m
- Calls the WoRMS webservice from Matlab to match taxonomic name with WoRMS AphiaID
- Utilizes WoRMS webservice from Matlab
SeaBASS https://seabass.gsfc.nasa.gov/# is the SeaWiFS Bio-optical Archive and Storage System, the publicly shared archive of in situ oceanographic and atmospheric data maintained by the NASA Ocean Biology Processing Group (OBPG).
- Example dataset: 2018 EXPORTS UVP Data https://seabass.gsfc.nasa.gov/archive/UAF/amcdonnell/EXPORTS/EXPORTSNP/archive
BCO-DMO https://www.bco-dmo.org/about-us is the Biological & Chemical Oceanography Data Management Office.
- Example dataset: Taxonomic data from P16N Repeat Hydrography cruise in 2015 https://www.bco-dmo.org/dataset/787966
- Example dataset: Particulate data from P16N Repeat Hydrography cruise in 2015 https://www.bco-dmo.org/dataset/787432
If you use this toolbox, please cite as follows: Brita Irving, (2021), UVP_submission_formatting, GitHub repository, https://github.com/britairving/UVP_submission_formatting/