Setup
Information on how to upload data to Ecotaxa or access their FTP site can be found on the PIQv website.
- Go to Ecotaxa's Particle Module website: https://ecotaxa.obs-vlfr.fr/part/ Note: you may need to login if the project is not public!
- Enter the "Ecotaxa Project" name and click "Apply Filters"
- Click "Export selection"
- Enable "Detailed Data" radio button with dropdown File Format ODV
- You can either save to the Ecotaxa FTP by checking that box, or download directly from the website by leaving it unchecked.
- Go to Ecotaxa's Image Module website: https://ecotaxa.obs-vlfr.fr/
- Login
- Click "Contribute to a project"
- Select the project you want to export
- In the top left there is a "Project" dropdown, select "Export" from that dropdown
- Enable "D.O.I. export" radio button
- Select "All images of each object" in the Export image files dropdown
- You can either save to the Ecotaxa FTP by checking that box, or try downloading directly from the website by leaving it unchecked.
You'll need to define at least two variables at the beginning of each script.
cruiseid = cruiseID, can be user defined or vessel number format. (e.g. "SR1812" is the 2018 EXPORTS North Pacific crusie on the R/V Sally Ride)
projectdir = Path to project folder where {_cruiseid_}_UVP_metadata.m script is and where data files are written
SeaBASS has very specific formatting requirements, with many header and data field requirements. Before using any of these scripts, you must compile project metadata into a script that can be called via the MATLAB eval function.
For example, UVP data has been archived for the 2018 EXPORTS North Pacific cruise on the R/V Sally Ride, SR1812. The following metadata file was used for Level2 taxonomic and particulate files, as well as the Level 1b taxonomic file.
function hdr = SR1812_UVP_metadata
%% function SR1812_UVP_METADATA
%
% Description:
% Define metadata used in file header
% Information for EXPORTSNP 2018 Sally Ride (Survey cruise)
%
% Output:
% hdr | structure containing requred metadata information
%
% Notes:
% SeaBASS formatting requirements
% No spaces are allowed in metadata headers
% COMMENTS must be preceeded by "!" and not "#"
%
% BCO-DMO formatting requirements
% Much less stringent, but follow basic formatting of SeaBASS for
% simplicity
%
% Author:
% Brita Irving <bkirving@alaska.edu>
%% Define necessary fields for processing
hdr.ecotaxaf = 'ecotaxa_export_1591_20210512_1846'; % folder & name of tsv file exported from Ecotaxa (image module > export > DOI option)
hdr.raw_wfile = 'EXPORTS-EXPORTSNP_UVP5-TaxonomicLevel1b_survey_20180814-20180909_R0.sb'; % name of the seabass file to write to
hdr.namespace = 'namespace_EXPORTSNP_UVP'; % name of the yml formatted namespace file
hdr.bad_object_id = {'ctdnear' 'ctdNaN'}; % removes these
hdr.dat_info_file = '/Users/bkirving/Documents/MATLAB/UVP_project_data/SR1812/uvp5_sn207_2018_exports_np_sr1812_DAT.csv'; % from UVP_DAT_file_info_merge.m
%hdr.limit_to_taxa = {'Salpida' 't008'}; % Cell array with limited taxa names
%% Define metadata that will be written to file header
hdr.investigators = 'Andrew_M_P_McDonnell';
hdr.affiliations = 'University_of_Alaska_Fairbanks';
hdr.contact = 'amcdonnell@alaska.edu';
hdr.experiment = 'EXPORTS';
hdr.cruise = 'EXPORTSNP';
hdr.station = 'NA';
hdr.calfiles = 'uvp_calibration_report_of_uvp5sn207a_from_uvp5sn002_20180129.pdf';
hdr.caldates = '20180129';
hdr.data_type = 'cast';
hdr.missing = '-9999';
hdr.delimiter = 'comma';
hdr.inst_mfr = 'Hydroptic'; % /instrument_manufacturer
hdr.inst_model = 'UVP5hd'; % /instrument_model
hdr.data_status.PAR = 'final';
hdr.data_status.ZOO = 'preliminary';
hdr.data_status.Level1b = 'preliminary';
hdr.documents.PAR = {'UVP_Particle_and_Zooplankton_Characterization_Protocol.pdf' 'Write_SEABASS_Level2_UVP_par.m'};
hdr.documents.ZOO = {'UVP_Particle_and_Zooplankton_Characterization_Protocol.pdf'...
'Write_SEABASS_Level2_UVP_zoo.m'...
'WoRMS_AphiaID_taxa_match.m'...
'Ecotaxa_Image_Key_EXPORTS_SR1812.pdf'};
hdr.documents.Level1b = {'UVP_Particle_and_Zooplankton_Characterization_Protocol_Level1b.pdf'...
'Ecotaxa_Image_Key_EXPORTS_SR1812.pdf,namespace_EXPORTSNP_UVP.yml'};
% Comments associated with PAR file
hdr.comments.PAR = {...
'! COMMENTS';...
'!';...
'! Underwater Vision Profiler 5 High Definition Unit Serial Number 207 manufactured by Hydroptic';...
'! This data was exported from the main UVP database Ecotaxa (https://ecotaxa.obs-vlfr.fr)';...
'! Ecotaxa Particle Module project identifier: sn207_2018_exports_np_sr1812_filtered';...
'! R/V Cruise Id = SR1812';...
'! Contributors: ';...
'! UVP operator: Jessica Pretty, University of Alaska Fairbanks';...
'! Data preparation: Brita Irving, University of Alaska Fairbanks';...
'!';...
'! depth represents the center of 5 meter depth bins';...
'! This data is inclusive of all types of particles and plankton'};
% Comments associated with ZOO/taxonomic file
hdr.comments.ZOO = {...
'! COMMENTS';...
'!';...
'! Underwater Vision Profiler 5 High Definition Unit Serial Number 207 manufactured by Hydroptic';...
'! This data was exported from the main UVP database Ecotaxa (https://ecotaxa.obs-vlfr.fr)';...
'! Ecotaxa Particle Module project identifier: sn207_2018_exports_np_sr1812_filtered';...
'! Ecotaxa project page: https://ecotaxa.obs-vlfr.fr/prj/1591';...
'! R/V Cruise Id = SR1812';...
'!';...
'! Contributors: ';...
'! UVP operator: Jessica Pretty, University of Alaska Fairbanks';...
'! Image validation: Rachel Lekanoff, Jessica Pretty, Tristan Biard, Stephanie O�Daly, Andrew McDonnell';...
'! Data preparation: Brita Irving, Rachel Lekanoff, University of Alaska Fairbanks';...
'!';...
'! Expert image validation: Debbie Steinberg and Karen S. Stamieszkin';...
'!';...
'! depth represents the center of 5 meter depth bins';...
'! t008 definition: rectangular/tabular fecal pellets, possibly from salps'};
% Comments associated with Level1b ZOO/taxonomic file
hdr.comments.Level1b = {...
'! COMMENTS';...
'!';...
'! Underwater Vision Profiler 5 High Definition Unit Serial Number 207 manufactured by Hydroptic';...
'! This data was exported from the main UVP database Ecotaxa (https://ecotaxa.obs-vlfr.fr)';...
'! Ecotaxa Particle Module project identifier: sn207_2018_exports_np_sr1812_filtered';...
'! Ecotaxa project page: https://ecotaxa.obs-vlfr.fr/prj/1591';...
'! R/V Cruise Id = SR1812';...
'!';...
'! Contributors: ';...
'! UVP operator: Jessica Pretty, University of Alaska Fairbanks';...
'! Image validation: Rachel Lekanoff, Jessica Pretty, Tristan Biard, Stephanie O�Daly, Andrew McDonnell';...
'! Data preparation: Brita Irving, University of Alaska Fairbanks';...
'!';...
'! Expert image validation: Debbie Steinberg and Karen S. Stamieszkin'};
%% Add more description to non-conforming categories, if known
% This will be written to the YAML file
hdr.nonconforming.not_living = 'category for all non-living or non-conforming particles';
hdr.nonconforming.artefact = 'unidentifiable non-living particles; out of focus, bubbles, human-made objects, etc.';
hdr.nonconforming.badfocus = 'the image was unfocused, blurry, too small, or otherwise unsuitable for a confident annotation';
hdr.nonconforming.bubble = 'air bubbles';
hdr.nonconforming.detritus = 'organic and non-living or senescent particles';
hdr.nonconforming.fiber = 'fibrous particles, possibly from synthetic materials or fecal pellets';
hdr.nonconforming.fluffy = 'unidentified detrital particles fluffy in apperance';
hdr.nonconforming.dark = 'unidentified detrital particles dark in apperance';
hdr.nonconforming.light = 'unidentified detrital particles lightin apperance';
hdr.nonconforming.temporary = 'category for t00x for unidentified but similar particles that can be grouped';
% Descriptions of temporary categories used in image validation
hdr.nonconforming.t001 = 'unsure if non-living or living particles, bean-like shape';
hdr.nonconforming.t002 = 'very long fibers or diatom chains or tentacles';
hdr.nonconforming.t003 = 'portion of possible cnidarian';
hdr.nonconforming.t004 = 'circular particle with one or two dark spots, unsure if living or non-living';
hdr.nonconforming.t005 = ''; % No description available
hdr.nonconforming.t006 = 'cylindrical/round fecal pellets';
hdr.nonconforming.t007 = 'possible fibrous particles or elongated fecal pellets';
hdr.nonconforming.t008 = 'rectangular/tabular fecal pellets, possibly from salps';
BCO-DMO does not have specific formatting requirements, so this formatting was modeled after SeaBASS to some extend.
For example, UVP data has been archived for the 2015 Repeat Hydrography cruise along the p16n line in the North Pacific on the R/V Roger Brown, RR1503. The following metadata file was used in the generation of the ZOO and PAR file.
function hdr = RB1503_UVP_metadata(ecotaxa_file)
%% function RB1503_UVP_METADATA
%
% Description:
% Define metadata used in file header
%
% Input:
% ecotaxa_file | char containing name of ecotaxa file exported (optional)
%
% Output:
% hdr | structure containing requred metadata information
%
% Notes:
% Units different for abundance in PAR [number/L] and ZOO [number/m^3]
% files when exported from Ecotaxa and have not changed this.
%
% BCO-DMO formatting requirements
% Much less stringent, but follow basic formatting of SeaBASS for
% simplicity
%
% Author:
% Brita Irving <bkirving@alaska.edu>
%%
hdr.investigators = 'Andrew M P McDonnell';
hdr.affiliations = 'University of Alaska Fairbanks';
hdr.contact = 'amcdonnell@alaska.edu';
hdr.experiment = 'Climate Variability and Predictability (CLIVAR)';
hdr.cruise = 'RB1503';
hdr.data_type = 'cast';
hdr.missing = '-9999';
hdr.inst_mfr = 'Hydroptic'; % /instrument_manufacturer
hdr.inst_model = 'UVP5'; % /instrument_model
hdr.ecotaxa_name = 'uvp5_sn009_2015_p16n';
% Comments associated with PAR file
% Note: SeaBASS format COMMENTS must be preceeded by "!" and not "#"
hdr.comments.PAR = {...
'# COMMENTS';...
'#';...
'# Underwater Vision Profiler 5 Unit Serial Number 009 manufactured by Hydroptic';...
'# This data was exported from Ecotaxa (https://ecotaxa.obs-vlfr.fr)';...
['# Ecotaxa export source file: ' ecotaxa_file];...
['# Ecotaxa Particle Module project identifier: ' hdr.ecotaxa_name];...
['# R/V Cruise Id = ' hdr.cruise];...
'# Contributors: ';...
'# UVP operators: Andrew M. P. McDonnell (Tahiti to Hawaii) and Jessica S. Turner (Hawaii to Alaska)';...
'# Data preparation: Brita Irving, University of Alaska Fairbanks';...
'#';...
'# This data is inclusive of all types of particles and plankton';...
'# bin_depth represents the center of each 5 meter depth bin';...
'# field names include size bin limits in mm (millimeter) and um (micrometer)';...
'#';...
'# Uncertainty due to counting statistics can be calculated as follows';...
'# For example:';...
'# relative uncertainty [none]';...
'# relative_unc = sqrt(abundance_(102-128_um) * volume)/(abundance_(102-128_um) * volume)';...
'# uncertainty in the number size distribution [number/L]';...
'# abundance_(102-128_um)_unc = relative_unc * abundance_(102-128_um)';...
'# uncertainty in the volume size distribution [mm^3/L]';...
'# biovolume_(102-128_um)_unc = relative_unc * biovolume_(102-128_um)';...
'#'};
% Comments associated with ZOO/taxonomic file
%
hdr.comments.ZOO = {...
'# COMMENTS';...
'#';...
'# Underwater Vision Profiler 5 Unit Serial Number 009 manufactured by Hydroptic';...
'# This data was exported from Ecotaxa (https://ecotaxa.obs-vlfr.fr)';...
['# Ecotaxa export source file: ' ecotaxa_file];...
'# Ecotaxa Particle Module project identifier: uvp5_sn009_2015_p16n';...
'# Ecotaxa project page: https://ecotaxa.obs-vlfr.fr/prj/43';...
['# R/V Cruise Id = ' hdr.cruise];...
'#';...
'# Contributors: ';...
'# UVP operators: Andrew M. P. McDonnell (Tahiti to Hawaii) and Jessica S. Turner (Hawaii to Alaska)';...
'# Image validation: Rachel Lekanoff, Jessica Pretty, Stephanie ODaly, and Benjamin LeBlanc';...
'# Data preparation: Brita Irving, University of Alaska Fairbanks';...
'#';...
'# Data quality disclaimer: image validation performed by non-experts and should be used with great care';...
'# See Ecotaxa_Image_Key_NorthPacific_UAF.pdf for example images of major taxonomic categories used for validation';...
'#';...
'# bin_depth represents the center of each 5 meter depth bin';...
'#';
'# Uncertainty due to counting statistics can be calculated as follows. ';...
'# For example:';...
'# relative uncertainty [none]';...
'# relative_unc = sqrt(abun_{#}id * volume)/(abun_{#}id * volume)';...
'# uncertainty in the abundance [number/m^3]';...
'# abun_{#}id_unc = relative_unc * abun_{#}id';...
'# uncertainty in the biovolume [mm^3/L]';...
'# biovol_{#}id_unc = relative_unc * biovol_{#}id';...
'# uncertainty in the average equivalent spherical diameter [mm]';...
'# avgesd_{#}id_unc = relative_unc * avgesd_{#}id';...
'#'};
% Descriptions of temporary categories used in image validation
hdr.temporary_category.t001 = 'unsure if non-living or living particles, bean-like shape';
hdr.temporary_category.t002 = 'very long fibers or diatom chains or tentacles';
hdr.temporary_category.t003 = 'portion of possible cnidarian';
hdr.temporary_category.t004 = 'circular particle with one or two dark spots, unsure if living or non-living';
hdr.temporary_category.t005 = ''; % No description available
hdr.temporary_category.t006 = 'cylindrical/round fecal pellets';
hdr.temporary_category.t007 = 'possible fibrous particles or elongated fecal pellets';
hdr.temporary_category.t008 = 'rectangular/tabular fecal pellets, possibly from salps';