-
Notifications
You must be signed in to change notification settings - Fork 1
Real Time File Description
#Table of Contents [Introduction and Scope](#intro) [Data File Definitions and Axis Dimensions](#axes) [Global Attributes](#global) [Variables](#variables) [File Naming Convention](#file_naming) [Reference Tables and Codes](#references)
<a name="intro"/> # Introduction and Scope This page describes the structure and contents of the netCDF file that will be used as part of the dissemination of real time profiling glider data. In creating this file several objectives were addressed. The file aims to be fully compliant with the Climate and Forecast Conventions (v1.6) including the usage of the so called discrete sampling geometries (see [Ch9 of CF 1.6](http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#discrete-sampling-geometries)). Additionally, the global attributes are compliant with [v1.1](http://wiki.esipfed.org/index.php/Attribute_Convention_for_Data_Discovery_%28ACDD%29) of the Attribute Conventions for Dataset Discovery. Finally the data files were designed so as to be easily aggregated using the [THREDDS Data Server](http://www.unidata.ucar.edu/software/tds/). By using aggregation, thousands of individual netCDF files each containing just one segment of data can be presented to the user as a virtual data set representing an entire deployment.
As described below, this file format uses the [Climate and Forecast](http://cf-pcmdi.llnl.gov/documents/cf-conventions/) Trajectory [Feature Type](http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/aphs04.html). This feature represents the most basic or raw geometric form of the data that is still scientifically useful. The trajectory file forms the basis of the complete [data flow](/IOOSProfilingGliders/Real-Time-File-Format/wiki/Real-Time-Data-Flow) from the glider operators to the data assembly center and ultimately onward to the National Data Buoy Center for GTS dissemination. There are other CF feature types that may be derived from the trajectory feature type. For example, the trajectoryProfile feature type is often more useful to modelers for data assimilation and to web or application developers for simple plotting purposes. Once the initial data flow is operational and the DAC is operating, we will explore creating derived types such as the trajectoryProfile.
## Version The template described here is v1.0.
V2.0 will be developed in 2013 and will be focused on harmonizing with IMOS and GROOM/EGO.
<a name="axes"/> # Data File Definitions and Axis Dimensions ## Sampling Patterns
The schematic and definitions below illustrate the sampling characteristics of a profiling glider. Within the real time [data flow](/IOOSProfilingGliders/Real-Time-File-Format/wiki/Real-Time-Data-Flow) individual files will be exchanged in several places. Specifically, the real time data transmitted between a glider operator and the Data Assembly Center will each contain one segment. The Data Assembly Center will pass on the segment files to NDBC who will then create GTS messages for each profile within the segment file. Finally, the DAC will collect and aggregate the segment files into virtual deployments. The aggregation capabilites of THREDDS allow for these virtual deployments to be presented to the user as if they were single files, even though a deployment may comprise thousands of segment files.
This vocabulary will be used in the glider files to enable easy access to one or more of these features from a glider deployment. For example, within a file or aggregations of files, the profile_id can be used to extract all of the data records corresponding to a single profile from within a dataset that may include hundreds or thousands of profiles.

<dl> <dt>Profile</dt> <dd>A single vertically oriented track of a glider, either upward or downward through the water column. A profile is one-half of a <strong>dive</strong>. </dd>
<dt>Dive</dt> <dd>A single vertical <strong>profile</strong> to depth followed by a vertical <strong>profile</strong> towards the surface. A dive does not necessarily begin with or terminate with a surfacing and/or gps fix. </dd>
<dt>Segment</dt> <dd>The set of data collected between 2 gps fixes obtained while the glider is on the surface of the water. The first gps fix is acquired prior to the beginning of a dive and the second gps fix is acquired following the completion of at least one dive. Glider <strong>segments</strong> always consist of at least one, and possibly many <strong>dives</strong>. </dd>
<dt>Deployment</dt> <dd>A series of one or more <strong>segments</strong> completed by a glider between the time of deployment and the time of recovery. </dd> </dl>
## Time Axes There are two axes along which geophysical variables are dimensioned:
time(length=unlimited)
time_uv(length=1)
## String Axes
STRING04 STRING2 STRING4
## The Trajectory Feature Type TODO: Address the various questions in this section.
One complete deployment is represented as a trajectory and THREDDS dataset (aggregation).
Example THREDDS aggregations can be found here: http://tds.maracoos.org/thredds/catalog/GliderScan/catalog.html
The CF feature type (aka [discrete sampling geometry](http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/aphs04.html)) used for these files is trajectory. Accordingly, each real time file contains a variable containing information identifying the trajectory.
The ncml depiction of this information is
<variable name="trajectory" shape="trajectory" type="short">
<attribute name="cf_role" value="trajectory_id" />
<attribute name="comment" value="A trajectory can span multiple data files each containing a single segment." />
<attribute name="long_name" value="Unique identifier for each trajectory feature corresponding to a complete deployment contained in the file" />
</variable>PROPOSAL. For real time individual files the length of the trajectory variable is 1.
What is the value of the trajectory variable? When many segment files are aggregated into a deployment the trajectory variable, which is identical across all of the segment files, indicates that all variables come from the same deployment.
TODO: IS THIS NECESSARY? Do we need to provide strict guidance for the usage of this variable and its shape?
Potential revisions to the template: 1. Use the trajectory variable as a deployment ID (NOTE: this requires an integer ID):
<variable name="trajectory" shape="trajectory" type="short">
<attribute name="cf_role" value="trajectory_id" />
<attribute name="comment" value="A trajectory can span multiple data files each containing a single segment." />
<attribute name="long_name" value="Unique identifier for the deployment to which this segment belongs" />
</variable>2. Same but use string variable type:
<variable name="trajectory" shape="strlen_32" type="char">
<attribute name="cf_role" value="trajectory_id" />
<attribute name="comment" value="A trajectory can span multiple data files each containing a single segment." />
<attribute name="long_name" value="Unique identifier for the trajectory to which this segment belongs" />
</variable><a name="global"/> # Global Attributes
The Global Attributes that are included in the real time glider files are listed and defined below. These attributes derive from several sources as indicated in the square brackets next to the terms. More information on these sources can be found at the following locations.
- <strong>CF1.6</strong> Section 2.6 of the current (v1.6) Climate and Forecast [conventions](http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#description-of-file-contents)
- <strong>ACDD</strong> Attribute Conventions for Dataset Discovery [Home page](http://wiki.esipfed.org/index.php?title=Category:Attribute_Conventions_Dataset_Discovery) and [Current Standard](http://wiki.esipfed.org/index.php/Attribute_Convention_for_Data_Discovery_%28ACDD%29)
- <strong>NODC</strong> Guidance from NOAA's National Oceanographic Data Center on netCDF templates to promote good stewardship and archiving. [NODC Templates](http://www.nodc.noaa.gov/data/formats/netcdf/) and [global attribute suggestions](http://www.nodc.noaa.gov/data/formats/netcdf/#guidancetable). In addition to listing useful attributes NODC provides a style guide to help in populating the global attributes with useful and clear information.
- <strong>IMOS/ANFOG</strong> IMOS Data Management manual version [3.1](http://imos.org.au/fileadmin/user_upload/shared/ANFOG/ANFOG_data_management3_1.pdf)
- <strong>IOOS</strong> Internal discussion within the IOOS Glider Data Team.
<dl> <dt>acknowledgement [ACDD]</dt> <dd>A place to acknowledge various type of support for the project that produced this data.</dd> <dt>cdm_data_type [ACDD]</dt> <dd>This attribute is used by THREDDS to identify the feature type, what THREDDS calls a "dataType". The current THREDDS choices are: Grid, Image, Station, Swath, and Trajectory. The mandatory value to be used with the glider file format is <strong>Trajectory</strong>.</dd> <dt>comment [CF/ACDD] </dt> <dd> Miscellaneous information about the data.</dd> <dt>contributor_name [ACDD]</dt> <dd> A comma separated list with the names of any individuals or institutions that contributed to the creation of this data. </dd> <dt>contributor_role [ACDD] </dt> <dd> A comma separated list with the roles assumed by the individuals or institutions that are referenced in contributor_name. </dd> <dt>creator_email [ACDD] </dt> <dd> Email address for the person principally responsible for creating the digital data set. </dd> <dt>creator_name [ACDD]</dt> <dd> Name of the person principally responsible for creating the digital data set. The "institution" attribute will be used if the "creator_name" attribute does not exist.</dd> <dt>creator_url [ACDD]</dt> <dd> URL to a page with reference information describing creation of the data set. </dd> <dt>date_created [ACDD]</dt> <dd> The date on which the data was created. Following [ISO 8601](http://en.wikipedia.org/wiki/ISO_8601), e.g., 2014-03-24T12:15:41+00:00 or 2014-03-24T12:15:41Z. </dd> <dt>date_issued [ACDD]</dt> <dd> The date on which this data was formally issued. Following [ISO 8601](http://en.wikipedia.org/wiki/ISO_8601), e.g., 2014-03-24T12:15:41+00:00 or 2014-03-24T12:15:41Z. </dd> <dt>date_modified [ACDD]</dt> <dd> The date on which this data was last modified. Following [ISO 8601](http://en.wikipedia.org/wiki/ISO_8601), e.g., 2014-03-24T12:15:41+00:00 or 2014-03-24T12:15:41Z. </dd> <dt>featureType [CF]</dt> <dd>The CF 1.6 DSG featureType used to encode the data in the data set (e.g trajectory) [Section 9.1](http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp6280704) </dd> <dt>format_version [IOOS]</dt> <dd> The version of the IOOS RT Glider template. (e.g. format_version=IOOS_Glider_NetCDF_Trajectory_Template_v1.0) TODO: Further define the controlled vocabulary from which this is taken.</dd> <dt>geospatial_bounds [ACDD]</dt> <dd> Describes geospatial extent using any of the geometric objects (2D or 3D) supported by the Well-Known Text (WKT) format. </dd> <dt>geospatial_lat_max [ACDD] </dt> <dd> The values of geospatial_lon_min and geospatial_lon_max reflect the actual longitude data values. Cases where geospatial_lon_min is greater than geospatial_lon_max indicate the bounding box extends from geospatial_lon_max, through the longitude range discontinuity meridian (either the antimeridian or Prime Meridian), to geospatial_lon_min. </dd> <dt>geospatial_lat_min [ACDD]</dt> <dd> Describes a simple latitude/longitude bounding box. geospatial_lat_min specifies the southernmost latitude; geospatial_lat_max specifies the northernmost latitude; geospatial_lon_min specifies the westernmost longitude; geospatial_lon_max specifies the easternmost longitude of the bounding box. </dd> <dt>geospatial_lat_resolution [ACDD] </dt> <dd> </dd> <dt>geospatial_lat_units [ACDD]</dt> <dd> Further refinement of the geospatial bounding box can be provided by using these units and resolution attributes.</dd> <dt>geospatial_lon_max [ACDD] </dt> <dd> </dd> <dt>geospatial_lon_min [ACDD] </dt> <dd> For a more detailed geospatial coverage, see the suggested geospatial attributes.</dd> <dt>geospatial_lon_resolution </dt> <dd> </dd> <dt>geospatial_lon_units [ACDD] </dt> <dd> </dd> <dt>geospatial_vertical_max [ACDD] </dt> <dd> </dd> <dt>geospatial_vertical_min [ACDD]</dt> <dd> Describes a simple vertical bounding box. For a more detailed geospatial coverage, see the suggested geospatial attributes.</td></tr> <dt>geospatial_vertical_positive [ACDD]</dt> <dd>This attribute indicates which direction is positive (a value of "up" means that z increases up, like units of height, while a value of "down" means that z increases downward, like units of pressure or depth). Required Usage: Select from up or down.</dd> <dt>geospatial_vertical_resolution [ACDD] </dt> <dd> </dd> <dt>geospatial_vertical_units [ACDD] </dt> <dd>This defines the units applied to the geospatial_vertical_min and geospatial_vertical_max attributes. Conform to [udunits](http://www.unidata.ucar.edu/software/udunits/). </dd> <dt>history [CF/ACDD] </dt> <dd> Provides an audit trail for modifications to the original data. Well-behaved generic netCDF filters will automatically append their name and the parameters with which they were invoked to the global history attribute of an input netCDF file. We recommend that each line begin with a timestamp indicating the date and time of day that the program was executed. </dd> <dt>id [ACDD] </dt> <dd> An identifier for the data set, provided by and unique within its naming authority. The combination of the "naming authority" and the "id" should be globally unique, but the id can be globally unique by itself also. IDs can be URLs, URNs, DOIs, meaningful text strings, a local key, or any other unique string of characters. The id should not include blanks.</dd> <dt>institution [CF/ACDD]</dt> <dd> Specifies where the original data was produced. </dd> <dt>keywords [ACDD]</dt> <dd> A comma separated list of key words and phrases.</dd> <dt>keywords_vocabulary [ACDD] </dt> <dd> If you are following a guideline for the words/phrases in your "keywords" attribute, put the name of that guideline here.</dd> <dt>license [ACDD] </dt> <dd> Describe the restrictions to data access and distribution.</dd> <dt>Metadata_Conventions [ACDD]</dt> <dd> This attribute should be set to "Unidata Dataset Discovery v1.0" or the most current version for netCDF files that follow this convention. Note: be careful with the case here. It is indeed "Metadata_Conventions", with the capitalization. Most attributes are normally lower case. *Also note that this is a proposed attribute, and not yet officially part of ACDD. TODO: Propose change to ACDD including a change to the capitalization and to the reference to Unidata. Suggest refering to the ACDD wiki at ESIP. </dd> <dt>metadata_link [ACDD]</dt> <dd> This attribute provides a link to a complete metadata record for this dataset or the collection that contains this dataset. This attribute is not included in Version 1 of the Unidata Attribute Convention for Data Discovery and is also a proposed attribute. It is recommended here because a complete metadata collection for a dataset will likely contain more information than can be included in granule formats. This attribute contains a link to that information. Note: This is a proposed attribute in ACDD. The latest version of the TDS "UDDC" service accepts either "Metadata_Link" or "metadata_link". NODC recommends the use of the lower case form to be more consistent with other attributes, which with only one or two exceptions are also lower case. </dd> <dt>naming_authority [ACDD]</dt> <dd>See id above. The combination of the "naming authority" and the "id" should be a globally unique identifier for the dataset. Recommended Usage: Recommend using the reverse URL of the institution. (ex., gov.noaa.nodc) </dd> <dt>processing_level [ACDD]</dt> <dd> A textual description of the processing (or quality control) level of the data.</dd> <dt>project [ACDD]</dt> <dd> The scientific project that produced the data.</dd> <dt>publisher_email [ACDD] </dt> <dd>The data publisher's name, URL, and email. The publisher may be an individual or an institution. </dd> <dt>publisher_name [ACDD]</dt> <dd> The data publisher's name, URL, and email. The publisher may be an individual or an institution.</dd> <dt>publisher_url [ACDD] </dt> <dd>The data publisher's name, URL, and email. The publisher may be an individual or an institution. </dd> <dt> references [CF]</dt> <dd> Published or web-based references that describe the data or methods used to produce it. </dt> <dt>sea_name [NODC] </dt> <dd> Contains information on the large scale oceanographic name in which the glider is deployed.</dd> <dt> source [CF] </dt> <dd> The definition from [CF](http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#description-of-file-contents): The method of production of the original data. If it was model-generated, source should name the model and its version, as specifically as could be useful. If it is observational, source should characterize it (e.g., "surface observation" or "radiosonde"). Recommended usage: "Observational data form a profiling glider".</dd> <dt>standard_name_vocabulary [ACDD]</dt> <dd> The name of the controlled vocabulary from which variable standard names are taken. As of 05 July 2013 the most current version of the [CF controlled vocabulary](http://cf-pcmdi.llnl.gov/documents/cf-standard-names) list is v25. Recommended usage: CF v25</dd> <dt>summary [ACDD] </dt> <dd> A paragraph describing the dataset.</dd> <dt>time_coverage_duration[ACDD] </dt> <dd> Describes the temporal coverage of the data as a time range. The temporal coverage of the data can be described with any of the following pairs of values: start/end, start/duration, or end/duration. Recommended Usage: Use ISO 8601 for date and time. </dd> <dt>time_coverage_end [ACDD] </dt> <dd> Describes the temporal coverage of the data as a time range. The temporal coverage of the data can be described with any of the following pairs of values: start/end, start/duration, or end/duration. Recommended Usage: Use ISO 8601 for date and time. </dd> <dt>time_coverage_resolution [ACDD] </dt> <dd> Describes the resolution of the time variable in the data. Recommended Usage: Use ISO 8601 for time resolution (examples: P1Y ,P3M, P10D). </dd> <dt>time_coverage_start [ACDD] </dt> <dd> Describes the temporal coverage of the data as a time range. The temporal coverage of the data can be described with any of the following pairs of values: start/end, start/duration, or end/duration. Recommended Usage: Use ISO 8601 for date and time. </dd> <dt>title [CF]</dt> <dd> A succinct description of what is in the dataset.</dd> </dl>
<a name="variables"/> # Variables
## Coordinate Variables time time_uv
## Data Variables Taken largely from the GROOM Data Management Manual v1.0 but also influenced by the NODC templates. When conflicts arise, NODC conventions are used.
TODO: Decide if <PARAM> is selected from or from someplace else. https://code.google.com/p/imos-toolbox/source/browse/trunk/IMOS/imosParameters.txt?spec=svn1368&r=1368
<X>: standardized attributes listed in reference tables [below](#references). <Y>: attributes whos values are set by the PI (Principal Investigator).
float <PARAM>(TIME);
<PARAM>:standard_name = “<X>”;
<PARAM>:units = “<Y>”;
<PARAM>:_FillValue = <Y>;
<PARAM>:long_name = “Y”;
<PARAM>:valid_min = <Y>;
<PARAM>:valid_max = <Y>;
<PARAM>:comment = “<Y>”;
<PARAM>:sensor_mount = <X>;
<PARAM>:sensor_orientation = <X>;
<PARAM>:sensor_name = <Y>;
<PARAM>:sensor_serial_number = <Y>;
<PARAM>:sensor_make = <Y>;
<PARAM>:sensor_model = <Y>;
<PARAM>:ancillary_variables = “<Y>” ;
<PARAM>:accuracy = <Y>;
<PARAM>:precision = <Y>;
<PARAM>:resolution = <Y>;
<PARAM>:cell_methods = “<X>”;
<PARAM>:DM_indicator = “<X>”
<PARAM>:reference_scale = “<Y>”
<PARAM>:coordinates = “time lat lon pressure”
<PARAM>:sdn_parameter_urn = “XXX”; # To be removed
<PARAM>:sdn_uom_urn = “XXX”; # To be removed
<PARAM>:sdn_uom_name = “XXX”; # To be removed
<PARAM>:glider_original_parameter_name = “XXX”; # TBD### Quality Control of Data Variables
byte <PARAM>_qc(TIME);
<PARAM>_qc:standard_name = “standard_name status_flag”;
<PARAM>_qc:long_name = “quality flag”;
<PARAM>_qc:conventions = “TODO: Define reference table. IODE reference.”;
<PARAM>_qc:_FillValue = -128b;
<PARAM>_qc:valid_min = 0b;
<PARAM>_qc:valid_max= 9b;
<PARAM>_qc:flag_values = 0b, 1b, 2b, 3b, 4b, 5b, 8b, 9b;
<PARAM>_qc:flag_meanings = “no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed interpolated_value missing_value” <dl> <dt>standard_name = "<standard_name> status_flag"; </dt> <dd> </dd> <dt> long_name = “quality flag”; </dt> <dd> </dd> <dt> conventions = “EGO reference table 2”; </dt> <dd> </dd> <dt> _FillValue = -128b; </dt> <dd> </dd> <dt> valid_min = 0b; </dt> <dd> </dd> <dt> valid_max= 9b; </dt> <dd> </dd> <dt> flag_values = 0b, 1b, 2b, 3b, 4b, 5b, 8b, 9b; </dt> <dd> </dd> <dt> flag_meanings = “no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed interpolated_value missing_value” </dt> <dd> </dd> </dl>
### Segment Averaged Velocity Estimates TODO: Add a description of the estimation process for the segment averaged velocity variables.
## Dimensionless Container Variables The NODC netCDF templates have introduced a convention that is very useful for encoding platform and instrument metadata in a modular way. Dimensionless container variables act as place-holders for platform and instrument metadata and are easily referenced from geophysical variables with variable attributes. For example, the following variable, platform, contains information describing the glider platform used to collect the measurements.
int platform
platform:type = "spray"
platform:wmo_id = "NNYYXX" ;
platform:comment = "Spray Glider sp111" ;
platform:id = "sp111" ;
platform:long_name = "Spray Glider sp111" ;
platform:instrument = "instrument_ctd" ;Similarly, the following instrument container variable contains metadata for the CTD sensor that is mounted on the above "spray" glider platform.
int instrument_ctd
instrument_ctd:comment = "Unpumped CTD with a nominal sampling rate of 1Hz." ;
instrument_ctd:serial_number = -1 ;
instrument_ctd:long_name = "Seabird SBD 41CP Conductivity, Temperature, Depth Sensor." ;
instrument_ctd:platform = "platform"<a name="optional"/> ## Optional Variables Discussion topic: Optional variables are acceptable to the DAC and will be passed through without further processing.
<a name="file_naming"/> # File Naming Convention The following list specifies the 4 file types which will be accepted by the IOOS National Glider Data Acquisition Center and the proposed naming conventions for each: <ul> <li><strong>glider-SN_yyyymmddTHHMMSS_rt0.nc</strong>: Real-time data with no QC. This is the minimum processing level accepted by the DAC and contains the raw data values with no operator provided quality control.</li> <li><strong>glider-SN_yyyymmddTHHMMSS_rt1.nc</strong>: Real-time data with operator provided QC</li> <li><strong>glider-SN_yyyymmddTHHMMSS_delayed0.nc</strong>: Delayed-mode data with no QC</li> <li><strong>glider-SN_yyyymmddTHHMMSS_delayed1.nc</strong>: Delayed-mode data with operator provided QC</li> </ul>
where <dl> <dt><strong>glider</strong></dt><dd> Identifying name or type abbreviation for the glider</dd> <dt><strong>SN</strong></dt><dd> vehicle serial number as provided by the manufacturer</dd> <dt><strong>yyyymmddTHHMMSS</strong></dt><dd> http://en.wikipedia.org/wiki/ISO_8601 [ISO 8601](http://en.wikipedia.org/wiki/ISO_8601) formatted date representing the start time of the data acquisition</dd> <dt><strong>'rt' or 'delayed'</strong></dt><dd> string specifying real-time (during deployment) or delayed mode (post-recovery) data acquisition</dd> <dt><strong>0 or 1</strong></dt><dd> quality control level where 0 corresponds to none and 1 corresponds to some level of operator applied quality control</dd> </dl>
Ideally, the <strong>glider-SN_yyyymmddTHHMMSS_rt1.nc</strong> files will be provided by the individual operators during the deployment and the <strong>glider-SN_yyyymmddTHHMMSS_delayed1.nc</strong> files will be provided after the glider has been recovered and the full data set processed. It is expected that all files containing operator QC'd data will provide the appropriate <strong>VARIABLE_qc</strong> variables with corresponding attributes from the <strong>[CF](http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html)</strong> specification.
<a name="references"/> # Reference Tables and Codes
QC Codes
Platform Types: The platfrom:type attribute is selected from the following controlled vocabulary. 1. slocum 2. spray 3. seaglider
## Quality Control