These R scripts process and clean data from Hypack, ASDL and CTDs on ROV surveys using Phantom, BOOTS or Heavy ROVs, into a 1 Hz time series in csv and spatial data formats. A number of summary files, figures and logs are produced for quality control checks.
This script reads in Hypack. RAW files and extracts all collected data readings, it builds a 1 Hz times series of the sensor data logged by Hypack during the survey, and converts position data from a projected coordinate reference system (UTM North Zones, 8 9 or 10) to an unprojected data set of latitude and longitude as decimal degrees. No interpolation of missing values is done at this stage. The script will preferrentially use the depth and position data records from the user defined 'preferred' depth and position sources. A dive log (in CSV format) is required to generate a 1Hz time series for each dive from the start and end time in UTC. Datetime fields in the dive log must be formatted yyyy-mm-dd hh:mm:ss. This time series can be generated for the entire dive or just the ontransect portions.
This script reads in the log files generated by Advanced Serial data logger. These data tend to vary by ROV, so there is a separate script for Phantom, BOOTS and Heavy BlueROV. Various cleaning steps are completed on the log files, and daily/hourly log files are aggregated into one "Master" log file for each device that spans the duration of the entire survey/cruise. For all log files, timestamps are generated to match the standard R POSIX timestamp format of YYYY-MM-DD HH:MM:SS, and any duplicate values where the readings occur more frequently than 1 Hz are removed. Master files are saved as .CSV files, one per device/sensor.
This script contains functions that check for data quality issues, fill data gaps via linear interpolation and removes outliers and smooths position data. It integrates data from both Hypack .RAW files, CTD data exported as Excel files (.xlsx) and Advanced Serial Data Logger's (ASDL) log files. Where data records are missing from the Hypack .RAW file, and RBR files, backup copies of the missing data are sought from the ASDL log files and are used to fill in the missing data if possible. Linear interpolation is conducted for position data (latitude & longitude), as well as the data from the ROV depth, ROV altitude, ship heading and MiniZeus slant range to target. Outliers in position data are removed and filled in via interpolation. Outliers are located based on the crossdistance between the ROV and the ship and the distances between along track ship positions. Lastly, two seperate smoothing functions are applied to the ROV position data: a running median function (https://stat.ethz.ch/R-manual/R-devel/library/stats/html/runmed.html) and a LOESS smoothing function (http://r-statistics.co/Loess-Regression-With-R.html). These two function both improve on the unsmoothed data set. In general, the LOESS smoother will provide a better (i.e. smoother) final output to the data set; however, in cases where the tracking was exceedingly poor, the running median smoothing function may provide a better output. Both smoothed data sets are provided as outputs from this script, to allow the end user to choose according to their needs. The user should check the dive maps saved in the 3.Figures directory to select the best smoothing option for the survey. Finally, a .CSV file is written for each transect or dive and the entire survey, at a 1 Hz interval and a summary of all dives including length, position and depth.
This script loads in exports from the QAQC script and generates spatial files (in KML and shapefile formats) that represent the location of transects or dives. Spatial lines are created for smoothed and unsmoothed position data.