docs/ADV_Example.ipynb

{
      "cells": [
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "# ADV Example"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "The following example shows a simple workflow for analyzing ADV data using DOLfYN's tools.\n",
                        "\n",
                        "A typical ADV data workflow is broken down into\n",
                        "  1. Review the raw data\n",
                        "      - Check timestamps\n",
                        "      - Look at velocity data quality, particularly for spiking\n",
                        "  2. Check for spurious datapoints and remove. Replace bad datapoints using interpolation if desired\n",
                        "  3. Rotate the data into principal flow coordinates (streamwise, cross-stream, vertical)\n",
                        "  4. Average the data into bins, or ensembles, of a set time length (normally 5 to 10 min)\n",
                        "  5. Calculate turbulence statistics (turbulence intensity, TKE, Reynolds stresses) of the measured flowfield\n",
                        "\n",
                        "Start by importing the necessary DOLfYN tools:"
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {},
                  "outputs": [],
                  "source": [
                        "# Import core DOLfYN functions\n",
                        "import dolfyn\n",
                        "# Import ADV-specific API tools\n",
                        "from dolfyn.adv import api"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "## Read Raw Instrument Data"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "DOLfYN currently only carries support for the Nortek Vector ADV. The example loaded here is a short clip of data from a test deployment to show DOLfN's capabilities.\n",
                        "\n",
                        "Start by reading in the raw datafile downloaded from the instrument. The `dolfyn.read` function reads the raw file and dumps the information into an xarray Dataset, which contains three groups of variables:\n",
                        "\n",
                        "1. Velocity, amplitude, and correlation of the Doppler velocimetry\n",
                        "2. Measurements of the instrument's bearing and environment\n",
                        "3. Orientation matrices DOLfYN uses for rotating through coordinate frames."
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {
                        "scrolled": true
                  },
                  "outputs": [],
                  "source": [
                        "ds = dolfyn.read('../dolfyn/example_data/vector_data01.VEC')"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "There are two ways to see what's in a DOLfYN Dataset. The first is to simply type the dataset's name to see the standard xarray output. To access a particular variable in a dataset, use dict-style (`ds['vel']`) or attribute-style syntax (`ds.vel`). See the [xarray docs](http://xarray.pydata.org/en/stable/getting-started-guide/quick-overview.html) for more details on how to use the xarray format."
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {},
                  "outputs": [],
                  "source": [
                        "# print the dataset\n",
                        "ds"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "A second way provided to look at data is through the *DOLfYN view*. This view has several convenience methods, shortcuts, and functions built-in. It includes an alternate – and somewhat more informative/compact – description of the data object when in interactive mode. This can be accessed using"
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {},
                  "outputs": [],
                  "source": [
                        "ds_dolfyn = ds.velds\n",
                        "ds_dolfyn"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "## QC'ing Data"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "ADV velocity data tends to have spikes due to Doppler noise, and the common way to \"despike\" the data is by using the phase-space algorithm by Goring and Nikora (2002). DOLfYN integrates this function using a 2-step approach: create a logical mask where True corresponds to a spike detection, and then utilize an interpolation function to replace the spikes."
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {
                        "scrolled": false
                  },
                  "outputs": [],
                  "source": [
                        "# Clean the file using the Goring+Nikora method:\n",
                        "mask = api.clean.GN2002(ds['vel'], npt=5000)\n",
                        "# Replace bad datapoints via cubic spline interpolation\n",
                        "ds['vel'] = api.clean.clean_fill(ds['vel'], mask, npt=12, method='cubic', maxgap=None)\n",
                        "\n",
                        "print('Percent of data containing spikes: {0:.2f}%'.format(100*mask.mean()))\n",
                        "\n",
                        "# If interpolation isn't desired:\n",
                        "ds_nan = ds.copy(deep=True)\n",
                        "ds_nan.coords['mask'] = (('dir','time'), ~mask)\n",
                        "ds_nan['vel'] = ds_nan['vel'].where(ds_nan.mask)"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "## Coordinate Rotations"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "Now that the data has been cleaned, the next step is to rotate the velocity data into true East, North, Up (ENU) coordinates.\n",
                        "\n",
                        "ADVs use an internal compass or magnetometer to determine magnetic ENU directions. The `set_declination` function takes the user supplied magnetic declination (which can be looked up online for specific coordinates) and adjusts the orientation matrix saved within the Dataset. (Note: the \"heading\" variable will not change).\n",
                        "\n",
                        "Instruments save vector data in the coordinate system specified in the deployment configuration file. To make the data useful, it must be rotated through coordinate systems (\"beam\"<->\"inst\"<->\"earth\"<->\"principal\"), done through the `rotate2` function. If the \"earth\" (ENU) coordinate system is specified, DOLfYN will automatically rotate the dataset through the necessary coordinate systems to get there. The `inplace` set as true will alter the input dataset \"in place\", a.k.a. it not create a new dataset."
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {},
                  "outputs": [],
                  "source": [
                        "# First set the magnetic declination\n",
                        "dolfyn.set_declination(ds, declin=10, inplace=True) # declination points 10 degrees East\n",
                        "\n",
                        "# Rotate that data from the instrument to earth frame (ENU):\n",
                        "dolfyn.rotate2(ds, 'earth', inplace=True)"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "Once in the true ENU frame of reference, we can calculate the principal flow direction for the velocity data and rotate it into the principal frame of reference (streamwise, cross-stream, vertical). Principal flow directions are aligned with and orthogonal to the flow streamlines at the measurement location. \n",
                        "\n",
                        "First, the principal flow direction must be calculated through `calc_principal_heading`. As a standard for DOLfYN functions, those that begin with \"calc_*\" require the velocity data for input. This function is different from others in DOLfYN in that it requires place the output in an attribute called \"principal_heading\", as shown below.\n",
                        "\n",
                        "Again we use `rotate2` to change coordinate systems."
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {},
                  "outputs": [],
                  "source": [
                        "ds.attrs['principal_heading'] = dolfyn.calc_principal_heading(ds['vel'])\n",
                        "dolfyn.rotate2(ds, 'principal')"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "## Averaging Data\n",
                        "The next step in ADV analysis is to average the velocity data into time bins (ensembles) and calculate turbulence statistics. These averaged values are then used to calculate turbulence statistics. There are two distinct methods for performing this operation, both of which utilize the same variable inputs and produce identical datasets.\n",
                        "\n",
                        "1. **Object-Oriented Approach** (standard): Define an 'averaging object', create a dataset binned in time, and calculate basic turbulence statistics. This is accomplished by initiating an object from the ADVBinner class and then feeding that object with our dataset.\n",
                        "\n",
                        "2. **Functional Approach** (simple): The same operations can be performed using the functional counterpart of ADVBinner, turbulence_statistics.\n",
                        "\n",
                        "Function inputs shown here are the dataset itself: \n",
                        " - `n_bin`: the number of elements in each bin; \n",
                        " - `fs`: the ADV's sampling frequency in Hz; \n",
                        " - `n_fft`: optional, the number of elements per FFT for spectral analysis; \n",
                        " - `freq_units`: optional, either in Hz or rad/s, of the calculated spectral frequency vector.\n",
                        "\n",
                        "All of the variables in the returned dataset have been bin-averaged, where each average is computed using the number of elements specified in `n_bins`. Additional variables in this dataset include the turbulent kinetic energy (TKE) vector (\"ds_binned.tke_vec\"), the Reynold's stresses (\"ds_binned.stress\"), and the power spectral densities (\"ds_binned.psd\"), calculated for each bin."
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {
                        "scrolled": true
                  },
                  "outputs": [],
                  "source": [
                        "# Option 1 (standard)\n",
                        "binner = api.ADVBinner(n_bin=ds.fs*600, fs=ds.fs, n_fft=1024)\n",
                        "ds_binned = binner.do_avg(ds)\n",
                        "\n",
                        "# Option 2 (simple)\n",
                        "# ds_binned = api.calc_turbulence(ds, n_bin=ds.fs*600, fs=ds.fs, n_fft=1024, freq_units=\"Hz\")"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "The benefit to using `ADVBinner` is that one has access to all of the velocity and turbulence analysis functions that DOLfYN contains. If basic analysis will suffice, the `calc_turbulence` function is the most convienent. Either option can still utilize DOLfYN's shortcuts.\n",
                        "\n",
                        "See the [DOLfYN API](https://dolfyn.readthedocs.io/en/latest/apidoc/dolfyn.binners.html) for the full list of functions and shortcuts. A few examples are shown below.\n",
                        "\n",
                        "Some things to know:\n",
                        "- All functions operate bin-by-bin.\n",
                        "- Some functions will fail if there are NaN's in the data stream (Notably the PSD functions)\n",
                        "- `do_*` functions return a full dataset. The first two inputs are the original dataset and the dataset containing the variables calculated by the function. If an output dataset is not given, it will create one.\n",
                        "- `calc_*` functions return a data variable, which can be added to the dataset with a variable of your choosing. If inputs weren't specified in `ADVBinner`, they can be called here. Most of these functions can take both 3D and 1D velocity vectors as inputs.\n",
                        "- \"Shorcuts\", as referred to in DOLfYN, are functions accessible by the xarray accessor `velds`, as shown below. The list of \"shorcuts\" available through `velds` are listed [here](https://dolfyn.readthedocs.io/en/latest/apidoc/dolfyn.shortcuts.html). Some shorcut variables require the raw dataset, some an averaged dataset.\n",
                        "\n",
                        "For instance, \n",
                        "- `do_var` calculates the binned-variance of each variable in the raw dataset, the complementary to `do_avg`. Variables returned by this function contain a \"_var\" suffix to their name.\n",
                        "- `calc_ti` is calculated from the ratio of the standard deviation of the horizontal velocity magnitude (equivalent to the RMS of turbulent velocity fluctuations) to the mean of the horizontal velocity magnitude\n",
                        "- `calc_psd` calculates the power spectral density (velocity spectra) of the velocity vector\n",
                        "- `calc_csd` calculates the cross spectral power density between each direction of the supplied DataArray. Note that inputs specified in creating the `ADVBinner` object can be overridden or additionally specified for a particular function call.\n",
                        "- `calc_epsilon_LT83` uses the Lumley and Terray 1983 algorithm to estimate the TKE dissipation rate from the isoropic turbulence cascade seen in the spectral. This requires the frequency range of the cascade as input.\n",
                        "- `calc_tke` calculates the TKE (Reynolds normal stress) components\n",
                        "- `calc_stress` calculates the Reynolds shear stress components\n"
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {
                        "scrolled": true
                  },
                  "outputs": [],
                  "source": [
                        "# Calculate the variance of each variable in the dataset and add to the averaged dataset\n",
                        "ds_binned = binner.do_var(ds, out_ds=ds_binned) \n",
                        "\n",
                        "# Calculate the turbulence intensity\n",
                        "ds_binned[\"TI\"] = binner.calc_ti(ds.velds.U_mag)\n",
                        "\n",
                        "# Calculate the power spectral density\n",
                        "ds_binned['auto_spectra'] = binner.calc_psd(ds['vel'], freq_units='Hz')\n",
                        "\n",
                        "# Calculate the cross power spectral density\n",
                        "ds_binned['cross_spectra'] = binner.calc_csd(ds['vel'], freq_units='Hz', n_fft_coh=512)\n",
                        "\n",
                        "# Calculate dissipation rate from isotropic turbulence cascade\n",
                        "ds_binned['dissipation'] = binner.calc_epsilon_LT83(ds_binned['auto_spectra'], ds_binned.velds.U_mag, freq_range=[0.5, 1])\n",
                        "\n",
                        "# Calculate the Reynolds stresses\n",
                        "ds_binned['tke_vec'] = binner.calc_tke(ds[\"vel\"])\n",
                        "ds_binned['stress_vec'] = binner.calc_stress(ds[\"vel\"])"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "Plotting can be taken care of through matplotlib. As an example, the mean spectrum in the streamwise direction is plotted here. This spectrum shows the mean energy density in the flow at a particular flow frequency."
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {},
                  "outputs": [],
                  "source": [
                        "import matplotlib.pyplot as plt\n",
                        "%matplotlib inline\n",
                        "\n",
                        "plt.figure()\n",
                        "plt.loglog(ds_binned['freq'], ds_binned['auto_spectra'].sel(S='Sxx').mean(dim='time'))\n",
                        "plt.xlabel('Frequency [Hz]')\n",
                        "plt.ylabel('Energy Density $\\mathrm{[m^2/s^s/Hz]}$')\n",
                        "plt.title('Streamwise Direction')"
                  ]
            },
            {
                  "cell_type": "markdown",
                  "metadata": {},
                  "source": [
                        "## Saving and Loading DOLfYN datasets\n",
                        "Datasets can be saved and reloaded using the `save` and `load` functions. Xarray is saved natively in netCDF format, hence the \".nc\" extension.\n",
                        "\n",
                        "Note: DOLfYN datasets cannot be saved using xarray's native `ds.to_netcdf`; however, DOLfYN datasets can be opened using `xarray.open_dataset`."
                  ]
            },
            {
                  "cell_type": "code",
                  "execution_count": null,
                  "metadata": {},
                  "outputs": [],
                  "source": [
                        "# Uncomment these lines to save and load to your current working directory\n",
                        "#dolfyn.save(ds, 'your_data.nc')\n",
                        "#ds_saved = dolfyn.load('your_data.nc')"
                  ]
            }
      ],
      "metadata": {
            "interpreter": {
                  "hash": "357206ab7e4935423e95e994af80e27e7e6c0672abcebb9d86ab743298213348"
            },
            "kernelspec": {
                  "display_name": "Python 3.9.7 ('base')",
                  "language": "python",
                  "name": "python3"
            },
            "language_info": {
                  "codemirror_mode": {
                        "name": "ipython",
                        "version": 3
                  },
                  "file_extension": ".py",
                  "mimetype": "text/x-python",
                  "name": "python",
                  "nbconvert_exporter": "python",
                  "pygments_lexer": "ipython3",
                  "version": "3.9.17"
            }
      },
      "nbformat": 4,
      "nbformat_minor": 4
}