Summer_Examples/Activity_2_More_Fun_with_OOI_Data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Activity 2 More Fun with OOI Data.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LiPthtG3isLc",
        "colab_type": "text"
      },
      "source": [
        "# Activity 2 - More Fun with OOI Data\n",
        "*Written by Sage Lichtenwalner, Rutgers University, May 31, 2019*\n",
        "\n",
        "The example was developed for the **June 2019 OOI Ocean Data Labs Workshop**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DRdYSjRDwSQJ",
        "colab_type": "text"
      },
      "source": [
        "## Introduction\n",
        "In this Python notebook, we will demonstrate some advanced techniques for working with data from the Ocean Observatories Initiative (OOI). \n",
        "\n",
        "This example was designed to run on Google's Colaboratory platform, though it should also work on any Jupyter notebook platform, assuming the required libraries are installed.  \n",
        "\n",
        "In this notebook, we will demonstrate the following **Data Discovery** steps:\n",
        "5. Quick Plots\n",
        "6. Basic Statistics and Analysis\n",
        "\n",
        "\n",
        "We will continue to use data from the **30m Dissolved Oxygen** sensor on the **[Global Irminger Sea Flanking Mooring A](https://oceanobservatories.org/site/gi03flma/)**, also known as **GI03FLMA-RIS01-03-DOSTAD000**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xCOIdXnRiKNo",
        "colab_type": "text"
      },
      "source": [
        "## Loading the Dataset\n",
        "\n",
        "As in the first example, the first thing we need to do is load the Python libraries we will need to load, process and plot our data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eHk8O3nWiDi4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import xarray as xr\n",
        "!pip install netcdf4==1.5.0\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "sns.set()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lsGiVhgKL8AW"
      },
      "source": [
        "Here is the URL to the datafile we will use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1_UgrbqO7W-f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "single_file = 'https://opendap.oceanobservatories.org/thredds/dodsC/ooi/sage-marine-rutgers/20190307T155319-GI03FLMA-RIS01-03-DOSTAD000-recovered_host-dosta_abcdjm_sio_instrument_recovered/deployment0001_GI03FLMA-RIS01-03-DOSTAD000-recovered_host-dosta_abcdjm_sio_instrument_recovered_20140912T201501-20150818T103001.nc'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g2L-dYR57jpX",
        "colab_type": "text"
      },
      "source": [
        "And now we can load the file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "POgkQjkF4Rai",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Load the data files\n",
        "ds = xr.open_dataset(single_file)\n",
        "ds = ds.swap_dims({'obs': 'time'}) #Swap dimensions\n",
        "df = ds.to_dataframe() #And convert to a Pandas DataFrame\n",
        "print('Dataset has %d points' % df.index.size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SM2nJAQw5_lE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Add your code here to display the first few rows of the dataframe"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7I-DaeO45Knw",
        "colab_type": "text"
      },
      "source": [
        "## Some Advanced Plots\n",
        "In the first activity, we used the \"built-in\" plotting features of Pandas to quickly generate plots.  For example..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7fWdyvOT5nht",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.ctdmo_seawater_temperature.plot();               "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W3v-5c3o6ObM",
        "colab_type": "text"
      },
      "source": [
        "But what if we want to change what is plotted on each axes?  For example if we wanted to plot temperature vs. pressure, or salinity?\n",
        "\n",
        "To do this, we need to use Matplotlib's plotting functions explicitly, which allows us to specify both the x and y axes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y1jBHUga6dSX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Plot Temperature vs. Pressure\n",
        "plt.plot(df.ctdmo_seawater_temperature,df.int_ctd_pressure, linestyle='',marker='.');\n",
        "\n",
        "# Flip the y-axis\n",
        "ax = plt.gca()\n",
        "ax.invert_yaxis()\n",
        "\n",
        "# Label the Plot\n",
        "plt.ylabel('Pressure')\n",
        "plt.xlabel('Temperature');\n",
        "plt.title('Pioneer Central Inshore Profiler');"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2bSX1OmrRT-U",
        "colab_type": "text"
      },
      "source": [
        "We can also use the scatter function, which is sometimes easier to use.\n",
        "\n",
        "Let's demonstrate this by creating a TS diagram."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Erk1_ed3-o-l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# TS Diagram\n",
        "plt.scatter(df.practical_salinity, df.ctdmo_seawater_temperature, s=5);\n",
        "\n",
        "# Label the Plot\n",
        "plt.xlabel('Salinity')\n",
        "plt.ylabel('Temperature');\n",
        "plt.title('Irminger Sea Flanking Mooring A 30m CTD');"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oOylYcId-3wf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Add your code here to plot temperature vs. DO"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HxHtpvfER_eo",
        "colab_type": "text"
      },
      "source": [
        "## Subsetting Datasets\n",
        "In the TS diagram above, there is some spurious data.  My guess is that the line of data heading off to the left is from when the instrument was recovered.\n",
        "\n",
        "Let's subset the data so we can remove that line, and additionally focus in on a narrower time period."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0fCpY0RbSK7y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df2 = df.loc['2014-10-01':'2014-10-31']\n",
        "\n",
        "df2.ctdmo_seawater_temperature.plot();"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H1msh9Q5YY5t",
        "colab_type": "text"
      },
      "source": [
        "## Adding Some Color"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rsk43YibSkQk",
        "colab_type": "text"
      },
      "source": [
        "Next we'll make a more advanced TS diagram, coloring each dot in time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZO5PMtrSjdX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.scatter(df2.practical_salinity,df2.ctdmo_seawater_temperature,s=5,c=df2.index, cmap='viridis')\n",
        "plt.xlabel('Salinity')\n",
        "plt.ylabel('Temperature')\n",
        "\n",
        "# Quick title from the file\n",
        "plt.title(ds.source); \n",
        "\n",
        "# Add a colorbar\n",
        "cbar = plt.colorbar(label='Time');\n",
        "\n",
        "# Fix the colorbar ticks\n",
        "import pandas as pd # We need pandas for this\n",
        "cbar.ax.set_yticklabels(pd.to_datetime(cbar.get_ticks()).strftime(date_format='%Y-%m-%d'));"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0JF4X9d9WpCT",
        "colab_type": "text"
      },
      "source": [
        "We used the default python colorbar for this plot (viridis), but there is a much larger [colormap collection](https://matplotlib.org/users/colormaps.html) available."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "we7qswY75OKh",
        "colab_type": "text"
      },
      "source": [
        "## Statistics"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IYT404m0S_lE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Prepare to be blown away...\n",
        "df.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TVMlkUNcTIbr",
        "colab_type": "text"
      },
      "source": [
        "There's a lot of variables included here, let's trim down the DataFrame to just include the variables we really want."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B2PE2laXTD2l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df[['ctdmo_seawater_temperature','dissolved_oxygen']].describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gfLpGa96UBhT",
        "colab_type": "text"
      },
      "source": [
        "You can also pull out a signle statistic for a single variable, using the max, mean, std and quantile functions.  When in doubt, try tab-complete."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oJK5J0_7TGJD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Add your code here to show statistics for individual variables"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VcEyt1X9R4Jb",
        "colab_type": "text"
      },
      "source": [
        "## Downsampling and Averaging\n",
        "We can also easily calculate hourly, daily and monthly averages. \n",
        "\n",
        "See the [pandas.resample](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.resample.html) doc for more, as well as this list of [offset options](http://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases).\n",
        "\n",
        "That said, if you want to use centered averaging, moving averages, or other more complicated averaging or filtering routines using irregular intervals, you might have to roll-your-own code.\n",
        "\n",
        "Here's a quick example... notice the legend labels we've also added."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vtj-VlNrUkG_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "fig, ax = plt.subplots()\n",
        "fig.set_size_inches(12, 6)\n",
        "df['ctdmo_seawater_temperature'].plot(ax=ax,label='Raw',linestyle='None',marker='.',markersize=2)\n",
        "df['ctdmo_seawater_temperature'].resample('D').mean().plot(ax=ax,label='Daily')\n",
        "df['ctdmo_seawater_temperature'].resample('5D').mean().plot(ax=ax,label='5 Day')\n",
        "df['ctdmo_seawater_temperature'].resample('MS').mean().plot(ax=ax,label='Monthly',marker='d') #MS=Month Start\n",
        "plt.legend();"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5RxH-2e1Uv3t",
        "colab_type": "text"
      },
      "source": [
        "We can adapt this code, and combine it with the export code from Activity 1 to create a downloadable CSV file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZV0NKFHlUvmS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df[['ctdmo_seawater_temperature','practical_salinity','dissolved_oxygen']].resample('D').mean().to_csv('ctd_daily_average.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CrkYRwvGWBH8",
        "colab_type": "text"
      },
      "source": [
        "## Choosing Variables Up Front\n",
        "If we don't want to include all of the variables in our DataFrame when we first generate it, we can specify just the ones we want.  This makes the dataset much smaller to work with and export.  Notice the double brackets `[['var1','var2']]` are needed when specifying a list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AFmJKv3tSFLN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Convert to DataFrame\n",
        "df = ds[['ctdmo_seawater_temperature','practical_salinity','dissolved_oxygen']].to_dataframe()\n",
        "\n",
        "# Drop unnecessary columns\n",
        "df = df.drop(columns=['obs','lon','lat']) \n",
        "\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sdM7I1Sh2hKK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# We can also subset the data to a specific time range\n",
        "import datetime\n",
        "df = df.loc[datetime.date(2014,10,1):datetime.date(2014,11,1)]\n",
        "\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7S2ebR80YwUZ",
        "colab_type": "text"
      },
      "source": [
        "## For even more fun...\n",
        "\n",
        "Well, those are all the basics.  You can check out **Activity 3** to explore some additional datasets, and you can check out the **Profile Examples** notebook to learn how to load and plot datasets from gliders and profilers.\n",
        "\n",
        "To continue the fun of playing with OOI data in python, I also recommend checking out these examples:\n",
        "\n",
        "* [Plotting DO Profiles](https://github.com/ooi-data-review/2018-data-workshops/blob/master/chemistry/examples/Example_3_DO_Profiles.ipynb) - Includes more examples on subsetting data and changing colormaps.\n",
        "* [Multi-instrument Quick Plots](https://github.com/ooi-data-review/2018-data-workshops/blob/master/chemistry/examples/extras1/Multi_instrument_Quick_Plots.ipynb)\n",
        "* [Plotting Profiler Data in Real-Time](https://github.com/ooi-data-review/2018-data-workshops/blob/master/chemistry/examples/extras1/Realtime_Plotting.ipynb)\n",
        "* [Working with ADCP Data](https://github.com/ooi-data-review/2018-data-workshops/blob/master/chemistry/examples/extras1/Working_with_ADCP_Data.ipynb)\n",
        "* And [several others](https://github.com/ooi-data-review/2018-data-workshops/tree/master/chemistry/examples)\n",
        "\n",
        "I'm also working on a number of new examples for the [Ocean Data Labs blog](https://datalab.marine.rutgers.edu/blog/).  Here are the first few...\n",
        "* [Comparison of Air and Seawater Temperatures](https://github.com/ooi-data-lab/blog-notebooks/blob/master/201901/Air%20and%20Sea%20Temps.ipynb)\n",
        "* [Some Basic Statistics of CTD Data](https://github.com/ooi-data-lab/blog-notebooks/blob/master/201901/Statistics.ipynb)\n",
        "* [Temperature Correlations](https://github.com/ooi-data-lab/blog-notebooks/blob/master/201902/Temperature%20Correlations.ipynb)\n",
        "\n",
        "Welcome to the OOI Data World.  Have fun exploring the deep!"
      ]
    }
  ]
}