notebooks/Surface_Data/Surface Data with Siphon and MetPy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"top\"></a>\n",
    "<div style=\"width:1000 px\">\n",
    "\n",
    "<div style=\"float:right; width:98 px; height:98px;\">\n",
    "<img src=\"https://raw.githubusercontent.com/Unidata/MetPy/master/metpy/plots/_static/unidata_150x150.png\" alt=\"Unidata Logo\" style=\"height: 98px;\">\n",
    "</div>\n",
    "\n",
    "<h1>Working with Surface Observations in Siphon and MetPy</h1>\n",
    "<h3>Unidata Python Workshop</h3>\n",
    "\n",
    "<div style=\"clear:both\"></div>\n",
    "</div>\n",
    "\n",
    "<hr style=\"height:2px;\">\n",
    "\n",
    "<div style=\"float:right; width:250 px\"><img src=\"http://weather-geek.net/images/metar_what.png\" alt=\"METAR\" style=\"height: 200px;\"></div>\n",
    "\n",
    "## Overview:\n",
    "\n",
    "* **Teaching:** 20 minutes\n",
    "* **Exercises:** 20 minutes\n",
    "\n",
    "### Questions\n",
    "1. What's the best way to get surface station data from a THREDDS data server?\n",
    "1. What's the best way to make a station plot of data?\n",
    "1. How can I request a time series of data for a single station?\n",
    "\n",
    "### Objectives\n",
    "1. <a href=\"#ncss\">Use the netCDF Subset Service (NCSS) to request a portion of the data</a>\n",
    "2. <a href=\"#stationplot\">Download data for a single time across stations and create a station plot</a>\n",
    "3. <a href=\"#timeseries\">Request a time series of data and plot</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"ncss\"></a>\n",
    "## 1. Using NCSS to get point data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from siphon.catalog import TDSCatalog\n",
    "\n",
    "# copied from the browser url box\n",
    "metar_cat_url = ('http://thredds.ucar.edu/thredds/catalog/'\n",
    "                 'irma/metar/catalog.xml?dataset=irma/metar/Metar_Station_Data_-_Irma_fc.cdmr')\n",
    "\n",
    "# Parse the xml\n",
    "catalog = TDSCatalog(metar_cat_url)\n",
    "\n",
    "# what datasets are here?\n",
    "print(list(catalog.datasets))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metar_dataset = catalog.datasets['Feature Collection']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we've grabbed the \"Feature Collection\" dataset, we can request a subset of the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Can safely ignore the warnings\n",
    "ncss = metar_dataset.subset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What variables do we have available?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ncss.variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"#top\">Top</a>\n",
    "<hr style=\"height:2px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"stationplot\"></a>\n",
    "## 2. Making a station plot\n",
    " * Make new NCSS query\n",
    " * Request data closest to a time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "query = ncss.query()\n",
    "query.lonlat_box(north=34, south=24, east=-80, west=-90)\n",
    "query.time(datetime(2017, 9, 10, 12))\n",
    "query.variables('temperature', 'dewpoint', 'altimeter_setting',\n",
    "                'wind_speed', 'wind_direction', 'sky_coverage')\n",
    "query.accept('csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the data\n",
    "data = ncss.get_data(query)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to pull apart the data and perform some modifications, like converting winds to components and convert sky coverage percent to codes (octets) suitable for plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import metpy.calc as mpcalc\n",
    "from metpy.units import units\n",
    "\n",
    "# Since we used the CSV data, this is just a dictionary of arrays\n",
    "lats = data['latitude']\n",
    "lons = data['longitude']\n",
    "tair = data['temperature']\n",
    "dewp = data['dewpoint']\n",
    "alt = data['altimeter_setting']\n",
    "\n",
    "# Convert wind to components\n",
    "u, v = mpcalc.wind_components(data['wind_speed'] * units.knots, data['wind_direction'] * units.degree)\n",
    "\n",
    "# Need to handle missing (NaN) and convert to proper code\n",
    "cloud_cover = 8 * data['sky_coverage'] / 100.\n",
    "cloud_cover[np.isnan(cloud_cover)] = 10\n",
    "cloud_cover = cloud_cover.astype(np.int)\n",
    "\n",
    "# For some reason these come back as bytes instead of strings\n",
    "stid = np.array([s.tostring().decode() for s in data['station']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the map using cartopy and MetPy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to create station plots with MetPy is to create an instance of `StationPlot` and call various plot methods, like `plot_parameter`, to plot arrays of data at locations relative to the center point.\n",
    "\n",
    "In addition to plotting values, `StationPlot` has support for plotting text strings, symbols, and plotting values using custom formatting.\n",
    "\n",
    "Plotting symbols involves mapping integer values to various custom font glyphs in our custom weather symbols font. MetPy provides mappings for converting WMO codes to their appropriate symbol. The `sky_cover` function below is one such mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import cartopy.crs as ccrs\n",
    "import cartopy.feature as cfeature\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from metpy.plots import StationPlot, sky_cover\n",
    "\n",
    "# Set up a plot with map features\n",
    "fig = plt.figure(figsize=(12, 12))\n",
    "proj = ccrs.Stereographic(central_longitude=-95, central_latitude=35)\n",
    "ax = fig.add_subplot(1, 1, 1, projection=proj)\n",
    "ax.add_feature(cfeature.STATES, edgecolor='black')\n",
    "ax.coastlines(resolution='50m')\n",
    "ax.gridlines()\n",
    "\n",
    "# Create a station plot pointing to an Axes to draw on as well as the location of points\n",
    "stationplot = StationPlot(ax, lons, lats, transform=ccrs.PlateCarree(),\n",
    "                          fontsize=12)\n",
    "stationplot.plot_parameter('NW', tair, color='red')\n",
    "\n",
    "# Add wind barbs\n",
    "stationplot.plot_barb(u, v)\n",
    "\n",
    "# Plot the sky cover symbols in the center. We give it the integer code values that\n",
    "# should be plotted, as well as a mapping class that can convert the integer values\n",
    "# to the appropriate font glyph.\n",
    "stationplot.plot_symbol('C', cloud_cover, sky_cover)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how there are so many overlapping stations? There's a utility in MetPy to help with that: `reduce_point_density`. This returns a mask we can apply to data to filter the points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Project points so that we're filtering based on the way the stations are laid out on the map\n",
    "proj = ccrs.Stereographic(central_longitude=-95, central_latitude=35)\n",
    "xy = proj.transform_points(ccrs.PlateCarree(), lons, lats)\n",
    "\n",
    "# Reduce point density so that there's only one point within a 200km circle\n",
    "mask = mpcalc.reduce_point_density(xy, 200000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we just plot with `arr[mask]` for every `arr` of data we use in plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up a plot with map features\n",
    "fig = plt.figure(figsize=(12, 12))\n",
    "ax = fig.add_subplot(1, 1, 1, projection=proj)\n",
    "ax.add_feature(cfeature.STATES, edgecolor='black')\n",
    "ax.coastlines(resolution='50m')\n",
    "ax.gridlines()\n",
    "\n",
    "# Create a station plot pointing to an Axes to draw on as well as the location of points\n",
    "stationplot = StationPlot(ax, lons[mask], lats[mask], transform=ccrs.PlateCarree(),\n",
    "                          fontsize=12)\n",
    "stationplot.plot_parameter('NW', tair[mask], color='red')\n",
    "stationplot.plot_barb(u[mask], v[mask])\n",
    "stationplot.plot_symbol('C', cloud_cover[mask], sky_cover)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More examples for MetPy Station Plots:\n",
    "- [MetPy Examples](https://unidata.github.io/MetPy/latest/examples/index.html)\n",
    "- [MetPy Symbol list](https://unidata.github.io/MetPy/latest/api/generated/metpy.plots.StationPlot.html#metpy.plots.StationPlot.plot_symbol)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <b>EXERCISE</b>:\n",
    "     <ul>\n",
    "        <li>Modify the station plot (reproduced below) to include dewpoint, altimeter setting, as well as the station id. The station id can be added using the `plot_text` method on `StationPlot`.</li>\n",
    "        <li>Re-mask the data to be a bit more finely spaced, say: 75km</li>\n",
    "        <li>Bonus Points: Use the `formatter` argument to `plot_parameter` to only plot the 3 significant digits of altimeter setting. (Tens, ones, tenths)</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use reduce_point_density\n",
    "\n",
    "# Set up a plot with map features\n",
    "fig = plt.figure(figsize=(12, 12))\n",
    "ax = fig.add_subplot(1, 1, 1, projection=proj)\n",
    "ax.add_feature(cfeature.STATES, edgecolor='black')\n",
    "ax.coastlines(resolution='50m')\n",
    "ax.gridlines()\n",
    "\n",
    "# Create a station plot pointing to an Axes to draw on as well as the location of points\n",
    "\n",
    "# Plot dewpoint\n",
    "\n",
    "# Plot altimeter setting--formatter can take a function that formats values\n",
    "\n",
    "# Plot station id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load solutions/reduce_density.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"#top\">Top</a>\n",
    "<hr style=\"height:2px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"timeseries\"></a>\n",
    "## 3. Time Series request and plot\n",
    "* Let's say we want the past days worth of data...\n",
    "* ...for Boulder (i.e. the lat/lon)\n",
    "* ...for the variables mean sea level pressure, air temperature, wind direction, and wind_speed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import timedelta\n",
    "\n",
    "# define the time range we are interested in\n",
    "end_time = datetime(2017, 9, 12, 0)\n",
    "start_time = end_time - timedelta(days=2)\n",
    "\n",
    "# build the query\n",
    "query = ncss.query()\n",
    "query.lonlat_point(-80.25, 25.8)\n",
    "query.time_range(start_time, end_time)\n",
    "query.variables('altimeter_setting', 'temperature', 'dewpoint',\n",
    "                'wind_direction', 'wind_speed')\n",
    "query.accept('csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's get the data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ncss.get_data(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(list(data.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What station did we get?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "station_id = data['station'][0].tostring()\n",
    "print(station_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That indicates that we have a Python `bytes` object, containing the 0-255 values corresponding to `'K', 'M', 'I', 'A'`. We can `decode` those bytes into a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "station_id = station_id.decode('ascii')\n",
    "print(station_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the time into datetime objects. We see we have an array with byte strings in it, like station id above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['time']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can use a list comprehension to turn this into a list of date time objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time = [datetime.strptime(s.decode('ascii'), '%Y-%m-%dT%H:%M:%SZ') for s in data['time']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now for the obligatory time series plot..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib.dates import DateFormatter, AutoDateLocator\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 6))\n",
    "ax.plot(time, data['wind_speed'], color='tab:blue')\n",
    "\n",
    "ax.set_title(f'Site: {station_id}      Date: {time[0]:%Y/%m/%d}')\n",
    "ax.set_xlabel('Hour of day')\n",
    "ax.set_ylabel('Wind Speed')\n",
    "ax.grid(True)\n",
    "\n",
    "# Improve on the default ticking\n",
    "locator = AutoDateLocator()\n",
    "hoursFmt = DateFormatter('%H')\n",
    "ax.xaxis.set_major_locator(locator)\n",
    "ax.xaxis.set_major_formatter(hoursFmt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <b>EXERCISE</b>:\n",
    "     <ul>\n",
    "        <li>Pick a different location</li>\n",
    "        <li>Plot temperature and dewpoint together on the same plot</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load solutions/time_series.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"#top\">Top</a>\n",
    "<hr style=\"height:2px;\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:devel]",
   "language": "python",
   "name": "conda-env-devel-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}