examples/getting_started/1-Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Why HoloViews?\n",
    "\n",
    "HoloViews is an [open-source](https://github.com/holoviz/holoviews) Python library for data analysis and visualization. Python already has excellent tools like numpy, pandas, and xarray for data processing, and bokeh and matplotlib for plotting, so why yet another library?\n",
    "\n",
    "**HoloViews helps you understand your data better, by letting you work seamlessly with both the data *and* its graphical representation.**\n",
    "\n",
    "HoloViews focuses on bundling your data together with the appropriate metadata to support both analysis and visualization, making your raw data *and* its visualization equally accessible at all times.  This process can be unfamiliar to those used to traditional data-processing and plotting tools, and this getting-started guide is meant to demonstrate how it all works at a high level.  More detailed information about each topic is then provided in the [User Guide.](../user_guide/)\n",
    "\n",
    "With HoloViews, instead of building a plot using direct calls to a plotting library, you first describe your data with a small amount of crucial semantic information required to make it visualizable, then you specify additional metadata as needed to determine more detailed aspects of your visualization. This approach provides immediate, automatic visualization that can be effortlessly requested at any time as your data evolves, rendered automatically by one of the supported plotting libraries (such as Bokeh or Matplotlib). \n",
    "\n",
    "\n",
    "## Tabulated data: subway stations\n",
    "\n",
    "To illustrate how this process works, we will demonstrate some of the key features of HoloViews using a collection of datasets related to transportation in New York City. First let's run some imports to make [numpy](http://numpy.org) and [pandas](http://pandas.pydata.org) accessible for loading the data. Here we start with a table of subway station information loaded from a CSV file with pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import holoviews as hv\n",
    "from holoviews import opts\n",
    "hv.extension('bokeh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the standard way to make the numpy and pandas libraries available in the namespace. We recommend always importing HoloViews as ``hv`` and if you haven't already installed HoloViews, check out the install instructions on our [homepage.](http://holoviews.org)\n",
    "\n",
    "Note that after importing HoloViews as ``hv`` we run ``hv.extension('bokeh')`` to load the bokeh plotting extension, allowing us to generate visualizations with [Bokeh](https://bokeh.org). In the next section we will see how you can use other plotting libraries such as [matplotlib](http://matplotlib.org) and even how you can mix and match between them.\n",
    "\n",
    "Now let's load our subway data using pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "station_info = pd.read_csv('../assets/station_info.csv')\n",
    "station_info.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that this table contains the subway station name, its latitude and longitude, the year it was opened, the number of services available from the station and their names, and finally the yearly ridership (in millions for 2015)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ``Elements`` of visualization\n",
    "\n",
    "We can immediately visualize some of the the data in this table as a scatter plot. Let's view how ridership varies with the number of services offered at each station:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scatter = hv.Scatter(station_info, 'services', 'ridership')\n",
    "scatter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we passed our dataframe to [``hv.Scatter``](../reference/elements/bokeh/Scatter.ipynb) to create an *object* called `scatter`, which is independent of any plotting library but here is visualized using bokeh. HoloViews provides a wide range of Element types, all visible in the [Reference Gallery.](http://holoviews.org/reference/index.html)\n",
    "\n",
    "In this example, `scatter` is a simple wrapper around our dataframe that knows that the 'services' column is the independent variable, normally plotted along the x-axis, and that the 'ridership' column is a dependent variable, plotted on the y-axis. These are our *dimensions* which we will describe in more detail a little later.\n",
    "\n",
    "Given that we have the handle ``scatter`` on our ``Scatter`` object, we can show that it is indeed an object and not a plot by printing it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(scatter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bokeh plot above is simply the rich, visual representation of ``scatter`` which is plotted automatically by HoloViews and displayed automatically in the [Jupyter notebook](https://jupyter.org/). Although HoloViews itself is independent of notebooks, this convenience makes working with HoloViews easiest in the notebook environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compositional ``Layouts``\n",
    "\n",
    "The class [``Scatter``](../reference/elements/bokeh/Scatter.ipynb) is a subclass of ``Element``.  As shown in our [element gallery,](http://holoviews.org/reference/index.html) Elements are the simplest viewable components in HoloViews. Now that we have a handle on ``scatter``, we can demonstrate the composition of these components:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layout = scatter + hv.Histogram(np.histogram(station_info['opened'], bins=24), kdims=['opened'])\n",
    "layout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a single line using the ``+`` operator, we created a new, compositional object called a ``Layout`` built from our scatter visualization and a ``Histogram`` that shows how many subway stations opened in Manhattan since 1900. Note that once again, all the plotting is happening behind the scenes. The ``layout`` is not a plot, it's a new object that exists independently of any given plotting system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(layout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Array data: taxi dropoffs\n",
    "\n",
    "So far we have visualized data in a [pandas ``DataFrame``](http://pandas.pydata.org/) but ``HoloViews`` is as agnostic to data formats as it is to plotting libraries; see [Applying Customizations](../user_guide/03-Applying_Customizations.ipynb) for more information. This means we can work with array data as easily as we can work with  tabular data. To demonstrate this, here are some [numpy arrays](http://www.numpy.org/) relating to taxi dropoff locations in New York City:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "taxi_dropoffs = {hour:arr for hour, arr in np.load('../assets/hourly_taxi_data.npz').items()}\n",
    "print('Hours: {hours}'.format(hours=', '.join(taxi_dropoffs.keys())))\n",
    "print('Taxi data contains {num} arrays (one per hour).\\nDescription of the first array:\\n'.format(num=len(taxi_dropoffs)))\n",
    "np.info(taxi_dropoffs['0'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, this dataset contains 24 arrays (one for each hour of the day) of taxi dropoff locations (by latitude and longitude), aggregated over one month in 2015. The array shown above contains the accumulated dropoffs for the first hour of the day."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compositional  ``Overlays``\n",
    "\n",
    "Once again, we can easily visualize this data with HoloViews by passing our array to [``hv.Image``](../reference/elements/bokeh/Image.ipynb) to create an object named ``image``.  This object has the spatial extent of the data declared as the ``bounds``, in terms of the corresponding range of latitudes and longitudes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bounds = (-74.05, 40.70, -73.90, 40.80)\n",
    "image = hv.Image(taxi_dropoffs['0'], ['lon','lat'], bounds=bounds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HoloViews supports ``numpy``, ``xarray``, and ``dask`` arrays when working with array data (see [Gridded Datasets](4-Gridded_Datasets.ipynb)).  We can also compose elements containing array data with those containing tabular data. To illustrate, let's pass our tabular station data to a [``Points``](../reference/elements/bokeh/Points.ipynb) element, which is used to mark positions in two-dimensional space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "points = hv.Points(station_info, ['lon','lat']).opts(color=\"red\")\n",
    "image + image * points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the left, we have the visual representation of the ``image`` object we declared. Using ``+`` we put it into a ``Layout`` together with a new compositional object created with the ``*`` operator called an ``Overlay``. This particular overlay displays the station positions on top of our image, which works correctly because the data in both elements exists in the same space, namely New York City.\n",
    "\n",
    "The `.opts()` method call for specifying the visual style is part of the HoloViews options system, which is described in the next ['Getting started' section](2-Customization.ipynb).\n",
    "\n",
    "This overlay on the right lets us see the location of all the subway stations in relation to our midnight taxi dropoffs. Of course,  HoloViews allows you to visually express more of the available information with our points.  For instance, you could represent the ridership of each subway by point color or point size. For more information see [Applying Customizations](../user_guide/03-Applying_Customizations.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Effortlessly exploring data\n",
    "\n",
    "You can keep composing datastructures until there are more dimensions than can fit simultaneously on your screen. For instance, you can visualize a dictionary of [``Images``](../reference/elements/bokeh/Image.ipynb) (one for every hour of the day) by declaring a ``HoloMap``: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = {int(hour):hv.Image(arr, ['lon','lat'], bounds=bounds) \n",
    "              for hour, arr in taxi_dropoffs.items()}\n",
    "hv.HoloMap(dictionary, kdims='Hour')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is yet another object that is rendered by the HoloViews plotting system with Bokeh behind the scenes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "holomap = hv.HoloMap(dictionary, kdims='Hour')\n",
    "print(holomap)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As this ``HoloMap`` is a container for our ``Image`` elements, we can use the methods it offers to return new containers. For instance, in the next cell we select three different hours of the morning from the ``HoloMap`` and display them as a ``Layout``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "holomap.select(Hour={3,6,9}).layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the ``select`` method picks values from the specified 'Hour' dimension. The various Elements like ``Scatter`` and ``Image`` all accept two types of dimensions: *key dimensions* (i.e., indexing dimensions or independent variables), and *value dimensions* (resulting data or dependent variables).  These attributes are named ``kdims`` and ``vdims``, respectively, and can be passed as the second and third positional argument for all Elements other than Histogram. As you can see above, the ``HoloMap`` of ``Image``s also has a ``kdims`` argument, allowing it to be indexed over those dimensions.  The ``kdims`` and ``vdims`` accept either single dimensions or lists of dimensions, and let you conveniently express the full space in which your data lives.\n",
    "\n",
    "Note how the ``Image`` elements where the holomap is constructed are declared using key dimensions of ``['lat','lon']``, which describes the fact that New York City is being viewed in terms of longitude and latitude. This semantic information is automatically mapped to our visualization by the HoloViews plotting system, which sets the x-axis and y-axis labels accordingly. In the case of the ``HoloMap`` we used a key dimension of ``'Hour'`` to declare that the interactive slider ranges over the hours of the day."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data as visualization\n",
    "\n",
    "Holomaps are able to compose with elements and other holomaps into overlay and layouts just as easily as you compose two elements together. Here is one such composition where we select a range of longitudes and latitudes from our [``Points``](../reference/elements/bokeh/Points.ipynb) before we overlay them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hotspot = points.select(lon=(-73.99, -73.96), lat=(40.75,40.765))\n",
    "composition = holomap * hotspot\n",
    "\n",
    "composition.opts(\n",
    "    opts.Image(xrotation=90),\n",
    "    opts.Points(color='red', marker='v', size=6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell above we created and styled a composite object with a few short lines of code. Furthermore, this composite object relates tabular and array data and is immediately presented in a way that can be explored interactively. This way of working enables highly productive exploration, allowing new insights to be gained easily. For instance, after exploring with the slider we notice a hotspot of taxi dropoffs at 7am, which we can select as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "composition.select(Hour=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now see that the slice of subway locations was chosen in relation to the hotspot in taxi dropoffs around 7am in the morning. This area of Manhattan just south of Central Park contains many popular tourist attractions, including Times Square, and we can infer that tourists often take short taxi rides from the subway stations into this area. Notice how the customizations applied to this new plot, just as for the original one, which is only possible because of how HoloViews lets you capture the data and its attributes independently of any particular visualization.\n",
    "\n",
    "At this point it may appear that HoloViews is about easily generating explorative, interactive visualizations *from* your data. In fact, as we have been building these visualizations we have actually been working *with* our data, as we can show by examining the ``.data`` attribute of our sliced subway locations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hotspot.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that slicing the HoloViews [``Points``](../reference/elements/bokeh/Points.ipynb) object in the visualization sliced the underlying data, with the structure of the table left intact. We can see that the Times Square 42nd Street station is indeed one of the subway stations surrounding our taxi dropoff hotspot. This seamless interplay and exchange between the raw data and easy-to-generate visualizations of it is crucial to how HoloViews helps you understand your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and rendering objects\n",
    "\n",
    "HoloViews renders objects using a variety of backends, making it easy to export plots to a variety of formats. The ``hv.save`` utility makes it trivial to save the rendered output of a HoloViews object to a file. For instance, if we wanted to save the ``composition`` object from above (including the widgets) to a standalone HTML file, we would simply run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hv.save(composition, 'holomap.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or to save a single frame with no widget:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hv.save(composition.last, 'image.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``hv.save`` defaults to the currently selected backend, which can be overridden by adding an argument like ``backend='matplotlib'``. The Bokeh backend supports ``png`` output, and the Matplotlib backend supports ``png``, ``svg``, ``gif``, or ``mp4``, selected by using the appropriate extension instead of `.html`. In each case, rendering to a given format may require additional libraries to be installed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Onwards\n",
    "\n",
    "The next getting-started section shows how to do [Customization](2-Customization.ipynb) of the visual appearance of your data, allowing you to highlight the most important features and change the look and feel. Other related topics for deeper study:\n",
    " \n",
    "* The above plots did not require any special geographic-data support, but when working with larger areas of the Earth's surface (for which curvature becomes significant) or when overlaying data with geographic features, the separate [GeoViews](https://geoviews.org) library provides convenient geo-specific extensions to HoloViews.\n",
    "* The taxi array data was derived from a very large tabular dataset and rasterized using [datashader](https://datashader.org), an optional add-on to HoloViews and Bokeh that makes it feasible to work with very large datasets in a web browser."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}