vis-layers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "95f0a171",
   "metadata": {},
   "source": [
    "(vis-layers)=\n",
    "# Layers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fdd3b8a",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In the previous chapters, you've learned much more than just how to make scatterplots, bar charts, and boxplots.\n",
    "You learned a foundation that you can use to make *any* type of plot with **lets-plot**.\n",
    "\n",
    "In this chapter, you'll expand on that foundation as you learn about the layered grammar of graphics.\n",
    "We'll start with a deeper dive into aesthetic mappings, geometric objects, and facets.\n",
    "Then, you will learn about statistical transformations **lets-plot** makes under the hood when creating a plot.\n",
    "These transformations are used to calculate new values to plot, such as the heights of bars in a bar plot or medians in a box plot.\n",
    "You will also learn about position adjustments, which modify how geoms are displayed in your plots.\n",
    "Finally, we'll briefly introduce coordinate systems.\n",
    "\n",
    "We will not cover every single function and option for each of these layers, but we will walk you through the most important and commonly used functionality provided by **lets-plot**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17575f3a",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "\n",
    "You will need to install the **letsplot** package for this chapter, as well as **pandas**.\n",
    "\n",
    "In your Python session, import the libraries we'll be using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a86fb211",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from lets_plot import *\n",
    "from lets_plot.geo_data import *\n",
    "from lets_plot.mapping import as_discrete\n",
    "\n",
    "LetsPlot.setup_html()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55b00fde",
   "metadata": {},
   "source": [
    "## Aesthetic mappings\n",
    "\n",
    "> \"The greatest value of a picture is when it forces us to notice what we never expected to see.\" --- John Tukey\n",
    "\n",
    "We're going to use the `mpg` dataset for this section, so let's download it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39a6d993",
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg = pd.read_csv(\n",
    "    \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\", index_col=0\n",
    ")\n",
    "\n",
    "mpg = mpg.astype(\n",
    "    {\n",
    "        \"manufacturer\": \"category\",\n",
    "        \"model\": \"category\",\n",
    "        \"displ\": \"double\",\n",
    "        \"year\": \"int64\",\n",
    "        \"cyl\": \"int64\",\n",
    "        \"trans\": \"category\",\n",
    "        \"drv\": \"category\",\n",
    "        \"cty\": \"double\",\n",
    "        \"hwy\": \"double\",\n",
    "        \"fl\": \"category\",\n",
    "        \"class\": \"category\",\n",
    "    }\n",
    ")\n",
    "mpg.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d6f1307",
   "metadata": {},
   "source": [
    "Among the variables in `mpg` are:\n",
    "\n",
    "1.  `displ`: A car's engine size, in liters.\n",
    "    A numerical variable.\n",
    "\n",
    "2.  `hwy`: A car's fuel efficiency on the highway, in miles per gallon (mpg).\n",
    "    A car with a low fuel efficiency consumes more fuel than a car with a high fuel efficiency when they travel the same distance.\n",
    "    A numerical variable.\n",
    "\n",
    "3.  `class`: Type of car.\n",
    "    A categorical variable.\n",
    "\n",
    "Let's start by visualising the relationship between `displ` and `hwy` for various `class`es of cars.\n",
    "We can do this with a scatterplot where the numerical variables are mapped to the `x` and `y` aesthetics and the categorical variable is mapped to an aesthetic like `color` or `shape`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe77349a",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\", color=\"class\")) + geom_point())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e77b5640",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\", shape=\"class\")) + geom_point())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53e51510",
   "metadata": {},
   "source": [
    "Similarly, we can map `class` to `size` or `alpha` aesthetics as well, which control the shape and the transparency of the points, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef221330",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\", size=\"class\")) + geom_point())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d042255e",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\", alpha=\"class\")) + geom_point())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec07349f",
   "metadata": {},
   "source": [
    "While we are able to do it, mapping an unordered discrete (categorical) variable (`class`) to an ordered aesthetic variable (`size` or `alpha`) is generally not a good idea because it implies a ranking that does not in fact exist.\n",
    "\n",
    "Once you map an aesthetic, **lets-plot** takes care of the rest.\n",
    "It selects a reasonable scale to use with the aesthetic, and it constructs a legend that explains the mapping between levels and values.\n",
    "For x and y aesthetics, **lets-plot** does not create a legend, but it creates an axis line with tick marks and a label.\n",
    "The axis line provides the same information as a legend; it explains the mapping between locations and values.\n",
    "\n",
    "You can also set the visual properties of your geom manually as an argument of your geom function (*outside* of `aes()`) instead of relying on a variable mapping to determine the appearance.\n",
    "For example, we can make all of the points in our plot blue:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "618edcb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\")) + geom_point(color=\"blue\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52611640",
   "metadata": {},
   "source": [
    "Here, the colour doesn't convey information about a variable, but only changes the appearance of the plot.\n",
    "You'll need to pick a value that makes sense for that aesthetic:\n",
    "\n",
    "-   The name of a color as a character string, e.g., `color = \"blue\"`\n",
    "-   The size of a point in mm, e.g., `size = 1`\n",
    "-   The shape of a point as a number, e.g, `shape = 1`.\n",
    "\n",
    "Try changing the above plot but, instead of specifying colour, try specifying the shape aesthetic. What do you get with shape set to 1, 2, or 3?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d49ff8",
   "metadata": {},
   "source": [
    "So far we have discussed aesthetics that we can map or set in a scatterplot, when using a point geom.\n",
    "\n",
    "The specific aesthetics you can use for a plot depend on the geom you use to represent the data.\n",
    "In the next section we dive deeper into geoms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f1da019",
   "metadata": {},
   "source": [
    "1.  Create a scatterplot of `hwy` vs. `displ` where the points are pink filled in triangles.\n",
    "\n",
    "2.  Why does the following code not result in a plot with blue points?\n",
    "\n",
    "    ```python\n",
    "    (\n",
    "        ggplot(mpg) + \n",
    "      geom_point(aes(x = \"displ\", y = \"hwy\", color = \"blue\"))\n",
    "    )\n",
    "    ```\n",
    "\n",
    "3.  What does the `stroke` aesthetic do?\n",
    "    What shapes does it work with?\n",
    "    (Hint: use `stroke` in the global aesthetic and `shape` in `geom_point()`)\n",
    "\n",
    "4.  Try changing the last plot from above but, instead of specifying colour, try specifying the shape aesthetic. What do you get with shape set to 1, 2, or 3?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83aa98f0",
   "metadata": {},
   "source": [
    "## Geometric objects\n",
    "\n",
    "How are these two plots similar?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "277a4c0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\")) + geom_point(size=4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07247ba9",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\")) + geom_smooth(method=\"loess\", size=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58824a10",
   "metadata": {},
   "source": [
    "Both plots contain the same x variable, the same y variable, and both describe the same data.\n",
    "But the plots are not identical.\n",
    "Each plot uses a different geometric object, geom, to represent the data.\n",
    "The plot on the left uses the point geom, and the plot on the right uses the smooth geom, a smooth line fitted to the data.\n",
    "\n",
    "To change the geom in your plot, change the geom function that you add to `ggplot()`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0db26c6c",
   "metadata": {},
   "source": [
    "Every geom function in **lets-plot** takes a `mapping` argument, either defined locally in the geom layer or globally in the `ggplot()` layer.\n",
    "However, not every aesthetic works with every geom.\n",
    "You could set the shape of a point, but you couldn't set the \"shape\" of a line.\n",
    "If you try, **lets-plot** will silently ignore that aesthetic mapping.\n",
    "On the other hand, you *could* set the linetype of a line.\n",
    "`geom_smooth()` will draw a different line, with a different linetype, for each unique value of the variable that you map to linetype.\n",
    "\n",
    "Let's take a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b20c825",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\", line=\"drv\")) + geom_smooth(method=\"loess\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84df3e78",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\", linetype=\"drv\")) + geom_smooth(method=\"loess\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b114911",
   "metadata": {},
   "source": [
    "Here, `geom_smooth()` separates the cars into three lines based on their `drv` value, which describes a car's drive train.\n",
    "One line describes all of the points that have a `4` value, one line describes all of the points that have an `f` value, and one line describes all of the points that have an `r` value.\n",
    "Here, `4` stands for four-wheel drive, `f` for front-wheel drive, and `r` for rear-wheel drive.\n",
    "\n",
    "If this is too confusing, we can make it clearer by overlaying the lines on top of the raw data and then coloring everything according to `drv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9e8d92f",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(mpg, aes(x=\"displ\", y=\"hwy\", color=\"drv\"))\n",
    "    + geom_point()\n",
    "    + geom_smooth(aes(linetype=\"drv\"), method=\"loess\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6392da1",
   "metadata": {},
   "source": [
    "Notice that this plot contains two geoms in the same graph.\n",
    "\n",
    "Many geoms, like `geom_smooth()`, use a single geometric object to display multiple rows of data.\n",
    "For these geoms, you can set the `group` aesthetic to a categorical variable to draw multiple objects.\n",
    "**lets-plot** will draw a separate object for each unique value of the grouping variable.\n",
    "In practice, **lets-plot** will automatically group the data for these geoms whenever you map an aesthetic to a discrete variable.\n",
    "It is convenient to rely on this feature because the `group` aesthetic by itself does not add a legend or distinguishing features to the geoms.\n",
    "\n",
    "Note that if you place mappings in a geom function, **lets-plot** will treat them as local mappings for the layer.\n",
    "It will use these mappings to extend or overwrite the global mappings *for that layer only*.\n",
    "This makes it possible to display different aesthetics in different layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3916558",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\")) + geom_point(aes(color=\"class\")) + geom_smooth())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88708546",
   "metadata": {},
   "source": [
    "You can use the same idea to specify different data for each layer.\n",
    "Here, we use red points as well as open circles to highlight two-seater cars.\n",
    "The local data argument in `geom_point()` overrides the global data argument in `ggplot()` for that layer only.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38870eb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(mpg, aes(x=\"displ\", y=\"hwy\"))\n",
    "    + geom_point()\n",
    "    + geom_point(data=mpg.loc[mpg[\"class\"] == \"2seater\", :], color=\"red\", size=2)\n",
    "    + geom_point(\n",
    "        data=mpg.loc[mpg[\"class\"] == \"2seater\", :], shape=1, size=3, color=\"red\"\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35a3d017",
   "metadata": {},
   "source": [
    "Geoms are the fundamental building blocks of **lets-plot**.\n",
    "You can completely transform the look of your plot by changing its geom, and different geoms can reveal different features of your data.\n",
    "\n",
    "**lets-plot** provides over 40 geoms but these don't cover all possible plots one could make. You can find an overview at the relevant part of the [**lets-plot** documentation](https://lets-plot.org/pages/api.html#geometries).\n",
    "\n",
    "If you need a geom that is not included, you have three main options:\n",
    "1. Look for packages that extend **lets-plot** and that do what you need\n",
    "2. Raise an issue on the [**lets-plot** Github page](https://github.com/JetBrains/lets-plot) requesting it as a new feature—but bear in mind that it might not be a priority for the maintainers, and there's no guarantee that they'll add it, depending on how useful it is for others and how easy it is to implemet.\n",
    "3. Turn to an imperative plotting package that gives you fine-grained control so you can build your own chart from the ground up—[**matplotlib**](https://matplotlib.org/) is absolutely excellent for this.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a12f36",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1.  What geom would you use to draw a line chart?\n",
    "    A boxplot?\n",
    "    A histogram?\n",
    "    An area chart?\n",
    "\n",
    "2.  What effect would running the previous example:\n",
    "\n",
    "    ```python\n",
    "    (\n",
    "    ggplot(mpg, aes(x = \"displ\", y = \"hwy\", alpha = \"class\")) +\n",
    "      geom_point()\n",
    "    )\n",
    "    ```\n",
    "    with the keyword argument `show_legend=False` have on the chart generated by this code?\n",
    "\n",
    "3.  What does the `se` argument to `geom_smooth()` do?\n",
    "\n",
    "4.  Recreate the Python code necessary to generate the following graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae75c5c1",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(mpg, aes(x=\"displ\", y=\"hwy\", color=\"drv\"))\n",
    "    + geom_smooth(aes(group=\"drv\"), se=False, method=\"loess\")\n",
    "    + geom_point()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c1d45ab",
   "metadata": {},
   "source": [
    "## Facets\n",
    "\n",
    "In {ref}`data-visualise`, you learned about faceting with `facet_wrap()`, which splits a plot into subplots that each display one subset of the data based on a categorical variable.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb651300",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\")) + geom_point() + facet_wrap(\"cyl\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cd9ef67",
   "metadata": {},
   "source": [
    "To facet your plot with the combination of two variables, switch from `facet_wrap()` to `facet_grid()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61481052",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\")) + geom_point() + facet_grid(\"drv\", \"cyl\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f502141",
   "metadata": {},
   "source": [
    "By default each of the facets share the same scale and range for x and y axes.\n",
    "This is useful when you want to compare data across facets, and is the recommended default, but it can be limiting when you want to visualise the relationship within each facet better.\n",
    "Setting the `scales` argument in a faceting function to `\"free\"` will allow for different axis scales across both rows and columns, `\"free_x\"` will allow for different scales across rows, and `\"free_y\"` will allow for different scales across columns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adcd9079",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(mpg, aes(x=\"displ\", y=\"hwy\"))\n",
    "    + geom_point()\n",
    "    + facet_grid(\"drv\", \"cyl\", scales=\"free_y\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ceb2a354",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg) + geom_point(aes(x=\"displ\", y=\"hwy\")) + facet_wrap(\"class\", nrow=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3a949f",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1.  What happens if you facet on a continuous variable?\n",
    "\n",
    "2.  What do the empty cells in plot with `facet_grid(\"drv\", \"cyl\")` mean?\n",
    "    Run the following code.\n",
    "    How do they relate to the resulting plot?\n",
    "\n",
    "    ```python\n",
    "        (\n",
    "          ggplot(mpg) + \n",
    "      geom_point(aes(x = \"drv\", y = \"cyl\"))\n",
    "        )\n",
    "    ```\n",
    "\n",
    "3.  What plots does the following code make?\n",
    "    What does omitting the second variable do?\n",
    "\n",
    "    ```python\n",
    "    (\n",
    "      ggplot(mpg) + \n",
    "      geom_point(aes(x = \"displ\", y = \"hwy\")) +\n",
    "      facet_grid(\"drv\")\n",
    "    )\n",
    "\n",
    "    (\n",
    "      ggplot(mpg) + \n",
    "      geom_point(aes(x = displ, y = \"hwy\")) +\n",
    "      facet_grid(\"cyl\")\n",
    "    )\n",
    "    ```\n",
    "\n",
    "4.  Take the first faceted plot in this section:\n",
    "\n",
    "    ```python\n",
    "    (\n",
    "        ggplot(mpg) + \n",
    "          geom_point(aes(x = \"displ\", y = \"hwy\")) + \n",
    "          facet_wrap(\"class\", nrow = 2)\n",
    "    )\n",
    "    ```\n",
    "\n",
    "    What are the advantages to using faceting instead of the color aesthetic?\n",
    "    What are the disadvantages?\n",
    "    How might the balance change if you had a larger dataset?\n",
    "\n",
    "\n",
    "5.  Read `help(facet_wrap)` or hover your mouse over `facet_wrap()` in Visual Studio Code.\n",
    "    What does `nrow` do?\n",
    "    What does `ncol` do?\n",
    "    What other options control the layout of the individual panels?\n",
    "    Why doesn't `facet_grid()` have `nrow` and `ncol` arguments?\n",
    "\n",
    "6.  Recreate the following plot using `facet_wrap()` instead of `facet_grid()`.\n",
    "    How do the positions of the facet labels change?\n",
    "\n",
    "    ```python\n",
    "    (\n",
    "        ggplot(mpg) + \n",
    "          geom_point(aes(x = \"displ\", y = \"hwy\")) +\n",
    "          facet_grid(\"drv\")\n",
    "    )\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cacf1fb5",
   "metadata": {},
   "source": [
    "## Statistical transformations\n",
    "\n",
    "Consider a basic bar chart, drawn with `geom_bar()` or `geom_col()`.\n",
    "The following chart displays the total number of diamonds in the `diamonds` dataset, grouped by `cut`.\n",
    "The `diamonds` dataset  contains information on \\~54,000 diamonds, including the `price`, `carat`, `color`, `clarity`, and `cut` of each diamond. We'll load it in a moment.\n",
    "The chart shows that more diamonds are available with high quality cuts than with low quality cuts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f379e31b",
   "metadata": {},
   "outputs": [],
   "source": [
    "diamonds = pd.read_csv(\n",
    "    \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/diamonds.csv\",\n",
    "    index_col=0,\n",
    ")\n",
    "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n",
    "diamonds[\"cut\"] = diamonds[\"cut\"].astype(\n",
    "    pd.CategoricalDtype(categories=diamonds_cut_order, ordered=True)\n",
    ")\n",
    "diamonds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8faf1ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(diamonds, aes(x=\"cut\")) + geom_bar())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a27666cf",
   "metadata": {},
   "source": [
    "On the x-axis, the chart displays `cut`, a variable from `diamonds`.\n",
    "On the y-axis, it displays count, but count is not a variable in `diamonds`!\n",
    "Where does count come from?\n",
    "Many graphs, like scatterplots, plot the raw values of your dataset.\n",
    "Other graphs, like bar charts, calculate new values to plot:\n",
    "\n",
    "-   Bar charts, histograms, and frequency polygons bin your data and then plot bin counts, the number of points that fall in each bin.\n",
    "\n",
    "-   Smoothers fit a model to your data and then plot predictions from the model.\n",
    "\n",
    "-   Boxplots compute the five-number summary of the distribution and then display that summary as a specially formatted box.\n",
    "\n",
    "The algorithm used to calculate new values for a graph is called a **stat**, short for statistical transformation.\n",
    "The figure below shows how this process works with `geom_bar()`.\n",
    "\n",
    "![](https://r4ds.hadley.nz/images/visualization-stat-bar.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62519f73",
   "metadata": {},
   "source": [
    "You can learn which stat a geom uses by inspecting the default value for the `stat` argument.\n",
    "For example, `help(geom_bar)` (or hovering your mouse over the function written in code) shows that the default value for `stat` is \"count\", which means that `geom_bar()` uses counts of the number of occurrences.\n",
    "\n",
    "Every geom has a default stat; and every stat has a default geom.\n",
    "This means that you can typically use geoms without worrying about the underlying statistical transformation.\n",
    "However, there are some reasons why you might need to use a stat explicitly; for example, you might want to override the default stat. In the code below, we change the stat of `geom_bar()` from count (the default) to identity. This lets us map the height of the bars to the raw values of a y variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca772dd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(\n",
    "        diamonds.value_counts(\"cut\").reset_index(name=\"counts\"),\n",
    "        aes(x=\"cut\", y=\"counts\"),\n",
    "    )\n",
    "    + geom_bar(stat=\"identity\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e365aaaf",
   "metadata": {},
   "source": [
    "## Position adjustments\n",
    "\n",
    "There's one more piece of magic associated with bar charts.\n",
    "You can colour a bar chart using either the `color` aesthetic, or, more usefully, the `fill` aesthetic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8da7d91",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"drv\", color=\"drv\")) + geom_bar())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "088e7550",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"drv\", fill=\"drv\")) + geom_bar())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3f7a0a3",
   "metadata": {},
   "source": [
    "Note what happens if you map the fill aesthetic to another variable, like `class`: the bars are automatically stacked.\n",
    "Each colored rectangle represents a combination of `drv` and `class`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "181c70d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"drv\", fill=\"class\")) + geom_bar())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3f74621",
   "metadata": {},
   "source": [
    "The stacking is performed automatically using the **position adjustment** specified by the `position` argument.\n",
    "If you don't want a stacked bar chart, you can use one of three other options: `\"identity\"`, `\"dodge\"` or `\"fill\"`.\n",
    "\n",
    "-   `position = \"identity\"` will place each object exactly where it falls in the context of the graph.\n",
    "    This is not very useful for bars, because it overlaps them.\n",
    "    To see that overlapping we usually need to make the bars slightly transparent by setting `alpha` to a small value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8e9c378",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"drv\", fill=\"class\")) + geom_bar(alpha=0.5, position=\"identity\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2aeffbf7",
   "metadata": {},
   "source": [
    "The identity position adjustment is more useful for 2d geoms, like points, where it is the default.\n",
    "\n",
    "-   `position = \"fill\"` works like stacking, but makes each set of stacked bars the same height.\n",
    "    This makes it easier to compare proportions across groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14205000",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"drv\", fill=\"class\")) + geom_bar(position=\"fill\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a8e72bd",
   "metadata": {},
   "source": [
    "-   `position = \"dodge\"` places overlapping objects directly *beside* one another.\n",
    "    This makes it easier to compare individual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c33c4a03",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"drv\", fill=\"class\")) + geom_bar(position=\"dodge\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd46f22b",
   "metadata": {},
   "source": [
    "There's one other type of adjustment that's not useful for bar charts, but can be very useful for scatterplots.\n",
    "Recall our first scatterplot.\n",
    "Did you notice that the plot displays only some of the points (even though there are 234 observations in the dataset)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba4161de",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\")) + geom_point())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28a3bfc7",
   "metadata": {},
   "source": [
    "The underlying values of `hwy` and `displ` are rounded so the points appear on a grid and many points overlap each other.\n",
    "This problem is known as **overplotting**.\n",
    "This arrangement makes it difficult to see the distribution of the data.\n",
    "Are the data points spread equally throughout the graph, or is there one special combination of `hwy` and `displ` that contains 109 values?\n",
    "\n",
    "You can avoid this gridding by setting the position adjustment to \"jitter\".\n",
    "`position = \"jitter\"` adds a small amount of random noise to each point.\n",
    "This spreads the points out because no two points are likely to receive the same amount of random noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "414ce7af",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"displ\", y=\"hwy\")) + geom_point(position=\"jitter\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96277926",
   "metadata": {},
   "source": [
    "Adding randomness seems like a strange way to improve your plot, but while it makes your graph less accurate at small scales, it makes your graph *more* revealing at large scales.\n",
    "\n",
    "Because this is such a useful operation, **letsplot** comes with a shorthand for `geom_point(position = \"jitter\")`: `geom_jitter()`.\n",
    "\n",
    "Of course, a more sophisticated way of dealing with overplotting is via a binscatter plot, which is available in the [**binsreg**](https://nppackages.github.io/binsreg/) package.\n",
    "\n",
    "To learn more about position adjustment, take a look at the [documentation](https://lets-plot.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90d09736",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1.  What is the problem with the following plot?\n",
    "    How could you improve it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bc38aef",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(mpg, aes(x=\"cty\", y=\"hwy\")) + geom_point())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57b6523f",
   "metadata": {},
   "source": [
    "2.  What, if anything, is the difference between the two plots?\n",
    "    Why?\n",
    "\n",
    "    ```python\n",
    "    (\n",
    "        ggplot(mpg, aes(x = \"displ\", y = \"hwy\")) +\n",
    "          geom_point()\n",
    "    )\n",
    "    (\n",
    "        ggplot(mpg, aes(x = \"displ\", y = \"hwy\")) +\n",
    "          geom_point(position = \"identity\")\n",
    "    )\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3138f75",
   "metadata": {},
   "source": [
    "3.  What parameters to `geom_jitter()` control the amount of jittering?\n",
    "\n",
    "4.  What's the default position adjustment for `geom_boxplot()`?\n",
    "    Create a visualisation of the `mpg` dataset that demonstrates it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "353de15d",
   "metadata": {},
   "source": [
    "### Co-ordinate Systems\n",
    "\n",
    "Note that, to create maps, **lets-plot** requires **geopandas**, and we strongly recommend you install **geopandas** using `conda install geopandas`, as it has a lot of complicated dependencies.\n",
    "\n",
    "Co-ordinate systems can certainly get complicated! They're very important in mathematics and physics but, more obviously, for maps. Most of the time, we're working in *cartesian* co-ordinates where everything is mapped out by x, y, and sometimes z (usually height or colour on continuous scales). But because the Earth is not always well-represented by cartesian co-ordinates (especially as you \"zoom out\"), you often need a different co-ordinate system to make visualisations of it. We don't want to spend too long on this, because it's a big topic, but what's important for you to know here is that **lets-plot** does have the capability work with map-based co-ordinate systems.\n",
    "\n",
    "The first element of plotting any map is retrieving the geographic information. Let's grab info on the UK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a96d3d89",
   "metadata": {},
   "outputs": [],
   "source": [
    "uk = geocode(\"state\").scope(\"United Kingdom\").get_boundaries(6)\n",
    "uk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b0add0f",
   "metadata": {},
   "source": [
    "Now let's pass this to `geom_map()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44102ffc",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot()\n",
    "    + geom_map(map=uk, fill=\"gray\", color=\"white\")\n",
    "    + coord_map(xlim=(-10, 6), ylim=(50, 59))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "003be0d6",
   "metadata": {},
   "source": [
    "We can spruce this up with some colours, and with some cities!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "487eda0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "names = [\"London\", \"Edinburgh\", \"Belfast\", \"Cardiff\"]\n",
    "states = [\"England\", \"Scotland\", \"Northern Ireland\", \"Wales\"]\n",
    "cities = geocode(names=names, states=states).ignore_not_found().get_centroids()\n",
    "cities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3cf5712",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot()\n",
    "    + geom_map(aes(fill=\"found name\"), map=uk, color=\"white\")\n",
    "    + coord_map(xlim=(-10, 6), ylim=(50, 59))\n",
    "    + geom_point(data=cities, size=5)\n",
    "    + theme(legend_position=\"none\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d302c31",
   "metadata": {},
   "source": [
    "## The layered grammar of graphics\n",
    "\n",
    "We can expand on the graphing template you learned already by adding position adjustments, stats, coordinate systems, and faceting:\n",
    "\n",
    "```python\n",
    "ggplot(data = <DATA>) + \n",
    "  <GEOM_FUNCTION>(\n",
    "     mapping = aes(<MAPPINGS>),\n",
    "     stat = <STAT>, \n",
    "     position = <POSITION>\n",
    "  ) +\n",
    "  <FACET_FUNCTION>\n",
    "```\n",
    "\n",
    "\n",
    "Our new template takes six parameters, the bracketed words that appear in the template.\n",
    "In practice, you rarely need to supply all seven parameters to make a graph because **lets-plot** will provide useful defaults for everything except the data, the mappings, and the geom function.\n",
    "\n",
    "The six parameters in the template compose the grammar of graphics, a formal system for building plots.\n",
    "The grammar of graphics is based on the insight that you can uniquely describe *any* plot as a combination of a dataset, a geom, a set of mappings, a stat, a position adjustment, a coordinate system, a faceting scheme, and a theme.\n",
    "\n",
    "To see how this works, consider how you could build a basic plot from scratch: you could start with a dataset and then transform it into the information that you want to display (with a stat).\n",
    "Next, you could choose a geometric object to represent each observation in the transformed data.\n",
    "You could then use the aesthetic properties of the geoms to represent variables in the data.\n",
    "You would map the values of each variable to the levels of an aesthetic.\n",
    "These steps are illustrated in the figure below.\n",
    "\n",
    "![](https://r4ds.hadley.nz/images/visualization-grammar.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0f7a7ff",
   "metadata": {},
   "source": [
    "At this point, you would have a complete graph, but you could further adjust the positions of the geoms within the coordinate system (a position adjustment) or split the graph into subplots (faceting).\n",
    "You could also extend the plot by adding one or more additional layers, where each additional layer uses a dataset, a geom, a set of mappings, a stat, and a position adjustment.\n",
    "\n",
    "You could use this method to create a lot of plots that you can imagine.\n",
    "\n",
    "## Summary\n",
    "\n",
    "In this chapter you learned about the layered grammar of graphics starting with aesthetics and geometries to build a simple plot, facets for splitting the plot into subsets, statistics for understanding how geoms are calculated, position adjustments for controlling the fine details of position when geoms might otherwise overlap, and coordinate systems which allow you to fundamentally change what `x` and `y` mean.\n",
    "\n",
    "The most useful further resource on **lets-plot** is the documentation, which you [can find here](https://lets-plot.org/)."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc"
  },
  "jupytext": {
   "cell_metadata_filter": "-all",
   "encoding": "# -*- coding: utf-8 -*-",
   "formats": "md:myst",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "toc-showtags": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}