getting-started/Using_Jupyter_notebooks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Jupyter notebooks in the GLAM Workbench"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The GLAM Workbench includes many [Jupyter notebooks](https://jupyter.org/). Jupyter lets you combine text, images, and live code within a single web page. So not only can you read about collections data, you can download it, analyse it, and visualise it &ndash; all within your browser!\n",
    "\n",
    "While the notebooks often include some fairly intimidating looking code, you don't need to understand the code to use them. As explained below, there's just a couple of basic conventions you need to keep in mind when running Jupyter notebooks. Once you've mastered these, you'll be able to use any of the tools or examples in this workbench.\n",
    "\n",
    "Of course, once you've developed a bit of confidence, you might want to start playing around with the code. That's how you learn. The GLAM Workbench isn't just a collection of tools, it's a starting point – from here you can explore, extend, and experiment!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running code in a notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the notebooks in the GLAM Workbench include snippets of real code. You can use this code to do things like download data, or create charts. The programming language used here is [Python](https://www.python.org/). It's popular in the data sciences and is generally pretty easy for humans to understand. \n",
    "\n",
    "The code in Jupyter notebooks is contained in cells, or boxes, on the page – you can identify code cells by the borders around them. \n",
    "\n",
    "To run code snippets:\n",
    "\n",
    "1. Click on the code cell (you'll see the cell becomes highlighted).\n",
    "2. Hit **Shift+Enter** (the code will run and you'll be moved on to the next cell).\n",
    "\n",
    "That's it – try it with the cell below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CLICK ON ME AND THEN HIT SHIFT+ENTER!\n",
    "\n",
    "# This makes the datetime module available to use\n",
    "import datetime\n",
    "\n",
    "# This creates a variable called 'date_now' and uses the datetime.date.today() function to set it to today's date.\n",
    "date_now = datetime.date.today()\n",
    "\n",
    "# This displays a nicely-formatted string containing the date\n",
    "print(f'Congratulations! You ran the code in this cell on {date_now}')\n",
    "\n",
    "# Hey! Have you noticed that lines starting with '#' are comments? They can help you understand what's going on in the code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p><b>Help! Nothing happens when I click!</b></p>\n",
    "\n",
    "\n",
    "If nothing happens when you click on a cell or hit **Shift+Enter** it's probably because you're viewing a static version of the notebook. See <a href=\"1-Introduction-and-table-of-contents.ipynb\">Introduction</a> for setting up a live version.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also run the code in a highlighted cell by clicking on the 'Run' icon in the toolbar, or by hitting **Control+Enter**. You'll notice that **Shift+Enter** runs the code and moves you on to the next cell, while **Control+Enter** leaves you where you are."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any variables or functions defined within a cell are made available to the rest of the notebook when you run it. In the cell above, we created a variable called `date_now`. Let's try using it in another cell. Just click on the cell below and hit **Shift+Enter** to extract the year component of `date_now`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run me to see how we can access the 'date_now' variable\n",
    "\n",
    "# This gets the year from the date stored in the date_now variable.\n",
    "date_now.year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p><b>Help! I get a weird 'Name Error' saying that 'date_now is not defined.</b></p>\n",
    "    <p>If you get a message about a NameError, make sure that you've run the first code cell (where we create the 'date_now' variable, before this one. Most Jupyter notebooks expect you to run cells in order, from top to bottom.</p>\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the notebooks in this workbench include many code cells defining functions and setting up variables for later use. Just run the cells in the order they appear to make sure that everything works as expected. If you get an error, check back to make sure all the relevant cells have been executed.\n",
    "\n",
    "You might have noticed that after a cell has been run a number appears in the square brackets &ndash; `[ ]:`. This helps you keep track of which cells have been executed. While the code in a cell is running you'll see an asterix &ndash; `[*]:`. This changes to a number once the code has completed.\n",
    "\n",
    "In some cases, cells will start processes that take a bit of time to complete. For example, harvesting series data from RecordSearch can take lots of time depending on how big the series is. You won't be able to run any more cells until the current one has finished. Just wait for the asterix to turn into a number and you'll be right to move on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding and editing cells to a notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many places throughout this workbench, you'll be asked to edit or add to the code. By doing that you can customise the code to your own research interests. Just click on any code cell to start editing. Try it in the cell below! \n",
    "\n",
    "1. Click on the cell to select it.\n",
    "2. You'll see the cursor turns into a normal text cursor. Just select `Tim` in the code below and type in your own name. Keep the quotes around your name.\n",
    "3. When you've finished editing, hit **Shift+Enter** to run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit this cell to add your name between the quotes, and then run it\n",
    "your_name = 'Tim' # <-- EDIT ME!\n",
    "\n",
    "print('Hi {}! Welcome to the OzGLAM workbench. 👋'.format(your_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p><b>Help! I get a weird 'SyntaxError' or 'Name Error'.</b></p>\n",
    "    <p>Make sure your name is enclosed in quotes (either single or double, as long as they match). Quotes indicate that you're working with a string or text value. Without them, Python will go looking for a variable with your name!</p>\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add a new cell, you just click the **+** icon in the toolbar at the top of the page. By default, the new cell will be expecting code, but if you want to add text (like this) just select 'Markdown' from the dropdown list of types in the toolbar. Here's some more information on [formatting markdown cells](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Working%20With%20Markdown%20Cells.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's try getting some data\n",
    "\n",
    "Once you know how to run and edit cells, you can start to do all sorts of interesting things – such as getting collection data from the National Museum of Australia!\n",
    "\n",
    "The NMA makes its collection data available [through an API](https://www.nma.gov.au/about/our-collection/our-apis) (Application Programming Interface). This means we can fire off queries and get results back in a structured form that we can process and analyse. To do this, we'll make use of [Requests](https://requests.kennethreitz.org/en/master/) – the go-to Python package for moving data around on the web.\n",
    "\n",
    "Let's try searching for a keyword in the NMA collections:\n",
    "\n",
    "1. Click on the cell below to select it.\n",
    "2. Look for the line starting with `keyword`, replace the value in quotes with anything you'd like to find in the NMA collection. Keep the quotes around the value.\n",
    "3. Hit **Shift+Enter** to run the cell and get some data.\n",
    "4. If your keyword can't be found, you'll see something like `data: []` – indicating that the list of results is empty. Just change the keyword value and try again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is how we import an external package or library into Python\n",
    "import requests\n",
    "\n",
    "keyword = 'stone' # <-- EDIT ME! (But leave the quotes)\n",
    "\n",
    "# This is the base url we use to send our search terms to the NMA API\n",
    "# The limit=1 tells the API we only want one record -- try changing it to see what happens!\n",
    "url = 'https://data.nma.gov.au/object?limit=1&text='\n",
    "\n",
    "# We combine our keyword with the base url and send our query off to the API\n",
    "response = requests.get(url + keyword)\n",
    "\n",
    "# We extract the results as JSON\n",
    "results = response.json()\n",
    "\n",
    "# Display the results\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data arrives back in a standard format known as [JSON](https://en.wikipedia.org/wiki/JSON) (JavaScript Object Notation). The main things to notice are that it contains labels and values, and these labels and values are arranged in some sort of hierarchy. If we understand the hierarchy, we can get back the value for any label – it's just a matter of following the path through the hierarchy until we get to the value we want. For example, here's how we get the value for `title`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS CELL to get the value of `title`\n",
    "results['data'][0]['title']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `[0]` says that we want the first record contained within the result data (although in this case there's only one anyway).\n",
    "\n",
    "You might also notice that `meta` includes a `total` value that tells us the total number of results matching our keyword. Here's how we get the to the `total` value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS CELL to get the number of search results\n",
    "results['meta']['results']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's make a dataset\n",
    "\n",
    "We made one request to the API and found out the number of items matching our keyword. By repeating this process multiple times with different keywords, we can start building up a picture of the NMA's holdings.\n",
    "\n",
    "Instead of defining a single `keyword` value, we'll create a list of `keywords`. Lists in Python are contained within square brackets. We'll loop through the list of keywords, getting the total number of results for each. Then we'll use [Pandas](https://pandas.pydata.org/), the all-purpose (and frighteningly powerful) data analysis package, to convert our data into a dataframe. Dataframes come with with all sorts of useful built-in methods for shaping and analysing data. \n",
    "\n",
    "1. Look for the line `keywords = ['cat', 'dog', 'kangaroo', 'koala']` in the cell below – this creates a list of keyword values.\n",
    "2. Edit the list – feel free to add, remove, or edit any of the keywords, but keep the quotes around each value. You can include as many keywords as you want.\n",
    "3. Run the cell to retrieve the data. Depending on how long your list is, it might take several seconds to complete. Pandas will display the results in a nicely-formatted table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# By convention pandas is assigned the shorthand 'pd' when we import it\n",
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "# The square brackets indicate that this is a list of values\n",
    "# Change or add values as you wish!\n",
    "keywords = ['cat', 'dog', 'kangaroo', 'koala'] # <-- EDIT ME!\n",
    "\n",
    "# This is an empty list to put our data in\n",
    "totals = []\n",
    "\n",
    "# We're going to loop through the keywords one at a time\n",
    "for keyword in keywords:\n",
    "    \n",
    "    # This is the same code we used above\n",
    "    # Now it's in a loop, so it gets repeated multiple times\n",
    "    response = requests.get(url + keyword)\n",
    "    data = response.json()\n",
    "    total = data['meta']['results']\n",
    "    \n",
    "    # Now we'll save the keyword and the total results to our list\n",
    "    totals.append({'keyword': keyword, 'total': total})\n",
    "    \n",
    "    # Anonymous access to the NMA API has usage limits -- here we put in a pause of 1 second to stay within the limits.\n",
    "    time.sleep(1)\n",
    "\n",
    "# Now we'll convert our raw data into a Pandas dataframe\n",
    "# Dataframes come with all sorts of useful methods for analysing/shaping data\n",
    "df = pd.DataFrame(totals)\n",
    "\n",
    "# Display the dataset\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset is tiny, so it's easy to see what's going on. If you have lots of data, Pandas can help you make sense of it. For example, we might want to find the keyword with the highest number of results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS CELL to find which row has the largest value\n",
    "df.loc[df['total'].idxmax()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or perhaps you'd like to know the total number of search results across all keywords. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS CELL to get the sum of all values\n",
    "df['total'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are just a couple of examples of how Pandas helps you work with tabular data. There are many more throughout the GLAM workbench!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's try visualising our dataset\n",
    "\n",
    "We've displayed our data as a table, but a chart would be easier to interpret at a glance. There are a number of charting and data visualisation packages available for Python, here we'll be using [Altair](https://altair-viz.github.io/). You just feed Altair a dataframe, and tell it the columns to display on each axis. Let's start with a simple bar chart that shows the keywords along the `x` axis, and the number of search results on the `y` axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import altair as alt\n",
    "\n",
    "# If you're using Jupyter Lab rather than Jupyter Notebook,\n",
    "# change 'notebook' to 'default' in the line below\n",
    "alt.renderers.enable('notebook')\n",
    "\n",
    "alt.Chart(df).mark_bar().encode(\n",
    "    x='keyword:N',\n",
    "    y='total:Q'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p><b>Help! I get a weird error saying 'require is not defined', or a message saying something about 'Vegalite'.</b></p>\n",
    "    <p>If you're using Altair in Jupyter Lab rather than Jupyter Notebook, you need to make an adjustment to the code above. Just change 'notebook' to 'default' in the line starting 'alt.renderers.enable' and run the cell again.</p>\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altair is easy to customise. Here's a few things you could try:\n",
    "\n",
    "* Switch the `x` and `y` values in the code above and see what happens.\n",
    "* Change `mark_bar` to `mark_line`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We'd better save our dataset for later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many of the notebooks in the GLAM Workbench help you harvest data from GLAM collections, just as we did above. Once you've created a new dataset, you'll probably want to save it. Pandas makes it easy to save your dataframe as a [CSV (Comma Separated Values)](https://en.wikipedia.org/wiki/Comma-separated_values) file. CSV files are simple text files that can be opened by any spreadsheet program. They're widely used for storing and sharing datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS CELL to save your dataset as a CSV file\n",
    "\n",
    "# This is the filename we'll use for the CSV file, edit if you want\n",
    "csv_file = 'my_nma_dataset.csv'  # <-- EDIT ME if you'd prefer a different filename\n",
    "\n",
    "# Save the dataframe as a CSV file\n",
    "df.to_csv(csv_file, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've created your CSV file you can download it from Jupyter to your local computer. The cell below will create a handy link to the file. If you're using the classic Jupyter Notebook, the link will download the CSV file immediately. If you're using Jupyter Lab, the link will open the CSV file. To download, right click on it in the file browser and choose 'Download'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS CELL to create a link to the CSV file\n",
    "from IPython.display import display, FileLink\n",
    "\n",
    "# Display a link to the CSV file\n",
    "display(FileLink(csv_file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What have you learnt?\n",
    "\n",
    "This notebook has introduced you to the basics of Jupyter. You've learnt how to run and edit cells, but you've also harvested some data from an API, created a dataset, visualised the dataset, and saved it as a CSV file. As you work through the GLAM Workbench you'll find yourself repeating this sort of pattern – getting, analysing, and saving data. You've also met some important tools like Requests, Pandas, and Altair. Once again you'll find them popping up all over the place. The examples used in this notebook might have been pretty simple, but they provide a good introduction to what the GLAM Workbench is all about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "Created by [Tim Sherrratt](https://timsherratt.org) ([@wragge](https://twitter.com/wragge)) as part of the [GLAM workbench](https://glam-workbench.github.io/).\n",
    "\n",
    "If you think this project is worthwhile you can [support it on Patreon](https://www.patreon.com/timsherratt)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}