static_files/presentations/03_Relational_Database_and_data_wrangling.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"colab":{"provenance":[],"toc_visible":true}},"nbformat_minor":0,"nbformat":4,"cells":[{"cell_type":"markdown","source":["**Lab 3 – Relational Database and data wrangling**"],"metadata":{"id":"1QiCFLer1FIe"}},{"cell_type":"markdown","source":["<table align=\"left\">\n","  <td>\n","    <a href=\"https://colab.research.google.com/github/phonchi/nsysu-math608/blob/master/static_files/presentations/03_Relational_Database_and_data_wrangling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n","  </td>\n","  <td>\n","    <a target=\"_blank\" href=\"https://kaggle.com/kernels/welcome?src=https://github.com/phonchi/nsysu-math608/blob/master/static_files/presentations/03_Relational_Database_and_data_wrangling.ipynb\"><img src=\"https://kaggle.com/static/images/open-in-kaggle.svg\" /></a>\n","  </td>\n","</table>"],"metadata":{"id":"9J5g6PDs1FIk"}},{"cell_type":"code","source":["# For SQL\n","from google.cloud import bigquery\n","\n","import numpy as np\n","import pandas as pd\n","\n","import matplotlib as mpl\n","from matplotlib import pyplot as plt\n","%matplotlib inline"],"metadata":{"id":"5bV_HvPiH-9i","execution":{"iopub.status.busy":"2022-04-23T03:33:52.948105Z","iopub.execute_input":"2022-04-23T03:33:52.948724Z","iopub.status.idle":"2022-04-23T03:33:52.976140Z","shell.execute_reply.started":"2022-04-23T03:33:52.948606Z","shell.execute_reply":"2022-04-23T03:33:52.975047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Queringing data with BigQuery"],"metadata":{"id":"6sb3RBiSo4Vf"}},{"cell_type":"markdown","source":["Structured Query Language, or SQL, is the programming language used with databases, and it is an important skill for any data scientist. In this example, you'll build your SQL skills using BigQuery, a web service work as database management system that lets you apply SQL to huge datasets."],"metadata":{"id":"qvr5MUX8pFwO"}},{"cell_type":"markdown","source":["### Preliminaries for google colab (optional)"],"metadata":{"id":"bk33xCJGcA-W"}},{"cell_type":"markdown","source":["We want to start exploring the Google BiqQuery [public datasets](https://cloud.google.com/bigquery/public-data/). Let's start by walking through the required setup steps, and then we can load and explore some data.\n","\n","If you are using colab. Follow [this quickstart guide](https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries), which will explain how to:\n","1. Create a [Cloud Platform project](https://console.cloud.google.com/cloud-resource-manager) if you don't have one already.\n","2. [Enable billing](https://support.google.com/cloud/answer/6293499#enable-billing) for the project (If you apply the free trial, you already satisfy this condition.)\n","3. [Enable the BigQuery API](https://console.cloud.google.com/flows/enableapi?apiid=bigquery)\n","4. [Enabling the Service account](https://cloud.google.com/docs/authentication/getting-started)\n","\n","Now we need to authenticate to gain access to the BigQuery API. We will create a client, specifying the service account key file (replace 'lunar-pact-378812-7a28b789bde2.json' with your key file)."],"metadata":{"id":"OvNovWe2Hobw"}},{"cell_type":"code","source":["# @title Setup\n","from google.oauth2 import service_account\n","\n","# TODO(developer): Set key_path to the path to the service account key file.\n","\n","key_path = \"lunar-pact-378812-7a28b789bde2.json\"\n","\n","credentials = service_account.Credentials.from_service_account_file(\n","    key_path\n",")"],"metadata":{"id":"8MUif75uqqXE"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# @title Alternative Setup\n","#from google.colab import auth\n","#from google.cloud import bigquery\n","#from google.colab import data_table\n","\n","#project = 'lunar-pact-378812' # Project ID inserted based on the query results selected to explore\n","#location = 'US' # Location inserted based on the query results selected to explore\n","#client = bigquery.Client(project=project, location=location)\n","#data_table.enable_dataframe_formatter()\n","#auth.authenticate_user()"],"metadata":{"id":"CLZXPyBqcysR"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Now that we're authenticated, we need to load the BigQuery package, and the `google.colab.data_table` package that can be used to display large pandas dataframes as an interactive data. Loading `data_table` is optional, but it will be useful for working with data in pandas."],"metadata":{"id":"CFpSwmOJIDS_"}},{"cell_type":"code","source":["#%load_ext google.cloud.bigquery\n","#%load_ext google.colab.data_table"],"metadata":{"id":"BnVn82RDGyQu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["client = bigquery.Client(credentials=credentials, project=credentials.project_id,)"],"metadata":{"id":"vcmdamwKG3Et"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Create the reference\n"],"metadata":{"id":"1UK6R1kBIaPC"}},{"cell_type":"markdown","source":["You can also work with Kaggle, which provide bigquery integration that you do not need to setup a google account. **Each Kaggle user can scan 5TB every 30 days for free.  Once you hit that limit, you'll have to wait for it to reset.** See https://www.kaggle.com/product-feedback/48573 for more details.\n"],"metadata":{"id":"ssJU-WK9MSSE"}},{"cell_type":"markdown","source":["The first step in the workflow is to create a [`Client`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client) object.  As you'll soon see, this `Client` object will play a central role in retrieving information from BigQuery datasets."],"metadata":{"id":"YRCSulcqCL_o"}},{"cell_type":"code","source":["# Uncomment below and create a \"Client\" object if you are using Kaggle\n","# client = bigquery.Client()"],"metadata":{"id":"NLmMfMf4CO7t","execution":{"iopub.status.busy":"2022-04-23T03:34:20.953692Z","iopub.execute_input":"2022-04-23T03:34:20.953969Z","iopub.status.idle":"2022-04-23T03:34:20.959198Z","shell.execute_reply.started":"2022-04-23T03:34:20.953941Z","shell.execute_reply":"2022-04-23T03:34:20.958252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["We'll work with a dataset of posts on Hacker News, a website focusing on computer science and cybersecurity news. In BigQuery, each dataset is contained in a corresponding project. In this case, our `hacker_news` dataset is contained in the `bigquery-public-data project`. \n","\n","To access the dataset, We begin by constructing a reference to the dataset with the [`dataset()`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.dataset.Dataset) method. Next, we use the [`get_dataset()`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_get_dataset) method, along with the reference we just constructed, to fetch the dataset.\n","\n","[See the full list of public datasets](https://console.cloud.google.com/marketplace/browse?filter=solution-type:dataset) or the [kaggle bigquery dataset](https://www.kaggle.com/datasets?search=bigquery) if you want to explore others."],"metadata":{"id":"OgWXWB1ICXLi"}},{"cell_type":"code","source":["# Construct a reference to the \"hacker_news\" dataset\n","dataset_ref = client.dataset(\"hacker_news\", project=\"bigquery-public-data\")\n","\n","# API request - fetch the dataset\n","dataset = client.get_dataset(dataset_ref)"],"metadata":{"id":"oC7ldHp4CRf-","execution":{"iopub.status.busy":"2022-04-23T03:34:22.839652Z","iopub.execute_input":"2022-04-23T03:34:22.840484Z","iopub.status.idle":"2022-04-23T03:34:23.269423Z","shell.execute_reply.started":"2022-04-23T03:34:22.840431Z","shell.execute_reply":"2022-04-23T03:34:23.268733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Every dataset is just a collection of tables. You can think of a dataset as a spreadsheet file containing multiple tables, all composed of rows and columns.We use the [`list_tables()`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_list_tables) method to list the tables in the dataset."],"metadata":{"id":"M8Xms4W_I1QQ"}},{"cell_type":"code","source":["# List all the tables in the \"hacker_news\" dataset\n","tables = list(client.list_tables(dataset))\n","\n","# Print names of all tables in the dataset (there are four!)\n","for table in tables:  \n","    print(table.table_id)"],"metadata":{"id":"M2gOPT8jCvCL","execution":{"iopub.status.busy":"2022-04-23T03:34:25.195022Z","iopub.execute_input":"2022-04-23T03:34:25.195282Z","iopub.status.idle":"2022-04-23T03:34:25.507541Z","shell.execute_reply.started":"2022-04-23T03:34:25.195254Z","shell.execute_reply":"2022-04-23T03:34:25.506966Z"},"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"outputId":"016fbae4-b0ed-4d7e-e31f-a0cbf7d72998","executionInfo":{"status":"ok","timestamp":1677904603647,"user_tz":-480,"elapsed":1737,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["comments\n","full\n","full_201510\n","stories\n"]}]},{"cell_type":"markdown","source":["Similar to how we fetched a dataset, we can fetch a table. In the code cell below, we fetch the `full` table in the hacker_news dataset"],"metadata":{"id":"EOCsUwyEJA0K"}},{"cell_type":"code","source":["# Construct a reference to the \"full\" table\n","table_ref = dataset_ref.table(\"full\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)"],"metadata":{"id":"Q6JCjVCGI6ev","execution":{"iopub.status.busy":"2022-04-23T03:34:26.126551Z","iopub.execute_input":"2022-04-23T03:34:26.127256Z","iopub.status.idle":"2022-04-23T03:34:26.318084Z","shell.execute_reply.started":"2022-04-23T03:34:26.127219Z","shell.execute_reply":"2022-04-23T03:34:26.317267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["In the next section, you'll explore the contents of this table in more detail.  For now, take the time to use the image below to consolidate what you've learned so far.\n","\n","![first_commands](https://i.imgur.com/biYqbUB.png)"],"metadata":{"id":"EQ_-OP8cJtqo"}},{"cell_type":"markdown","source":["### Table schema\n","\n","The structure of a table is called its **schema**.  **We need to understand a table's schema to effectively pull out the data we want.** \n","\n","In this example, we'll investigate the `full` table that we fetched above."],"metadata":{"id":"RK2iKPr0J295"}},{"cell_type":"code","source":["# Print information on all the columns in the \"full\" table in the \"hacker_news\" dataset\n","table.schema"],"metadata":{"id":"oAcsp296JMPg","execution":{"iopub.status.busy":"2022-04-23T03:34:27.440643Z","iopub.execute_input":"2022-04-23T03:34:27.440936Z","iopub.status.idle":"2022-04-23T03:34:27.448446Z","shell.execute_reply.started":"2022-04-23T03:34:27.440908Z","shell.execute_reply":"2022-04-23T03:34:27.447878Z"},"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"outputId":"f9434633-ae45-4753-8361-d3dfee5f2d17","executionInfo":{"status":"ok","timestamp":1677904675853,"user_tz":-480,"elapsed":266,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[SchemaField('title', 'STRING', 'NULLABLE', None, 'Story title', (), None),\n"," SchemaField('url', 'STRING', 'NULLABLE', None, 'Story url', (), None),\n"," SchemaField('text', 'STRING', 'NULLABLE', None, 'Story or comment text', (), None),\n"," SchemaField('dead', 'BOOLEAN', 'NULLABLE', None, 'Is dead?', (), None),\n"," SchemaField('by', 'STRING', 'NULLABLE', None, \"The username of the item's author.\", (), None),\n"," SchemaField('score', 'INTEGER', 'NULLABLE', None, 'Story score', (), None),\n"," SchemaField('time', 'INTEGER', 'NULLABLE', None, 'Unix time', (), None),\n"," SchemaField('timestamp', 'TIMESTAMP', 'NULLABLE', None, 'Timestamp for the unix time', (), None),\n"," SchemaField('type', 'STRING', 'NULLABLE', None, 'Type of details (comment, comment_ranking, poll, story, job, pollopt)', (), None),\n"," SchemaField('id', 'INTEGER', 'NULLABLE', None, \"The item's unique id.\", (), None),\n"," SchemaField('parent', 'INTEGER', 'NULLABLE', None, 'Parent comment ID', (), None),\n"," SchemaField('descendants', 'INTEGER', 'NULLABLE', None, 'Number of story or poll descendants', (), None),\n"," SchemaField('ranking', 'INTEGER', 'NULLABLE', None, 'Comment ranking', (), None),\n"," SchemaField('deleted', 'BOOLEAN', 'NULLABLE', None, 'Is deleted?', (), None)]"]},"metadata":{},"execution_count":7}]},{"cell_type":"markdown","source":["Each [`SchemaField`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.schema.SchemaField) tells us about a specific column (which we also refer to as a **field**). In order, the information is:\n","\n","* The **name** of the column\n","* The **field type** (or datatype) in the column\n","* The **mode** of the column (`'NULLABLE'` means that a column allows NULL values, and is the default)\n","* A **description** of the data in that column\n","\n","For instance, the field has the SchemaField:\n","\n","`SchemaField('by', 'string', 'NULLABLE', \"The username of the item's author.\",())`\n","\n","This tells us:\n","- the field (or column) is called `by`,\n","- the data in this field is strings, \n","- NULL values are allowed, and\n","- it contains the usernames corresponding to each item's author.\n","\n","We can use the [`list_rows()`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_list_rows) method to check just the first five lines of of the `full` table to make sure this is right.  This returns a BigQuery [`RowIterator`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.table.RowIterator) object that can quickly be converted to a pandas DataFrame with the [`to_dataframe()`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.table.RowIterator#google_cloud_bigquery_table_RowIterator_to_dataframe) method."],"metadata":{"id":"Pm6u6FfGKA8v"}},{"cell_type":"code","source":["# Preview the first five lines of the \"full\" table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"id":"0eFyTIvFJ7ML","execution":{"iopub.status.busy":"2022-04-23T03:34:28.459739Z","iopub.execute_input":"2022-04-23T03:34:28.460370Z","iopub.status.idle":"2022-04-23T03:34:29.255110Z","shell.execute_reply.started":"2022-04-23T03:34:28.460329Z","shell.execute_reply":"2022-04-23T03:34:29.254606Z"},"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"13188c22-367f-405e-9cd3-022da9446d64","executionInfo":{"status":"ok","timestamp":1677904727459,"user_tz":-480,"elapsed":741,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["  title   url                                               text  dead  \\\n","0  None  None  I would rather just have wired earbuds, period...  <NA>   \n","1  None  None                                               DNS?  <NA>   \n","2  None  None  These benchmarks seem pretty good.  Filterable...  <NA>   \n","3  None  None  Oh really?<p>* Excel alone uses 86.1MB of priv...  <NA>   \n","4  None  None  These systems are useless.  Of the many flaws:...  <NA>   \n","\n","           by  score        time                 timestamp     type        id  \\\n","0       zeveb   <NA>  1591717736 2020-06-09 15:48:56+00:00  comment  23467666   \n","1         nly   <NA>  1572810465 2019-11-03 19:47:45+00:00  comment  21436112   \n","2      mrkeen   <NA>  1591717727 2020-06-09 15:48:47+00:00  comment  23467665   \n","3  oceanswave   <NA>  1462987532 2016-05-11 17:25:32+00:00  comment  11677248   \n","4      nyxxie   <NA>  1572810473 2019-11-03 19:47:53+00:00  comment  21436113   \n","\n","     parent  descendants  ranking  deleted  \n","0  23456782         <NA>     <NA>     <NA>  \n","1  21435130         <NA>     <NA>     <NA>  \n","2  23467426         <NA>     <NA>     <NA>  \n","3  11676886         <NA>     <NA>     <NA>  \n","4  21435025         <NA>     <NA>     <NA>  "],"text/html":["\n","  <div id=\"df-2667d889-c6ff-454a-a160-971fbbaf38d0\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>title</th>\n","      <th>url</th>\n","      <th>text</th>\n","      <th>dead</th>\n","      <th>by</th>\n","      <th>score</th>\n","      <th>time</th>\n","      <th>timestamp</th>\n","      <th>type</th>\n","      <th>id</th>\n","      <th>parent</th>\n","      <th>descendants</th>\n","      <th>ranking</th>\n","      <th>deleted</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>I would rather just have wired earbuds, period...</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>zeveb</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>1591717736</td>\n","      <td>2020-06-09 15:48:56+00:00</td>\n","      <td>comment</td>\n","      <td>23467666</td>\n","      <td>23456782</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>DNS?</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>nly</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>1572810465</td>\n","      <td>2019-11-03 19:47:45+00:00</td>\n","      <td>comment</td>\n","      <td>21436112</td>\n","      <td>21435130</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>These benchmarks seem pretty good.  Filterable...</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>mrkeen</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>1591717727</td>\n","      <td>2020-06-09 15:48:47+00:00</td>\n","      <td>comment</td>\n","      <td>23467665</td>\n","      <td>23467426</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>Oh really?&lt;p&gt;* Excel alone uses 86.1MB of priv...</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>oceanswave</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>1462987532</td>\n","      <td>2016-05-11 17:25:32+00:00</td>\n","      <td>comment</td>\n","      <td>11677248</td>\n","      <td>11676886</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>These systems are useless.  Of the many flaws:...</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>nyxxie</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>1572810473</td>\n","      <td>2019-11-03 19:47:53+00:00</td>\n","      <td>comment</td>\n","      <td>21436113</td>\n","      <td>21435025</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2667d889-c6ff-454a-a160-971fbbaf38d0')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-2667d889-c6ff-454a-a160-971fbbaf38d0 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-2667d889-c6ff-454a-a160-971fbbaf38d0');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":8}]},{"cell_type":"markdown","source":["The `list_rows()` method will also let us look at just the information in a specific column. If we want to see the first five entries in the `by` column, for example, we can do that!"],"metadata":{"id":"tfGNKgW3KxWW"}},{"cell_type":"code","source":["# Preview the first five entries in the \"by\" column of the \"full\" table\n","client.list_rows(table, selected_fields=table.schema[4:5], max_results=5).to_dataframe()"],"metadata":{"id":"ghYSN97rKc6f","execution":{"iopub.status.busy":"2022-04-23T03:34:30.428333Z","iopub.execute_input":"2022-04-23T03:34:30.428784Z","iopub.status.idle":"2022-04-23T03:34:30.823825Z","shell.execute_reply.started":"2022-04-23T03:34:30.428728Z","shell.execute_reply":"2022-04-23T03:34:30.823228Z"},"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"a0c18034-a3a0-4229-f190-8824fe070c4c","executionInfo":{"status":"ok","timestamp":1677904734510,"user_tz":-480,"elapsed":295,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["           by\n","0       zeveb\n","1         nly\n","2      mrkeen\n","3  oceanswave\n","4      nyxxie"],"text/html":["\n","  <div id=\"df-f9ddd70a-f873-4167-a8d2-ff8239b82e02\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>by</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>zeveb</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>nly</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>mrkeen</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>oceanswave</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>nyxxie</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f9ddd70a-f873-4167-a8d2-ff8239b82e02')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-f9ddd70a-f873-4167-a8d2-ff8239b82e02 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-f9ddd70a-f873-4167-a8d2-ff8239b82e02');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":9}]},{"cell_type":"markdown","source":["### Select, From & Where"],"metadata":{"id":"83w8rxkkM5TE"}},{"cell_type":"markdown","source":["Now that you know how to access and examine a dataset, you're ready to write your first SQL query!  As you'll soon see, **SQL queries will help you sort through a massive dataset, to retrieve only the information that you need.** We'll begin by using the keywords **SELECT**, **FROM**, and **WHERE** to get data from specific columns based on conditions you specify. "],"metadata":{"id":"wrRPIX4PMsKK"}},{"cell_type":"markdown","source":["We'll use an [OpenAQ](https://openaq.org) dataset about air quality. First, we'll set up everything we need to run queries and take a quick peek at what tables are in our database."],"metadata":{"id":"yZwYMG_FNm5S"}},{"cell_type":"code","source":["# Construct a reference to the \"openaq\" dataset\n","dataset_ref = client.dataset(\"openaq\", project=\"bigquery-public-data\")\n","\n","# API request - fetch the dataset\n","dataset = client.get_dataset(dataset_ref)\n","\n","# List all the tables in the \"openaq\" dataset\n","tables = list(client.list_tables(dataset))\n","\n","# Print names of all tables in the dataset (there's only one!)\n","for table in tables:  \n","    print(table.table_id)"],"metadata":{"id":"72Zhkk0wK3oN","execution":{"iopub.status.busy":"2022-04-23T03:34:34.103065Z","iopub.execute_input":"2022-04-23T03:34:34.103476Z","iopub.status.idle":"2022-04-23T03:34:34.569656Z","shell.execute_reply.started":"2022-04-23T03:34:34.103437Z","shell.execute_reply":"2022-04-23T03:34:34.568998Z"},"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"outputId":"277fe2a4-ba82-47f9-e020-307c5072e1dd","executionInfo":{"status":"ok","timestamp":1677904760800,"user_tz":-480,"elapsed":751,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["global_air_quality\n"]}]},{"cell_type":"markdown","source":["The dataset contains only one table, called `global_air_quality`.  We'll fetch the table and take a peek at the first few rows to see what sort of data it contains."],"metadata":{"id":"x1czVLqAOEcs"}},{"cell_type":"code","source":["# Construct a reference to the \"global_air_quality\" table\n","table_ref = dataset_ref.table(\"global_air_quality\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)\n","\n","# Preview the first five lines of the \"global_air_quality\" table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"id":"kYAuivzjOAM9","execution":{"iopub.status.busy":"2022-04-23T03:34:36.196047Z","iopub.execute_input":"2022-04-23T03:34:36.196746Z","iopub.status.idle":"2022-04-23T03:34:36.908751Z","shell.execute_reply.started":"2022-04-23T03:34:36.196712Z","shell.execute_reply":"2022-04-23T03:34:36.907911Z"},"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"b8d62a1c-2e79-4152-bdd7-953848d61089","executionInfo":{"status":"ok","timestamp":1677904763223,"user_tz":-480,"elapsed":893,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                  location      city country pollutant    value  \\\n","0   Borówiec, ul. Drapałka  Borówiec      PL        bc  0.85217   \n","1    Kraków, ul. Bulwarowa    Kraków      PL        bc  0.91284   \n","2          Płock, ul. Reja     Płock      PL        bc  1.41000   \n","3  Elbląg, ul. Bażyńskiego    Elbląg      PL        bc  0.33607   \n","4  Piastów, ul. Pułaskiego   Piastów      PL        bc  0.51000   \n","\n","                  timestamp   unit source_name  latitude  longitude  \\\n","0 2022-04-28 07:00:00+00:00  µg/m³        GIOS       1.0  52.276794   \n","1 2022-04-27 23:00:00+00:00  µg/m³        GIOS       1.0  50.069308   \n","2 2022-03-30 04:00:00+00:00  µg/m³        GIOS       1.0  52.550938   \n","3 2022-05-03 13:00:00+00:00  µg/m³        GIOS       1.0  54.167847   \n","4 2022-05-11 05:00:00+00:00  µg/m³        GIOS       1.0  52.191728   \n","\n","   averaged_over_in_hours       location_geom  \n","0               17.074114  POINT(52.276794 1)  \n","1               20.053492  POINT(50.069308 1)  \n","2               19.709791  POINT(52.550938 1)  \n","3               19.410942  POINT(54.167847 1)  \n","4               20.837489  POINT(52.191728 1)  "],"text/html":["\n","  <div id=\"df-cc02e0ea-9008-43ad-9b94-c0a9e8209669\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>location</th>\n","      <th>city</th>\n","      <th>country</th>\n","      <th>pollutant</th>\n","      <th>value</th>\n","      <th>timestamp</th>\n","      <th>unit</th>\n","      <th>source_name</th>\n","      <th>latitude</th>\n","      <th>longitude</th>\n","      <th>averaged_over_in_hours</th>\n","      <th>location_geom</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Borówiec, ul. Drapałka</td>\n","      <td>Borówiec</td>\n","      <td>PL</td>\n","      <td>bc</td>\n","      <td>0.85217</td>\n","      <td>2022-04-28 07:00:00+00:00</td>\n","      <td>µg/m³</td>\n","      <td>GIOS</td>\n","      <td>1.0</td>\n","      <td>52.276794</td>\n","      <td>17.074114</td>\n","      <td>POINT(52.276794 1)</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Kraków, ul. Bulwarowa</td>\n","      <td>Kraków</td>\n","      <td>PL</td>\n","      <td>bc</td>\n","      <td>0.91284</td>\n","      <td>2022-04-27 23:00:00+00:00</td>\n","      <td>µg/m³</td>\n","      <td>GIOS</td>\n","      <td>1.0</td>\n","      <td>50.069308</td>\n","      <td>20.053492</td>\n","      <td>POINT(50.069308 1)</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Płock, ul. Reja</td>\n","      <td>Płock</td>\n","      <td>PL</td>\n","      <td>bc</td>\n","      <td>1.41000</td>\n","      <td>2022-03-30 04:00:00+00:00</td>\n","      <td>µg/m³</td>\n","      <td>GIOS</td>\n","      <td>1.0</td>\n","      <td>52.550938</td>\n","      <td>19.709791</td>\n","      <td>POINT(52.550938 1)</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Elbląg, ul. Bażyńskiego</td>\n","      <td>Elbląg</td>\n","      <td>PL</td>\n","      <td>bc</td>\n","      <td>0.33607</td>\n","      <td>2022-05-03 13:00:00+00:00</td>\n","      <td>µg/m³</td>\n","      <td>GIOS</td>\n","      <td>1.0</td>\n","      <td>54.167847</td>\n","      <td>19.410942</td>\n","      <td>POINT(54.167847 1)</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Piastów, ul. Pułaskiego</td>\n","      <td>Piastów</td>\n","      <td>PL</td>\n","      <td>bc</td>\n","      <td>0.51000</td>\n","      <td>2022-05-11 05:00:00+00:00</td>\n","      <td>µg/m³</td>\n","      <td>GIOS</td>\n","      <td>1.0</td>\n","      <td>52.191728</td>\n","      <td>20.837489</td>\n","      <td>POINT(52.191728 1)</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cc02e0ea-9008-43ad-9b94-c0a9e8209669')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-cc02e0ea-9008-43ad-9b94-c0a9e8209669 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-cc02e0ea-9008-43ad-9b94-c0a9e8209669');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":11}]},{"cell_type":"markdown","source":["let's put together a query. Say we want to select all the values from the `city` column that are in rows where the `country` column is `'US'` (for \"United States\")."],"metadata":{"id":"1g2H5wNrOJrB"}},{"cell_type":"code","source":["# Query to select all the items from the \"city\" column where the \"country\" column is 'US'\n","# SQL is almost completely case and indentation insensitive. The capitalization and\n","# indentation style here is preferred style.\n","query = \"\"\"\n","        SELECT city\n","        FROM `bigquery-public-data.openaq.global_air_quality`\n","        WHERE country = 'US'\n","        \"\"\""],"metadata":{"id":"HNVL36G6OI9z","execution":{"iopub.status.busy":"2022-04-23T03:34:37.669408Z","iopub.execute_input":"2022-04-23T03:34:37.669919Z","iopub.status.idle":"2022-04-23T03:34:37.674122Z","shell.execute_reply.started":"2022-04-23T03:34:37.669880Z","shell.execute_reply":"2022-04-23T03:34:37.673374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Notice also that SQL statements requires single quotes for its strings inside python string (we use triple quotation mark here). We begin by setting up the query with the [`query()`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_query) method."],"metadata":{"id":"lrUVBJqmOfgD"}},{"cell_type":"code","source":["# Set up the query\n","query_job = client.query(query)\n","\n","# API request - run the query, and return a pandas DataFrame\n","us_cities = query_job.to_dataframe()"],"metadata":{"id":"3P7iBjrQbxC5","execution":{"iopub.status.busy":"2022-04-23T03:35:55.758822Z","iopub.execute_input":"2022-04-23T03:35:55.759311Z","iopub.status.idle":"2022-04-23T03:35:59.638985Z","shell.execute_reply.started":"2022-04-23T03:35:55.759280Z","shell.execute_reply":"2022-04-23T03:35:59.638236Z"},"trusted":true,"executionInfo":{"status":"ok","timestamp":1677904814527,"user_tz":-480,"elapsed":6364,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":13,"outputs":[]},{"cell_type":"markdown","source":["Now we've got a pandas DataFrame called `us_cities`, which we can use like any other DataFrame."],"metadata":{"id":"28V2b3IMderS"}},{"cell_type":"code","source":["# What five cities have the most measurements?\n","us_cities.city.value_counts().head()"],"metadata":{"id":"GBNcJ4NtdeAm","execution":{"iopub.status.busy":"2022-04-23T03:36:03.918735Z","iopub.execute_input":"2022-04-23T03:36:03.919522Z","iopub.status.idle":"2022-04-23T03:36:03.937814Z","shell.execute_reply.started":"2022-04-23T03:36:03.919480Z","shell.execute_reply":"2022-04-23T03:36:03.936847Z"},"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"outputId":"d3fdf5bd-ac14-4e0b-9ec9-358088ac0c20","executionInfo":{"status":"ok","timestamp":1677904819538,"user_tz":-480,"elapsed":461,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":14,"outputs":[{"output_type":"execute_result","data":{"text/plain":["Phoenix-Mesa-Scottsdale                     39414\n","Los Angeles-Long Beach-Santa Ana            27479\n","Riverside-San Bernardino-Ontario            26887\n","New York-Northern New Jersey-Long Island    25417\n","San Francisco-Oakland-Fremont               22710\n","Name: city, dtype: int64"]},"metadata":{},"execution_count":14}]},{"cell_type":"markdown","source":["If you want multiple columns, you can select them with a comma between the names:"],"metadata":{"id":"yDRyu1cR4fLU"}},{"cell_type":"code","source":["query = \"\"\"\n","        SELECT city, country\n","        FROM `bigquery-public-data.openaq.global_air_quality`\n","        WHERE country = 'US'\n","        \"\"\""],"metadata":{"execution":{"iopub.status.busy":"2022-04-23T03:40:51.487848Z","iopub.execute_input":"2022-04-23T03:40:51.488176Z","iopub.status.idle":"2022-04-23T03:40:51.492074Z","shell.execute_reply.started":"2022-04-23T03:40:51.488143Z","shell.execute_reply":"2022-04-23T03:40:51.491286Z"},"trusted":true,"id":"Ixf_qpmo4fLU","executionInfo":{"status":"ok","timestamp":1677904837686,"user_tz":-480,"elapsed":289,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":15,"outputs":[]},{"cell_type":"markdown","source":["You can select all columns with a `*` like this:"],"metadata":{"id":"3BNBGKPC4fLU"}},{"cell_type":"code","source":["query = \"\"\"\n","        SELECT *\n","        FROM `bigquery-public-data.openaq.global_air_quality`\n","        WHERE country = 'US'\n","        \"\"\""],"metadata":{"id":"LQ29UlAi4fLU"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Querying big dataset"],"metadata":{"id":"w2hxfX8Q4fLV"}},{"cell_type":"markdown","source":["You can estimate the size of any query before running it. Here is an example using the Hacker News dataset. To see how much data a query will scan, we create a [`QueryJobConfig`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.job.QueryJobConfig) object and set the `dry_run` parameter to `True`."],"metadata":{"id":"u0R_qYE64fLV"}},{"cell_type":"code","source":["# Query to get the score column from every row where the type column has value \"job\"\n","query = \"\"\"\n","        SELECT score, title\n","        FROM `bigquery-public-data.hacker_news.full`\n","        WHERE type = \"job\" \n","        \"\"\"\n","\n","# Create a QueryJobConfig object to estimate size of query without running it\n","dry_run_config = bigquery.QueryJobConfig(dry_run=True)\n","\n","# API request - dry run query to estimate costs\n","dry_run_query_job = client.query(query, job_config=dry_run_config)\n","\n","print(\"This query will process {} bytes.\".format(dry_run_query_job.total_bytes_processed))"],"metadata":{"execution":{"iopub.status.busy":"2022-04-23T03:43:43.636879Z","iopub.execute_input":"2022-04-23T03:43:43.637619Z","iopub.status.idle":"2022-04-23T03:43:44.083153Z","shell.execute_reply.started":"2022-04-23T03:43:43.637561Z","shell.execute_reply":"2022-04-23T03:43:44.082302Z"},"trusted":true,"id":"wI2YIn_74fLV","colab":{"base_uri":"https://localhost:8080/"},"outputId":"7c54439f-ddc1-4e53-fe2b-a7a90e2af989","executionInfo":{"status":"ok","timestamp":1677904896083,"user_tz":-480,"elapsed":792,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":16,"outputs":[{"output_type":"stream","name":"stdout","text":["This query will process 553320240 bytes.\n"]}]},{"cell_type":"markdown","source":["You can also specify a parameter when running the query to limit how much data you are willing to scan. Here's an example with a low limit."],"metadata":{"id":"nkAUYjlr4fLV"}},{"cell_type":"code","source":["# Only run the query if it's less than 1 MB\n","ONE_MB = 1000*1000\n","safe_config = bigquery.QueryJobConfig(maximum_bytes_billed=ONE_MB)\n","\n","# Set up the query (will only run if it's less than 1 MB)\n","safe_query_job = client.query(query, job_config=safe_config)\n","\n","# API request - try to run the query, and return a pandas DataFrame\n","safe_query_job.to_dataframe()"],"metadata":{"execution":{"iopub.status.busy":"2022-04-23T03:44:14.959744Z","iopub.execute_input":"2022-04-23T03:44:14.960252Z","iopub.status.idle":"2022-04-23T03:44:15.424113Z","shell.execute_reply.started":"2022-04-23T03:44:14.960218Z","shell.execute_reply":"2022-04-23T03:44:15.422855Z"},"trusted":true,"id":"u2DeZlki4fLV","colab":{"base_uri":"https://localhost:8080/","height":399},"outputId":"a234343a-071c-4900-d046-7bb80ae804b1","executionInfo":{"status":"error","timestamp":1677904931810,"user_tz":-480,"elapsed":1087,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":17,"outputs":[{"output_type":"error","ename":"InternalServerError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mInternalServerError\u001b[0m                       Traceback (most recent call last)","\u001b[0;32m<ipython-input-17-3ffb41147547>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;31m# API request - try to run the query, and return a pandas DataFrame\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0msafe_query_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_dataframe\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/job/query.py\u001b[0m in \u001b[0;36mto_dataframe\u001b[0;34m(self, bqstorage_client, dtypes, progress_bar_type, create_bqstorage_client, max_results, geography_as_object)\u001b[0m\n\u001b[1;32m   1685\u001b[0m                 \u001b[0;34m:\u001b[0m\u001b[0mmod\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mshapely\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0mlibrary\u001b[0m \u001b[0mcannot\u001b[0m \u001b[0mbe\u001b[0m \u001b[0mimported\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1686\u001b[0m         \"\"\"\n\u001b[0;32m-> 1687\u001b[0;31m         \u001b[0mquery_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwait_for_query\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprogress_bar_type\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_results\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_results\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1688\u001b[0m         return query_result.to_dataframe(\n\u001b[1;32m   1689\u001b[0m             \u001b[0mbqstorage_client\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbqstorage_client\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/_tqdm_helpers.py\u001b[0m in \u001b[0;36mwait_for_query\u001b[0;34m(query_job, progress_bar_type, max_results)\u001b[0m\n\u001b[1;32m    102\u001b[0m     )\n\u001b[1;32m    103\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mprogress_bar\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 104\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mquery_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmax_results\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_results\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    105\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    106\u001b[0m     \u001b[0mi\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/job/query.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self, page_size, max_results, retry, timeout, start_index, job_retry)\u001b[0m\n\u001b[1;32m   1497\u001b[0m                 \u001b[0mdo_get_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjob_retry\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdo_get_result\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1498\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1499\u001b[0;31m             \u001b[0mdo_get_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1500\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1501\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGoogleAPICallError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/api_core/retry.py\u001b[0m in \u001b[0;36mretry_wrapped_func\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    347\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_initial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maximum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmultiplier\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_multiplier\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    348\u001b[0m             )\n\u001b[0;32m--> 349\u001b[0;31m             return retry_target(\n\u001b[0m\u001b[1;32m    350\u001b[0m                 \u001b[0mtarget\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    351\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_predicate\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/api_core/retry.py\u001b[0m in \u001b[0;36mretry_target\u001b[0;34m(target, predicate, sleep_generator, timeout, on_error, **kwargs)\u001b[0m\n\u001b[1;32m    189\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0msleep\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msleep_generator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    190\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 191\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mtarget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    192\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    193\u001b[0m         \u001b[0;31m# pylint: disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/job/query.py\u001b[0m in \u001b[0;36mdo_get_result\u001b[0;34m()\u001b[0m\n\u001b[1;32m   1487\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_job_retry\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjob_retry\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1488\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1489\u001b[0;31m                 \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mQueryJob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mretry\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mretry\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1490\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1491\u001b[0m                 \u001b[0;31m# Since the job could already be \"done\" (e.g. got a finished job\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/job/base.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self, retry, timeout)\u001b[0m\n\u001b[1;32m    726\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    727\u001b[0m         \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mretry\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mDEFAULT_RETRY\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m\"retry\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mretry\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 728\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_AsyncJob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    729\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    730\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcancelled\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/api_core/future/polling.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self, timeout, retry, polling)\u001b[0m\n\u001b[1;32m    259\u001b[0m             \u001b[0;31m# pylint: disable=raising-bad-type\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    260\u001b[0m             \u001b[0;31m# Pylint doesn't recognize that this is valid in this case.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 261\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    262\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    263\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mInternalServerError\u001b[0m: 500 Query exceeded limit for bytes billed: 1000000. 553648128 or higher required.\n\nLocation: US\nJob ID: 1cf6cfb2-4173-4ad2-af6e-762f014e560c\n"]}]},{"cell_type":"markdown","source":["In this case, the query was cancelled, because the limit of 1 MB was exceeded.  However, we can also increase the limit to run the query successfully!"],"metadata":{"id":"1Zs8R3_I4fLV"}},{"cell_type":"markdown","source":["### Group By, Having & Count"],"metadata":{"id":"-H9JRY8H6y6Q"}},{"cell_type":"markdown","source":["Now that you can select raw data, you're ready to learn how to group your data and count things within those groups."],"metadata":{"id":"vjHQ4EZe4fLV"}},{"cell_type":"markdown","source":["The Hacker News dataset contains information on stories and comments from the Hacker News social networking site. We'll work with the `comments` table and begin by printing the first few rows"],"metadata":{"id":"jlxSqT5I4fLW"}},{"cell_type":"code","source":["# Construct a reference to the \"hacker_news\" dataset\n","dataset_ref = client.dataset(\"hacker_news\", project=\"bigquery-public-data\")\n","\n","# API request - fetch the dataset\n","dataset = client.get_dataset(dataset_ref)\n","\n","# Construct a reference to the \"comments\" table\n","table_ref = dataset_ref.table(\"comments\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)\n","\n","# Preview the first five lines of the \"comments\" table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"execution":{"iopub.status.busy":"2022-04-23T03:50:17.465447Z","iopub.execute_input":"2022-04-23T03:50:17.465984Z","iopub.status.idle":"2022-04-23T03:50:18.444486Z","shell.execute_reply.started":"2022-04-23T03:50:17.465945Z","shell.execute_reply":"2022-04-23T03:50:18.443937Z"},"trusted":true,"id":"lbhZ1WrH4fLW","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"c3df653f-233f-4318-b8de-bc41cb0be246","executionInfo":{"status":"ok","timestamp":1677904944129,"user_tz":-480,"elapsed":1071,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":18,"outputs":[{"output_type":"execute_result","data":{"text/plain":["        id    by author        time                   time_ts  text   parent  \\\n","0  9734136  None   None  1434565400 2015-06-17 18:23:20+00:00  None  9733698   \n","1  4921158  None   None  1355496966 2012-12-14 14:56:06+00:00  None  4921100   \n","2  7500568  None   None  1396261158 2014-03-31 10:19:18+00:00  None  7499385   \n","3  8909635  None   None  1421627275 2015-01-19 00:27:55+00:00  None  8901135   \n","4  9256463  None   None  1427204705 2015-03-24 13:45:05+00:00  None  9256346   \n","\n","   deleted  dead  ranking  \n","0     True  <NA>        0  \n","1     True  <NA>        0  \n","2     True  <NA>        0  \n","3     True  <NA>        0  \n","4     True  <NA>        0  "],"text/html":["\n","  <div id=\"df-53bc3cb9-663f-483e-a32a-76ae1864ac39\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>by</th>\n","      <th>author</th>\n","      <th>time</th>\n","      <th>time_ts</th>\n","      <th>text</th>\n","      <th>parent</th>\n","      <th>deleted</th>\n","      <th>dead</th>\n","      <th>ranking</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>9734136</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1434565400</td>\n","      <td>2015-06-17 18:23:20+00:00</td>\n","      <td>None</td>\n","      <td>9733698</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>4921158</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1355496966</td>\n","      <td>2012-12-14 14:56:06+00:00</td>\n","      <td>None</td>\n","      <td>4921100</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>7500568</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1396261158</td>\n","      <td>2014-03-31 10:19:18+00:00</td>\n","      <td>None</td>\n","      <td>7499385</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>8909635</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1421627275</td>\n","      <td>2015-01-19 00:27:55+00:00</td>\n","      <td>None</td>\n","      <td>8901135</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>9256463</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1427204705</td>\n","      <td>2015-03-24 13:45:05+00:00</td>\n","      <td>None</td>\n","      <td>9256346</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-53bc3cb9-663f-483e-a32a-76ae1864ac39')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-53bc3cb9-663f-483e-a32a-76ae1864ac39 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-53bc3cb9-663f-483e-a32a-76ae1864ac39');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":18}]},{"cell_type":"markdown","source":["Let's use the table to see which comments generated the most replies.  Since:\n","- the `parent` column indicates the comment that was replied to, and \n","- the `id` column has the unique ID used to identify each comment, \n","\n","we can **GROUP BY** the `parent` column and **COUNT()** the `id` column in order to figure out the number of comments that were made as responses to a specific comment.\n","\n","Furthermore, since we're only interested in popular comments, we'll look at comments with more than ten replies.  So, we'll only return groups **HAVING** more than ten ID's."],"metadata":{"id":"5xQZeTKP4fLW"}},{"cell_type":"code","source":["# Query to select comments that received more than 10 replies\n","query_popular = \"\"\"\n","                SELECT parent, COUNT(id)\n","                FROM `bigquery-public-data.hacker_news.comments`\n","                GROUP BY parent\n","                HAVING COUNT(id) > 10\n","                \"\"\""],"metadata":{"execution":{"iopub.status.busy":"2022-04-23T03:51:01.675659Z","iopub.execute_input":"2022-04-23T03:51:01.675922Z","iopub.status.idle":"2022-04-23T03:51:01.682999Z","shell.execute_reply.started":"2022-04-23T03:51:01.675896Z","shell.execute_reply":"2022-04-23T03:51:01.682129Z"},"trusted":true,"id":"GTGRV8GE4fLW","executionInfo":{"status":"ok","timestamp":1677904947798,"user_tz":-480,"elapsed":2,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":19,"outputs":[]},{"cell_type":"code","source":["# Set up the query (cancel the query if it would use too much of \n","# your quota, with the limit set to 10 GB)\n","safe_config = bigquery.QueryJobConfig(maximum_bytes_billed=10**10)\n","query_job = client.query(query_popular, job_config=safe_config)\n","\n","# API request - run the query, and convert the results to a pandas DataFrame\n","popular_comments = query_job.to_dataframe()\n","\n","# Print the first five rows of the DataFrame\n","popular_comments.head()"],"metadata":{"execution":{"iopub.status.busy":"2022-04-23T03:51:25.840649Z","iopub.execute_input":"2022-04-23T03:51:25.841060Z","iopub.status.idle":"2022-04-23T03:51:32.497782Z","shell.execute_reply.started":"2022-04-23T03:51:25.841031Z","shell.execute_reply":"2022-04-23T03:51:32.496981Z"},"trusted":true,"id":"Wb7hgSV24fLW","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"547de823-1dd6-46cd-cdc0-028f947f0a32","executionInfo":{"status":"ok","timestamp":1677904952563,"user_tz":-480,"elapsed":4766,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":20,"outputs":[{"output_type":"execute_result","data":{"text/plain":["    parent  f0_\n","0  2385424   55\n","1  8441979   57\n","2  7634152   46\n","3  9062758   49\n","4  5694173   61"],"text/html":["\n","  <div id=\"df-36c5785d-e5cc-4839-a039-b37aa532c175\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>parent</th>\n","      <th>f0_</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2385424</td>\n","      <td>55</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>8441979</td>\n","      <td>57</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>7634152</td>\n","      <td>46</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>9062758</td>\n","      <td>49</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>5694173</td>\n","      <td>61</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-36c5785d-e5cc-4839-a039-b37aa532c175')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-36c5785d-e5cc-4839-a039-b37aa532c175 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-36c5785d-e5cc-4839-a039-b37aa532c175');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":20}]},{"cell_type":"code","source":["popular_comments"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"TVI78T52tigN","outputId":"0b576bc4-f1df-4b66-d190-3a9ddd5fe105","executionInfo":{"status":"ok","timestamp":1677904952563,"user_tz":-480,"elapsed":5,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":21,"outputs":[{"output_type":"execute_result","data":{"text/plain":["        parent  f0_\n","0      2385424   55\n","1      8441979   57\n","2      7634152   46\n","3      9062758   49\n","4      5694173   61\n","...        ...  ...\n","77363  1748827   37\n","77364  3657756   37\n","77365  2873865   37\n","77366  9395540   37\n","77367  1772903   37\n","\n","[77368 rows x 2 columns]"],"text/html":["\n","  <div id=\"df-1ec3b4f4-92b3-4086-aad9-b948f9b4778e\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>parent</th>\n","      <th>f0_</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2385424</td>\n","      <td>55</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>8441979</td>\n","      <td>57</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>7634152</td>\n","      <td>46</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>9062758</td>\n","      <td>49</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>5694173</td>\n","      <td>61</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>77363</th>\n","      <td>1748827</td>\n","      <td>37</td>\n","    </tr>\n","    <tr>\n","      <th>77364</th>\n","      <td>3657756</td>\n","      <td>37</td>\n","    </tr>\n","    <tr>\n","      <th>77365</th>\n","      <td>2873865</td>\n","      <td>37</td>\n","    </tr>\n","    <tr>\n","      <th>77366</th>\n","      <td>9395540</td>\n","      <td>37</td>\n","    </tr>\n","    <tr>\n","      <th>77367</th>\n","      <td>1772903</td>\n","      <td>37</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>77368 rows × 2 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1ec3b4f4-92b3-4086-aad9-b948f9b4778e')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-1ec3b4f4-92b3-4086-aad9-b948f9b4778e button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-1ec3b4f4-92b3-4086-aad9-b948f9b4778e');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":21}]},{"cell_type":"markdown","source":["Each row in the `popular_comments` DataFrame corresponds to a comment that received more than ten replies."],"metadata":{"id":"j1eXKXzN4fLX"}},{"cell_type":"markdown","source":["A couple hints to make your queries even better:\n","- The column resulting from `COUNT(id)` was called `f0__`. That's not a very descriptive name. You can change the name by adding `AS NumPosts` after you specify the aggregation. This is called **aliasing**.\n","- If you are ever unsure what to put inside the **COUNT()** function, you can do `COUNT(1)` to count the rows in each group. Most people find it especially readable, because we know it's not focusing on other columns. It also scans less data than if supplied column names (making it faster and using less of your data access quota).\n","\n","Using these tricks, we can rewrite our query:"],"metadata":{"id":"UVvf6W2c9LbB"}},{"cell_type":"code","source":["# Improved version of earlier query, now with aliasing & improved readability\n","query_improved = \"\"\"\n","                 SELECT parent, COUNT(1) AS NumPosts\n","                 FROM `bigquery-public-data.hacker_news.comments`\n","                 GROUP BY parent\n","                 HAVING COUNT(1) > 10\n","                 \"\"\"\n","\n","safe_config = bigquery.QueryJobConfig(maximum_bytes_billed=10**10)\n","query_job = client.query(query_improved, job_config=safe_config)\n","\n","# API request - run the query, and convert the results to a pandas DataFrame\n","improved_df = query_job.to_dataframe()\n","\n","# Print the first five rows of the DataFrame\n","improved_df.head()"],"metadata":{"execution":{"iopub.status.busy":"2022-04-23T03:53:18.501704Z","iopub.execute_input":"2022-04-23T03:53:18.501994Z","iopub.status.idle":"2022-04-23T03:53:24.596965Z","shell.execute_reply.started":"2022-04-23T03:53:18.501962Z","shell.execute_reply":"2022-04-23T03:53:24.596042Z"},"trusted":true,"id":"YCqkWvK_4fLX","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"ed4a6f41-1e06-4c1c-d1dd-8f5d4db42ab2","executionInfo":{"status":"ok","timestamp":1677904957388,"user_tz":-480,"elapsed":4829,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":22,"outputs":[{"output_type":"execute_result","data":{"text/plain":["    parent  NumPosts\n","0  2970550        63\n","1  8254532        40\n","2  7687784        44\n","3  4101992        53\n","4  1632878        39"],"text/html":["\n","  <div id=\"df-ad9c1f95-d765-4d33-8e5d-7c3470b0e704\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>parent</th>\n","      <th>NumPosts</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2970550</td>\n","      <td>63</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>8254532</td>\n","      <td>40</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>7687784</td>\n","      <td>44</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>4101992</td>\n","      <td>53</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>1632878</td>\n","      <td>39</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ad9c1f95-d765-4d33-8e5d-7c3470b0e704')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-ad9c1f95-d765-4d33-8e5d-7c3470b0e704 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-ad9c1f95-d765-4d33-8e5d-7c3470b0e704');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":22}]},{"cell_type":"code","source":["improved_df"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"e4yVjvH2tcBf","outputId":"1a455b8b-0efc-4c8f-e13c-bd785d716486","executionInfo":{"status":"ok","timestamp":1677904957389,"user_tz":-480,"elapsed":6,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":23,"outputs":[{"output_type":"execute_result","data":{"text/plain":["        parent  NumPosts\n","0      2970550        63\n","1      8254532        40\n","2      7687784        44\n","3      4101992        53\n","4      1632878        39\n","...        ...       ...\n","77363  7844298        37\n","77364  7864644        37\n","77365  2866332        37\n","77366  1885594        37\n","77367  2175321        37\n","\n","[77368 rows x 2 columns]"],"text/html":["\n","  <div id=\"df-e4e931ef-1e18-4e6a-8a11-10141bce2dba\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>parent</th>\n","      <th>NumPosts</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2970550</td>\n","      <td>63</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>8254532</td>\n","      <td>40</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>7687784</td>\n","      <td>44</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>4101992</td>\n","      <td>53</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>1632878</td>\n","      <td>39</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>77363</th>\n","      <td>7844298</td>\n","      <td>37</td>\n","    </tr>\n","    <tr>\n","      <th>77364</th>\n","      <td>7864644</td>\n","      <td>37</td>\n","    </tr>\n","    <tr>\n","      <th>77365</th>\n","      <td>2866332</td>\n","      <td>37</td>\n","    </tr>\n","    <tr>\n","      <th>77366</th>\n","      <td>1885594</td>\n","      <td>37</td>\n","    </tr>\n","    <tr>\n","      <th>77367</th>\n","      <td>2175321</td>\n","      <td>37</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>77368 rows × 2 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e4e931ef-1e18-4e6a-8a11-10141bce2dba')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-e4e931ef-1e18-4e6a-8a11-10141bce2dba button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-e4e931ef-1e18-4e6a-8a11-10141bce2dba');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":23}]},{"cell_type":"markdown","source":["Now you have the data you want, and it has descriptive names. "],"metadata":{"id":"BzWeL50dw8rg"}},{"cell_type":"markdown","source":["#### Note on using **GROUP BY**"],"metadata":{"id":"LTa9oaK-w_6g"}},{"cell_type":"markdown","source":["Note that because it tells SQL how to apply aggregate functions (like **COUNT()**), it doesn't make sense to use **GROUP BY** without an aggregate function.  Similarly, if you have any **GROUP BY** clause, then all variables must be passed to either a\n","1. **GROUP BY** command, or\n","2. an aggregation function.\n","\n","Consider the query below:\n","\n"],"metadata":{"id":"hNf8IMK_4fLX"}},{"cell_type":"code","source":["query_good = \"\"\"\n","             SELECT parent, COUNT(id)\n","             FROM `bigquery-public-data.hacker_news.comments`\n","             GROUP BY parent\n","             \"\"\""],"metadata":{"id":"dTzcM2C04fLX","executionInfo":{"status":"ok","timestamp":1677904957389,"user_tz":-480,"elapsed":5,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":24,"outputs":[]},{"cell_type":"markdown","source":["Note that there are two variables: `parent` and `id`. \n","- `parent` was passed to a **GROUP BY** command (in `GROUP BY parent`), and \n","- `id` was passed to an aggregate function (in `COUNT(id)`).\n","\n","And the query below won't work, because the `author` column isn't passed to an aggregate function or a **GROUP BY** clause:"],"metadata":{"id":"oOpimGND4fLX"}},{"cell_type":"code","source":["query_bad = \"\"\"\n","            SELECT author, parent, COUNT(id)\n","            FROM `bigquery-public-data.hacker_news.comments`\n","            GROUP BY parent\n","            \"\"\""],"metadata":{"id":"57X5jB1J4fLX","executionInfo":{"status":"ok","timestamp":1677904957389,"user_tz":-480,"elapsed":4,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":25,"outputs":[]},{"cell_type":"code","source":["safe_config = bigquery.QueryJobConfig(maximum_bytes_billed=10**10)\n","query_job = client.query(query_bad, job_config=safe_config)\n","\n","# API request - run the query, and convert the results to a pandas DataFrame\n","improved_df = query_job.to_dataframe()\n","\n","# Print the first five rows of the DataFrame\n","improved_df.head()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":434},"id":"voa8Erj2yrk_","outputId":"98030991-b11b-4cab-b355-1c669df47bd9","executionInfo":{"status":"error","timestamp":1677904957827,"user_tz":-480,"elapsed":442,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":26,"outputs":[{"output_type":"error","ename":"BadRequest","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mBadRequest\u001b[0m                                Traceback (most recent call last)","\u001b[0;32m<ipython-input-26-25bce487f594>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# API request - run the query, and convert the results to a pandas DataFrame\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mimproved_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquery_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_dataframe\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m# Print the first five rows of the DataFrame\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/job/query.py\u001b[0m in \u001b[0;36mto_dataframe\u001b[0;34m(self, bqstorage_client, dtypes, progress_bar_type, create_bqstorage_client, max_results, geography_as_object)\u001b[0m\n\u001b[1;32m   1685\u001b[0m                 \u001b[0;34m:\u001b[0m\u001b[0mmod\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mshapely\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0mlibrary\u001b[0m \u001b[0mcannot\u001b[0m \u001b[0mbe\u001b[0m \u001b[0mimported\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1686\u001b[0m         \"\"\"\n\u001b[0;32m-> 1687\u001b[0;31m         \u001b[0mquery_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwait_for_query\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprogress_bar_type\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_results\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_results\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1688\u001b[0m         return query_result.to_dataframe(\n\u001b[1;32m   1689\u001b[0m             \u001b[0mbqstorage_client\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbqstorage_client\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/_tqdm_helpers.py\u001b[0m in \u001b[0;36mwait_for_query\u001b[0;34m(query_job, progress_bar_type, max_results)\u001b[0m\n\u001b[1;32m    102\u001b[0m     )\n\u001b[1;32m    103\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mprogress_bar\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 104\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mquery_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmax_results\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_results\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    105\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    106\u001b[0m     \u001b[0mi\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/job/query.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self, page_size, max_results, retry, timeout, start_index, job_retry)\u001b[0m\n\u001b[1;32m   1497\u001b[0m                 \u001b[0mdo_get_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjob_retry\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdo_get_result\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1498\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1499\u001b[0;31m             \u001b[0mdo_get_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1500\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1501\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGoogleAPICallError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/api_core/retry.py\u001b[0m in \u001b[0;36mretry_wrapped_func\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    347\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_initial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maximum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmultiplier\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_multiplier\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    348\u001b[0m             )\n\u001b[0;32m--> 349\u001b[0;31m             return retry_target(\n\u001b[0m\u001b[1;32m    350\u001b[0m                 \u001b[0mtarget\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    351\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_predicate\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/api_core/retry.py\u001b[0m in \u001b[0;36mretry_target\u001b[0;34m(target, predicate, sleep_generator, timeout, on_error, **kwargs)\u001b[0m\n\u001b[1;32m    189\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0msleep\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msleep_generator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    190\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 191\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mtarget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    192\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    193\u001b[0m         \u001b[0;31m# pylint: disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/job/query.py\u001b[0m in \u001b[0;36mdo_get_result\u001b[0;34m()\u001b[0m\n\u001b[1;32m   1487\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_job_retry\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjob_retry\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1488\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1489\u001b[0;31m                 \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mQueryJob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mretry\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mretry\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1490\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1491\u001b[0m                 \u001b[0;31m# Since the job could already be \"done\" (e.g. got a finished job\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/bigquery/job/base.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self, retry, timeout)\u001b[0m\n\u001b[1;32m    726\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    727\u001b[0m         \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mretry\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mDEFAULT_RETRY\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m\"retry\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mretry\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 728\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_AsyncJob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    729\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    730\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcancelled\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/api_core/future/polling.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self, timeout, retry, polling)\u001b[0m\n\u001b[1;32m    259\u001b[0m             \u001b[0;31m# pylint: disable=raising-bad-type\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    260\u001b[0m             \u001b[0;31m# Pylint doesn't recognize that this is valid in this case.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 261\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    262\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    263\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mBadRequest\u001b[0m: 400 SELECT list expression references column author which is neither grouped nor aggregated at [2:20]\n\nLocation: US\nJob ID: 6582f1e3-c61a-4a5e-a89a-c06487b6fe5c\n"]}]},{"cell_type":"markdown","source":["### Order By"],"metadata":{"id":"RYy0qPJ89w6r"}},{"cell_type":"markdown","source":["Frequently, you’ll want to sort your results. **Let's use the US Traffic Fatality Records database, which contains information on traffic accidents in the US where at least one person died.**\n","\n","We'll investigate the `accident_2015` table. Here is a view of the first few rows. "],"metadata":{"id":"T7euCJRKBCxu"}},{"cell_type":"code","source":["# Construct a reference to the \"nhtsa_traffic_fatalities\" dataset\n","dataset_ref = client.dataset(\"nhtsa_traffic_fatalities\", project=\"bigquery-public-data\")\n","\n","# API request - fetch the dataset\n","dataset = client.get_dataset(dataset_ref)\n","\n","# Construct a reference to the \"accident_2015\" table\n","table_ref = dataset_ref.table(\"accident_2015\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)\n","\n","# Preview the first five lines of the \"accident_2015\" table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"id":"JpqOyftp9wVT","executionInfo":{"status":"aborted","timestamp":1677904957828,"user_tz":-480,"elapsed":3,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Let's use the table to determine how the number of accidents varies with the day of the week.  Since:\n","- the `consecutive_number` column contains a unique ID for each accident, and\n","- the `timestamp_of_crash` column contains the date of the accident in [DATETIME](https://cloud.google.com/bigquery/docs/reference/standard-sql/date_functions) format,\n","\n","we can:\n","- **EXTRACT** the day of the week (as `day_of_week` in the query below) from the `timestamp_of_crash` column, and\n","- **GROUP BY** the day of the week, before we **COUNT** the `consecutive_number` column to determine the number of accidents for each day of the week.\n","\n","Then we sort the table with an **ORDER BY** clause, so the days with the most accidents are returned first."],"metadata":{"id":"UUJ2fuKmBSc1"}},{"cell_type":"code","source":["# Query to find out the number of accidents for each day of the week\n","query = \"\"\"\n","        SELECT COUNT(consecutive_number) AS num_accidents, \n","               EXTRACT(DAYOFWEEK FROM timestamp_of_crash) AS day_of_week\n","        FROM `bigquery-public-data.nhtsa_traffic_fatalities.accident_2015`\n","        GROUP BY day_of_week\n","        ORDER BY num_accidents DESC\n","        \"\"\""],"metadata":{"id":"4I_8JldMBc04","executionInfo":{"status":"ok","timestamp":1677904958457,"user_tz":-480,"elapsed":2,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":27,"outputs":[]},{"cell_type":"code","source":["# Set up the query (cancel the query if it would use too much of \n","# your quota, with the limit set to 1 GB)\n","safe_config = bigquery.QueryJobConfig(maximum_bytes_billed=10**9)\n","query_job = client.query(query, job_config=safe_config)\n","\n","# API request - run the query, and convert the results to a pandas DataFrame\n","accidents_by_day = query_job.to_dataframe()\n","\n","# Print the DataFrame\n","accidents_by_day"],"metadata":{"id":"B6Dks4_hBrLv","colab":{"base_uri":"https://localhost:8080/","height":269},"outputId":"1b7328f6-2197-434c-9901-635057921419","executionInfo":{"status":"ok","timestamp":1677904962422,"user_tz":-480,"elapsed":3091,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":28,"outputs":[{"output_type":"execute_result","data":{"text/plain":["   num_accidents  day_of_week\n","0           5659            7\n","1           5298            1\n","2           4916            6\n","3           4460            5\n","4           4182            4\n","5           4038            2\n","6           3985            3"],"text/html":["\n","  <div id=\"df-60df7ecf-164e-409f-9659-78da70e9d1d1\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>num_accidents</th>\n","      <th>day_of_week</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>5659</td>\n","      <td>7</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>5298</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>4916</td>\n","      <td>6</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>4460</td>\n","      <td>5</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>4182</td>\n","      <td>4</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>4038</td>\n","      <td>2</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>3985</td>\n","      <td>3</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-60df7ecf-164e-409f-9659-78da70e9d1d1')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-60df7ecf-164e-409f-9659-78da70e9d1d1 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-60df7ecf-164e-409f-9659-78da70e9d1d1');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":28}]},{"cell_type":"markdown","source":["Notice that the data is sorted by the `num_accidents` column, where the days with more traffic accidents appear first.\n","\n","To map the numbers returned for the `day_of_week` column to the actual day, you might consult [the BigQuery documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/date_functions) on the DAYOFWEEK function. It says that it returns \"an integer between 1 (Sunday) and 7 (Saturday), inclusively\". So, in 2015, most fatal motor accidents in the US occured on Sunday and Saturday, while the fewest happened on Tuesday."],"metadata":{"id":"vQ6usjVXBt4z"}},{"cell_type":"markdown","source":["### As and With"],"metadata":{"id":"gC3QTzVCI6ST"}},{"cell_type":"markdown","source":["On its own, `AS` is a convenient way to clean up the data returned by your query. **We're going to use a common table expression (CTE)** to find out **how many Bitcoin transactions were made each day for the entire timespan of a bitcoin transaction dataset.**\n","\n","We'll investigate the transactions table. Here is a view of the first few rows."],"metadata":{"id":"gsAQF0yII_2j"}},{"cell_type":"code","source":["# Construct a reference to the \"crypto_bitcoin\" dataset\n","dataset_ref = client.dataset(\"crypto_bitcoin\", project=\"bigquery-public-data\")\n","\n","# API request - fetch the dataset\n","dataset = client.get_dataset(dataset_ref)\n","\n","# Construct a reference to the \"transactions\" table\n","table_ref = dataset_ref.table(\"transactions\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)\n","\n","# Preview the first five lines of the \"transactions\" table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"id":"PN9j763YI-rz","colab":{"base_uri":"https://localhost:8080/","height":444},"outputId":"0f810490-f2ee-4744-dbc2-5cb730f71760","executionInfo":{"status":"ok","timestamp":1677904964332,"user_tz":-480,"elapsed":942,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":29,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                                                hash  size  virtual_size  \\\n","0  a16f3ce4dd5deb92d98ef5cf8afeaf0775ebca408f708b...   275           275   \n","1  591e91f809d716912ca1d4a9295e70c3e78bab077683f7...   275           275   \n","2  12b5633bad1f9c167d523ad1aa1947b2732a865bf5414e...   276           276   \n","3  828ef3b079f9c23829c56fe86e85b4a69d9e06e5b54ea5...   276           276   \n","4  35288d269cee1941eaebb2ea85e32b42cdb2b04284a56d...   277           277   \n","\n","   version  lock_time                                         block_hash  \\\n","0        1          0  00000000dc55860c8a29c58d45209318fa9e9dc2c1833a...   \n","1        1          0  0000000054487811fc4ff7a95be738aa5ad9320c394c48...   \n","2        1          0  00000000f46e513f038baf6f2d9a95b2a28d8a6c985bcf...   \n","3        1          0  00000000fb5b44edc7a1aa105075564a179d65506e2bd2...   \n","4        1          0  00000000689051c09ff2cd091cc4c22c10b965eb8db3ad...   \n","\n","   block_number           block_timestamp block_timestamp_month  input_count  \\\n","0           181 2009-01-12 06:02:13+00:00            2009-01-01            1   \n","1           182 2009-01-12 06:12:16+00:00            2009-01-01            1   \n","2           183 2009-01-12 06:34:22+00:00            2009-01-01            1   \n","3           248 2009-01-12 20:04:20+00:00            2009-01-01            1   \n","4           545 2009-01-15 05:48:32+00:00            2009-01-01            1   \n","\n","   output_count           input_value          output_value  is_coinbase  \\\n","0             2  4000000000.000000000  4000000000.000000000        False   \n","1             2  3000000000.000000000  3000000000.000000000        False   \n","2             2  2900000000.000000000  2900000000.000000000        False   \n","3             2  2800000000.000000000  2800000000.000000000        False   \n","4             2  2500000000.000000000  2500000000.000000000        False   \n","\n","    fee                                             inputs  \\\n","0  0E-9  [{'index': 0, 'spent_transaction_hash': 'f4184...   \n","1  0E-9  [{'index': 0, 'spent_transaction_hash': 'a16f3...   \n","2  0E-9  [{'index': 0, 'spent_transaction_hash': '591e9...   \n","3  0E-9  [{'index': 0, 'spent_transaction_hash': '12b56...   \n","4  0E-9  [{'index': 0, 'spent_transaction_hash': 'd71fd...   \n","\n","                                             outputs  \n","0  [{'index': 0, 'script_asm': '04b5abd412d4341b4...  \n","1  [{'index': 0, 'script_asm': '0401518fa1d1e1e3e...  \n","2  [{'index': 0, 'script_asm': '04baa9d3665315562...  \n","3  [{'index': 0, 'script_asm': '04bed827d37474bef...  \n","4  [{'index': 0, 'script_asm': '044a656f065871a35...  "],"text/html":["\n","  <div id=\"df-6de6cf2c-be5b-4dc5-a97f-820f6f2603a9\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>hash</th>\n","      <th>size</th>\n","      <th>virtual_size</th>\n","      <th>version</th>\n","      <th>lock_time</th>\n","      <th>block_hash</th>\n","      <th>block_number</th>\n","      <th>block_timestamp</th>\n","      <th>block_timestamp_month</th>\n","      <th>input_count</th>\n","      <th>output_count</th>\n","      <th>input_value</th>\n","      <th>output_value</th>\n","      <th>is_coinbase</th>\n","      <th>fee</th>\n","      <th>inputs</th>\n","      <th>outputs</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>a16f3ce4dd5deb92d98ef5cf8afeaf0775ebca408f708b...</td>\n","      <td>275</td>\n","      <td>275</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>00000000dc55860c8a29c58d45209318fa9e9dc2c1833a...</td>\n","      <td>181</td>\n","      <td>2009-01-12 06:02:13+00:00</td>\n","      <td>2009-01-01</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>4000000000.000000000</td>\n","      <td>4000000000.000000000</td>\n","      <td>False</td>\n","      <td>0E-9</td>\n","      <td>[{'index': 0, 'spent_transaction_hash': 'f4184...</td>\n","      <td>[{'index': 0, 'script_asm': '04b5abd412d4341b4...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>591e91f809d716912ca1d4a9295e70c3e78bab077683f7...</td>\n","      <td>275</td>\n","      <td>275</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0000000054487811fc4ff7a95be738aa5ad9320c394c48...</td>\n","      <td>182</td>\n","      <td>2009-01-12 06:12:16+00:00</td>\n","      <td>2009-01-01</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>3000000000.000000000</td>\n","      <td>3000000000.000000000</td>\n","      <td>False</td>\n","      <td>0E-9</td>\n","      <td>[{'index': 0, 'spent_transaction_hash': 'a16f3...</td>\n","      <td>[{'index': 0, 'script_asm': '0401518fa1d1e1e3e...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>12b5633bad1f9c167d523ad1aa1947b2732a865bf5414e...</td>\n","      <td>276</td>\n","      <td>276</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>00000000f46e513f038baf6f2d9a95b2a28d8a6c985bcf...</td>\n","      <td>183</td>\n","      <td>2009-01-12 06:34:22+00:00</td>\n","      <td>2009-01-01</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>2900000000.000000000</td>\n","      <td>2900000000.000000000</td>\n","      <td>False</td>\n","      <td>0E-9</td>\n","      <td>[{'index': 0, 'spent_transaction_hash': '591e9...</td>\n","      <td>[{'index': 0, 'script_asm': '04baa9d3665315562...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>828ef3b079f9c23829c56fe86e85b4a69d9e06e5b54ea5...</td>\n","      <td>276</td>\n","      <td>276</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>00000000fb5b44edc7a1aa105075564a179d65506e2bd2...</td>\n","      <td>248</td>\n","      <td>2009-01-12 20:04:20+00:00</td>\n","      <td>2009-01-01</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>2800000000.000000000</td>\n","      <td>2800000000.000000000</td>\n","      <td>False</td>\n","      <td>0E-9</td>\n","      <td>[{'index': 0, 'spent_transaction_hash': '12b56...</td>\n","      <td>[{'index': 0, 'script_asm': '04bed827d37474bef...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>35288d269cee1941eaebb2ea85e32b42cdb2b04284a56d...</td>\n","      <td>277</td>\n","      <td>277</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>00000000689051c09ff2cd091cc4c22c10b965eb8db3ad...</td>\n","      <td>545</td>\n","      <td>2009-01-15 05:48:32+00:00</td>\n","      <td>2009-01-01</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>2500000000.000000000</td>\n","      <td>2500000000.000000000</td>\n","      <td>False</td>\n","      <td>0E-9</td>\n","      <td>[{'index': 0, 'spent_transaction_hash': 'd71fd...</td>\n","      <td>[{'index': 0, 'script_asm': '044a656f065871a35...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6de6cf2c-be5b-4dc5-a97f-820f6f2603a9')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-6de6cf2c-be5b-4dc5-a97f-820f6f2603a9 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-6de6cf2c-be5b-4dc5-a97f-820f6f2603a9');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":29}]},{"cell_type":"markdown","source":["Since the `block_timestamp` column contains the date of each transaction in DATETIME format, we'll convert these into DATE format using the **DATE()** command.\n","\n","We do that using a CTE, and then the next part of the query counts the number of transactions for each date and sorts the table so that earlier dates appear first. "],"metadata":{"id":"afJXEFTGK1vO"}},{"cell_type":"code","source":["# Query to select the number of transactions per date, sorted by date\n","query_with_CTE = \"\"\" \n","                 WITH time AS \n","                 (\n","                     SELECT DATE(block_timestamp) AS trans_date\n","                     FROM `bigquery-public-data.crypto_bitcoin.transactions`\n","                 )\n","                 SELECT COUNT(1) AS transactions,\n","                        trans_date\n","                 FROM time\n","                 GROUP BY trans_date\n","                 ORDER BY trans_date\n","                 \"\"\"\n","\n","# Set up the query (cancel the query if it would use too much of \n","# your quota, with the limit set to 10 GB)\n","safe_config = bigquery.QueryJobConfig(maximum_bytes_billed=10**10)\n","query_job = client.query(query_with_CTE, job_config=safe_config)\n","\n","# API request - run the query, and convert the results to a pandas DataFrame\n","transactions_by_date = query_job.to_dataframe()\n","\n","# Print the first five rows\n","transactions_by_date.head()"],"metadata":{"id":"yvWGKLH2B8Qa","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"73c6074a-9701-43e2-82f9-d212c1750417","executionInfo":{"status":"ok","timestamp":1677904978087,"user_tz":-480,"elapsed":12341,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":30,"outputs":[{"output_type":"execute_result","data":{"text/plain":["   transactions  trans_date\n","0             1  2009-01-03\n","1            14  2009-01-09\n","2            61  2009-01-10\n","3            93  2009-01-11\n","4           101  2009-01-12"],"text/html":["\n","  <div id=\"df-5045fd7a-3f26-4889-9cac-fd8deff3d18b\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>transactions</th>\n","      <th>trans_date</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>2009-01-03</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>14</td>\n","      <td>2009-01-09</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>61</td>\n","      <td>2009-01-10</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>93</td>\n","      <td>2009-01-11</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>101</td>\n","      <td>2009-01-12</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5045fd7a-3f26-4889-9cac-fd8deff3d18b')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-5045fd7a-3f26-4889-9cac-fd8deff3d18b button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-5045fd7a-3f26-4889-9cac-fd8deff3d18b');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":30}]},{"cell_type":"markdown","source":["Since they're returned sorted, we can easily plot the raw results to show us the number of Bitcoin transactions per day over the whole timespan of this dataset."],"metadata":{"id":"gqgnFoJ1LMYf"}},{"cell_type":"code","source":["transactions_by_date.set_index('trans_date').plot()"],"metadata":{"id":"qXRnbHIELMzn","colab":{"base_uri":"https://localhost:8080/","height":297},"outputId":"2365b1e6-b1b7-442f-ad5e-69af791de67a","executionInfo":{"status":"ok","timestamp":1677904978603,"user_tz":-480,"elapsed":523,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":31,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<AxesSubplot:xlabel='trans_date'>"]},"metadata":{},"execution_count":31},{"output_type":"display_data","data":{"text/plain":["<Figure size 432x288 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYkAAAEHCAYAAABbZ7oVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3vElEQVR4nO3dd3xV9f348dc7CSGEPQIyDQoiSxAioDgYggFUtI6vG63Kz6JVa63Gah24sLV1Vq0iRW3rqLVKBaQMGZYZZC9JACVhj4QZyPj8/rjnXu69uTvnjiTv5+ORB/d+zjmf8yHjvO9nizEGpZRSypekeBdAKaVU4tIgoZRSyi8NEkoppfzSIKGUUsovDRJKKaX8Sol3AezWokULk5mZGe9iKKVUtbJ8+fJ9xpgM7/QaFyQyMzPJzc2NdzGUUqpaEZEffaVrc5NSSim/QgoSIrJNRNaIyEoRybXSmonITBHZbP3b1EoXEXldRPJEZLWI9HHLZ4x1/mYRGeOW3tfKP8+6VgLdQymlVGyEU5MYbIzpbYzJst7nALONMZ2B2dZ7gBFAZ+trLPA2OB74wFNAf6Af8JTbQ/9t4G6367KD3EMppVQMVKVPYjQwyHr9ATAXeNRK/9A41vtYLCJNRKS1de5MY8wBABGZCWSLyFygkTFmsZX+IXAVMD3APZRS1VBpaSkFBQWUlJTEuyi1VlpaGu3ataNOnTohnR9qkDDAf0XEAH8xxrwLtDLG7LSO7wJaWa/bAtvdri2w0gKlF/hIJ8A9lFLVUEFBAQ0bNiQzMxOrVVnFkDGG/fv3U1BQQMeOHUO6JtQgcaExplBEWgIzRWSj142NFUCiJtA9RGQsjqYtOnToEM1iKKWqoKSkRANEHIkIzZs3Z+/evSFfE1KfhDGm0Pp3D/BvHH0Ku61mJKx/91inFwLt3S5vZ6UFSm/nI50A9/Au37vGmCxjTFZGRqVhvkqpBKIBIr7C/f4HDRIiUl9EGjpfA8OBtcAUwDlCaQzwlfV6CnCbNcppAFBsNRnNAIaLSFOrw3o4MMM6dkhEBlijmm7zysvXPZRSNio+VkrutgPxLoZKQKHUJFoB34nIKmApMNUY8w0wARgmIpuBS633ANOALUAe8B4wDsDqsH4WWGZ9jXd2YlvnTLSuycfRaU2AeyilbDTmr0u59p1FlJVXxLsoUVVUVMRbb70V72J4mDx5Mjt27HC9v+uuu1i/fn0cS+QpaJ+EMWYL0MtH+n5gqI90A9zrJ69JwCQf6blAj1DvoZSy19rCYsAxQqUmcwaJcePGeaSXlZWRkhKfBSgmT55Mjx49aNOmDQATJ06MSzn80RnXSqlaIycnh/z8fHr37s15553HRRddxJVXXkm3bt0AuOqqq+jbty/du3fn3XffdV3XoEEDHn/8cXr16sWAAQPYvXs3AP/85z/p0aMHvXr14uKLLwZg27ZtXHTRRfTp04c+ffqwcOFCVz4vvfQSPXv2pFevXuTk5PD555+Tm5vLzTffTO/evTl+/DiDBg1yLS308ccf07NnT3r06MGjjz4aUXmqqsat3aSUqh6e+c861u84ZGue3do04qkruvs9PmHCBNauXcvKlSuZO3cuo0aNYu3ata7hoJMmTaJZs2YcP36c8847j2uuuYbmzZtz9OhRBgwYwPPPP88jjzzCe++9xxNPPMH48eOZMWMGbdu2paioCICWLVsyc+ZM0tLS2Lx5MzfeeCO5ublMnz6dr776iiVLlpCens6BAwdo1qwZb775Ji+//DJZWVkeZd2xYwePPvooy5cvp2nTpgwfPpwvv/ySq666KqzyVJXWJJRStVa/fv085gu8/vrrrk/n27dvZ/PmzQCkpqZy+eWXA9C3b1+2bdsGwMCBA7n99tt57733KC8vBxwTBu+++2569uzJdddd5+pfmDVrFnfccQfp6ekANGvWLGDZli1bxqBBg8jIyCAlJYWbb76Z+fPnh12eqtKahFLKxcSwUyLQJ/5YqV+/vuv13LlzmTVrFosWLSI9PZ1Bgwa5ZobXqVPHNXQ0OTmZsrIyAN555x2WLFnC1KlT6du3L8uXL+eNN96gVatWrFq1ioqKCtLS0mwvdzjlad68eZXupTUJpZSLqeFd1w0bNuTw4cM+jxUXF9O0aVPS09PZuHEjixcvDppffn4+/fv3Z/z48WRkZLB9+3aKi4tp3bo1SUlJfPTRR65P9MOGDeOvf/0rx44dA+DAgQMBy9SvXz/mzZvHvn37KC8v5+OPP+aSSy4JuzxVpTUJpVSt0bx5cwYOHEiPHj2oV68erVqdWuknOzubd955h65du9KlSxcGDBgQNL/f/OY3bN68GWMMQ4cOpVevXowbN45rrrmGDz/8kOzsbFdtJTs7m5UrV5KVlUVqaiojR47khRde4Pbbb+eee+6hXr16LFq0yJV369atmTBhAoMHD8YYw6hRoxg9enTY5akqMbGsX8ZAVlaW0U2HlApPp99Oo6zCsPHZbNLqJEftPhs2bKBr165Ry1+FxtfPQUSWu63y7aLNTUqpGt7IpKpCg4RSSim/NEgopXAu+RaL1uea1sRd3YT7/dcgoZSKWXNTWloa+/fv10ARJ879JMIZlqujm5RSLtEeAtuuXTsKCgrC2s9A2cu5M12oNEgopWKmTp06HjOcnTUK3WMicWlzk1LKJdatQB0fm8ZVby0MfqKKGw0SSqlTHddxuPeq7UVxuKsKlQYJpZTOk1B+aZBQSlFe4QgTOupIedMgoZRSyi8NEkopF61HKG8aJJRSLtrapLxpkFBKKeWXBgml1Clak1BeNEgopVxq+s50KnwaJJRSSvmlQUIp5dJ7/Ew27/a9B7SqnTRIKKU8TF2zM95FUAlEg4RStZzOslaBaJBQqpbTGKEC0SChVC1XoVFCBaBBQqlaLvfHgx7vBd0ASJ2iQUKpWm7eD7qVqPJPg4RStVxFRWI0N60tLOb5qeu1Iz3BhBwkRCRZRFaIyNfW+44iskRE8kTkUxFJtdLrWu/zrOOZbnk8ZqVvEpHL3NKzrbQ8EclxS/d5D6WUfeLVJ3H0RJnrdWbOVC5/4zveW7CVktKKuJRH+RZOTeIBYIPb+5eAV4wxnYCDwJ1W+p3AQSv9Fes8RKQbcAPQHcgG3rICTzLwZ2AE0A240To30D2UUjaJR0Viw85DdH9qRuxvHMSGnYf4z6od8S5GQgkpSIhIO2AUMNF6L8AQ4HPrlA+Aq6zXo633WMeHWuePBj4xxpwwxmwF8oB+1leeMWaLMeYk8AkwOsg9lFI2iUdNYsRrC2J+z1CMeG0Bv/x4RbyLkVBCrUm8CjwCOOuBzYEiY4yzvlgAtLVetwW2A1jHi63zXele1/hLD3QPDyIyVkRyRSR3717thFMqHP9Y8lO8i+Bh/c7ieBdBuQkaJETkcmCPMWZ5DMoTEWPMu8aYLGNMVkZGRryLo1S1cqIssfoAJi7YGu8iKDcpIZwzELhSREYCaUAj4DWgiYikWJ/02wGF1vmFQHugQERSgMbAfrd0J/drfKXvD3APpVSUSJynSejgpsQStCZhjHnMGNPOGJOJo+N5jjHmZuBb4FrrtDHAV9brKdZ7rONzjGNM2xTgBmv0U0egM7AUWAZ0tkYypVr3mGJd4+8eSqkaSve0SCxVmSfxKPCQiOTh6D9430p/H2hupT8E5AAYY9YBnwHrgW+Ae40x5VYt4T5gBo7RU59Z5wa6h1KqhkqQaRvKEkpzk4sxZi4w13q9BcfIJO9zSoDr/Fz/PPC8j/RpwDQf6T7voZSqubS5KbHojGularnTGqXFuwge7J5x/fSUdUyYvtHWPBNN/t4jfLI0OqPUNEgoVct1btXA4328l/fbWVxia36TF27jnXn5tuaZaEa+toCcL9ZEJe+wmpuUUjVPoi0Vvn7noajmP++HvfTv2Iy0OsmutCEvz6VeanKAqxJbNIcxa01CqVquIrGmSUTVhp2HGDNpKU9PWeeRvmXfUdbtiG5wqq40SChVy+0/eiLeRaik+HhpVFanLT5eCjiCggqNBgmlaqB5P+zliNsqq4H8sPtIlEsTvl7P/JczfjuNzJypAc8rK69gZ/FxdhWXUFYeuEr0t8U/nnpjHCvPvjlnsx3FjaulWw9ENX8NEkrVMIVFxxkzaSk9IlxldfLCbfYWyD3v/21lyMtzbcvvxekbOf/FOQx4cTbPfr0+4LlPfLmWrVYNwtkP86eZP/g8NzNnKtf/ZZFt5YymaJdTg4RSNcz8Ku40t//oSZtKUtnT/1kfdlPP3sP+m8Omr9npej17456geT0WxgigaH9Cry40SChVw4TzIKwOznt+lt9j+46cCmjhrDnl3tuxZW/iNbclEg0SSqnqyy0wFB0rDfky92G/Q/44L+j5323ex5qCxF/CPBpbv2qQUKqWWpi/j+U/Hox3MWxzuCS0jvpw3PTeYgBueX8JV7z5ne352y0aU150Mp1StdRN7y2J6f0mLtgS8bVzNu5myNmtKqUHa2Ha5Wf29oqfikK678L8/SGdlyhKyspJT7X3sa41CaVU1C3M38dzUzdEfP0v/7GCtYXhN/d8vnx78JNqkPw99s//0CChVDV17GQZmTlTmbJqh99zyhNk3e2Cg8erdP3Rk+Vc/sZ3/OrTlRw/We5K9+6sLiktJxwJ8u1JaBoklKqGDpWUsmyboz/hgU9W+D3vX98XuF7n7TnMX6r5Qnf/XlHIlysdG1SuLSympNRzAt0/lxd4vN+UgBMFoykauwpqn4RS1dDwP81n1yFHe7sxUHTsJE3SUyud99a3eVyf5dgd+Oq3FnK4pIw7BnYkNSW2nw/tfHY5O2cvf6NyR/IJqyYxccEWDpWU8Z8AtaxQ/bD7cJXziJY9h+1dMdcXrUkoVQ05A4RT7/EzfZ4nbh8tnc008d7D2g6bdvl+cDsDyHNTN/D6bHuW3Bj+ynxb8omG5ds8R6dF42erQUKpGmyrj9nNwYZJdm7ZIPAJcWYwfhclrDCG7zbvi3GJ4qdOcvQf4RoklKohgi1w52QIHCUa1atjR3Gi5q1v8zl41PfEOQO8+W31X7QvVEleT3CJwpZRGiSUqiGmrd0V0nnBahLRmLVrp8Ki4zz6r9U+j1UYQ1l5/Mv/9yU/cufkZTG/r3ZcK6X8qqgwIc2gDhokbCpPNPlbBt0YOHYyvGGw0fD4v9fGuwi20ZqEUtVIoE/5SUnCvE3BV0IN1txUnecOGGOivv1pbaM1CaWqgeLjpew+VMLwV+bzx+t6+TzntVk/kL83+IzboK1JNjc37Sou4Tef+24eslt1DnB2aJJuf3+SBgmlqoFez/zX9frX/1zl85xQAgQEb06y+0E74MXZ9mYYQLy7U16esYk3v82L2/2TotApoc1NStUywTqmgzVHJbK9R6I/uSyQeAaIaNEgoVQtE6ymEOtP4306NLEtr4+X1q4F/WJBg4RStU2CBYkHLj3LtrwSZUHDWDDGsCXEJsaq0CChVC1RZj1AF+YHnpG8fuchKmrRwzae9hwqITNnKrnbwt9P+4vvC6u0/HqoNEgoVcu8NTf4SrAlZbGba1CdlpL6aPGPYU02zMyZ6nedKYBFWxybGk1euC3ssqzdUXl/jWjUAjVIKFXLVBgT9EFXFoOaRNsm9QDo2KJ+1O9ll999uZavV+8M65ovVhT4PeZcgDGS73ZyjFZq1CGwStUyxhB0Zna5TUtb7D7kf7TRLQNO57bzT6d+3er1GDp47KRtebke8xF8u4+HucFSpILWJEQkTUSWisgqEVknIs9Y6R1FZImI5InIpyKSaqXXtd7nWccz3fJ6zErfJCKXuaVnW2l5IpLjlu7zHkqpyBmotFmPr3PsMDfIDPDqFiAg/Cadr1ft5HCJ7wUJnZWBUIYdZ+ZM5Ykv17je2xmsAgmluekEMMQY0wvoDWSLyADgJeAVY0wn4CBwp3X+ncBBK/0V6zxEpBtwA9AdyAbeEpFkEUkG/gyMALoBN1rnEuAeSqkIGWPYsi/wjm0VNjVu527zX2MJ1oGeqMJdALGw6Dg5X6zxeSzcVVv/tvgn1+u1hbFZfiRokDAOzt+oOtaXAYYAn1vpHwBXWa9HW++xjg8VR8PbaOATY8wJY8xWIA/oZ33lGWO2GGNOAp8Ao61r/N1DKRUhY+DtIJ3XVY0R/1m1g7WFxZW2E3V3uMT3In2JLpLumh1Fgff4juT7/dOBY+FfFIGQ6nrWp/3lQCccn/rzgSJjjPOnXAC0tV63BbYDGGPKRKQYaG6lL3bL1v2a7V7p/a1r/N3Du3xjgbEAHTp0COW/pFStZTBBl2+o6qzrX37sf99tp5IYtanbLZLvjL/A4mpuMriGHSclJdZ4r5BGNxljyo0xvYF2OD75nx3NQoXLGPOuMSbLGJOVkZER7+IoldCMcTSBBD4p8vxnrd8dcjm8dWiWHvmNYySi/Tb8XOMMBwZDn+dmhrTOVayb6cIaAmuMKQK+Bc4HmoiIsybSDii0XhcC7QGs442B/e7pXtf4S98f4B5KqRD9tN+zWWLznsD9EVC1jmt/GwJ581WZ+dWwzlW4c2xEEiOOeu1xsWDzXvq/MMtjPkrRsVL2HD61LevGXYd87ptxz0fL/ZctCutuhTK6KUNEmliv6wHDgA04gsW11mljgK+s11Os91jH5xhH6J0C3GCNfuoIdAaWAsuAztZIplQcndtTrGv83UMpFaKffxD+DmlV6ZPYfzS0UTfiFiVm//oSlj9xKW0a14v8xjHi60Gcv/cIczb6r0HleQXmW99fyu5DJ8jf41hWY8a6ytdmv7qAHk/NYNzfPYPCoZIyio/5Hi0VDaH0SbQGPrD6JZKAz4wxX4vIeuATEXkOWAG8b53/PvCRiOQBB3A89DHGrBORz4D1QBlwrzGmHEBE7gNmAMnAJGPMOiuvR/3cQykVoh/3h7++j12jmwJxr0icmdEACK2WE2++thIf+sd5IV+fmTPV9Xqbj5/Npl2H+S7vVJPStDW72OM236RN4zTemR981rxdggYJY8xq4Fwf6Vtw9E94p5cA1/nJ63ngeR/p04Bpod5DKRW6a/q0C/uaWKzc5Ku5Kd77QYQikgCaVsd3o42v2duXvTq/Ulq/F071VewoLgk6Os1OuiyHUjXcaY3Twr4mos7ZMPkKEs0bRDZftu/pTatYmtBFsvhh8/p1o1CS2NAgoVQN9+qszWFfE4tP9L6G4Z7VqmFEeT15ebfgJ9kkknkSZRUVlFcYTpYFnumeiKrfnHilqrFNuw7TvEEqLRrE9pPlzuIgQ17jwM7ZADFa6w6A8ggi6O5DJ+j/wiz2HYnNUhp20pqEUjF02avzw+rkBNi82/9S06EKtlaTt1h0XNv5ZI/l9heRNsVVxwABGiSUirni46EPXywpLWfYK5U7MsMV7uzmSHZ4e3XWDywLY/OcUENEcpIwsFPzsMvjrW6KPY877+9NTd+gSYOEUgnspK/xlhEI90P7KxH0Y7w6azPXvbPI7/GLOrcIqUwTftaTs09ryL9+cQGLHhvCsscvDboQXiif7pNtWu7COya8MusHW/K1QzQqgNonoVQCWP7jAfL3HuX6rPYe6camfs7sVxeEdf73QfabiMQ7t/Rl467DXPP2QsB/TeKGfh24oZ/nGmyBglyoAbBxvTocO1n19aK8m+LemJNX5TwTmdYklEoA17y9iEc+r7ycRVlFfEbDRKNPon7dFLq2PjV6SWLZ2wx8OvZ8nrmye5XziaQprjrTIKFUnJSVVwRdQjpWz6Msr3kG0eq4dm82snOx02Cl7dSyAR2apzPmgswq3+v977a6Xn+3uXruiREODRJK2aCktJyjPhZjC+SZ/6zngglzAq7DE4tJbQC/85pnEMptyyuMx3IRoXCvPIS74Q7Aiz/rydyHB3mkZTYPvkf246O6hn2vUNzy/pKo5JtINEgoZYNL/vAt3Z+aEdY1szY4FnU7etJ/cIlVw4b3fdxXI/XnxWkb6PfCbA6EuKBfJRHUJNo0qUdmi1NB4bUbevPp2AFBr2vf9NTCgWe0CB5U1CkaJJSywe5DwR+q3pxNOt4zj40xLMzbx8myitjMVwjBgaMnKfLaU3nOxj3WsRMh13g8axKha9nQsbRIemqyR/qwbq1o2SgtYM2nf8dmdGp5qi8k0KY+d17YMaTy5IYx1Le60yChVJw4R7cmef0VTlywlZsmLuGsJ6bHdJJYIH2enUnv8TM90pxDSssrHEtlh8K9iSmcfuvxo7vzh2vPqdR3Eoo2TTyXH39gqO89K/6XM4TfXd6N1GTHDyTQva59Z1FY812qMw0SSsXJviOO2od3TeL5aRtcrwdOmBOTskTS93EqSBgu/VNoE/4i7ZOoXzeF67LaVxoRdWrug//yj+zZ2uP9Fb3asG3CKC4+y3MXy7ZWMJn3yCC+GHcBH97Zr9LcDncvTN3g91i8ROMzhQYJpeJs2ppTy0Vv3Rf+3g/x4gxun+VuD3LmKe6PeDtGwNZNcTQ/ecc49xalYd1aBc2nV7vGrtetG9ejT4empKemBNxO1X1XuXgac/7pUc1fg4RSMeJv+YYnv1rnem3HOk2x4nzIT164LYxr3IfARm+eRJ3k4I+2UO4eqIgJ0l3Elb3bRjV/DRJKxUBFheGtucFn5sbruePrvsECViTPeDvDgvunf+/yh7sEh7/ve6IEgnjSZTmUioH7P1nhsQuZv2GjK7cXxahEnnw9DFdsL6JzgP0dIpkMblflYe0zl7k6mH0Jt5biPU8kFDGeMB43GiSUigHvbSrv+Wi5z/NiuS1lVa3feSjsa+xaiqNB3cCPrnBv071No7DLMP+HvWFfEw1nnxbZRk2h0uYmpeKgMMhyHLWBsbFxzbsm1Lt9E9vy9udggJnysVS/bgq/v+acqOWvQUKpOPCeGxF/lR/Yj3y+mr8t/tGW3Iee3bLyHaPY3v/C1T0BAjZJhVKWS0MYGVXTJdyvqlI1xcQFW8jMmepzVFMk6xaF47zM8Ced+fLEl2ttyeedW/vakk+o6lkzswPVVto2ref3mNPgLi3Z8sJI28pVHWmQUCpKnrMmW/1x5qZKx+zq9GxeP9Vnm/TFnTNoXK+OPTdx45wAGK5QhqRWhfdkQNcUuwC1lSdD7KwOtIxHoonGgpAaJJSKsi9X7KiUVpU9Ca4+99S4+OW/G8Y3D17s87xwnm3GwD/u7s/wIM0rHy2yp/kJKq/DZJdfDzuL+kE6tgHS6iS7ylDtR7pGMY5pkFAqyo74WEL8RFnkmwmFMgfAEP5IogvObEHX1oFH+bw+ZzMFB4+Fle/sX1/iM32CjZ2t7g/5Xw7t7KqpBXv4D+3qCIop1ai2EGsaJJSKMl8LwSVHeZD94C4teTS7S9jXBXuoGgNjP/Q9fNdbnw5NOKNFfc7MaODzeIsGdcMsnX/eo5lC7fN5+bpz+O7RwaTViU6tpibQIKFUHOwKc7Med76and+86VxyRpztet+zXWOu6NUm9DwDHCst96z1lJSGtmbRF+MGMsdrg6Bo8X7IhxqD66Yk066p//WZYiGcZrfM5rEvqwYJpaqRVU8O9zli5/Jz2nDPJWd6pNk1gurpKes83h8qKWPEawsCXnNV79ADVDSc6rhO/N6GZvVTQz43Z0RX/nPfhVEsTWUaJJSqRhqnhz5iKZzJas5n6XV921U6Ns9rZvG+IyfYEGS2dWpKfB8tzmU5LjjT/1Lf0dY0xJ9VOHEsu8dp9HRbsyoWNEgopVza+1gau+Bg+LPDAw2/TauTxP1DOoWdZziSkoSZv7qYv8R4foa7r+69MOjyIdVB9f8fKBUH/15RwKrtxTx9ZffY3zzAJ8+5Dw+izBpeG+5KqHb69XD/neYbnx0RkzIEWpww2p66ohsdmqfzi0Fn8ocZlefJuAulT+L+IZ0Y3v001/surRqyyccqvdFoXQtakxCR9iLyrYisF5F1IvKAld5MRGaKyGbr36ZWuojI6yKSJyKrRaSPW15jrPM3i8gYt/S+IrLGuuZ1scbu+buHUvH2q09XhbWPQqxktqhPp5aO0UTODXlCYXfbfTxGC01/4CK+vHdgzO/r7fJzWnPHQMde2RkN/Y/gcq639NCws4Lm+dDwLvRo67+ZKZofB0JpbioDfm2M6QYMAO4VkW5ADjDbGNMZmG29BxgBdLa+xgJvg+OBDzwF9Af6AU+5PfTfBu52uy7bSvd3D6US1kW/n8NDn66MdzFCEu++Azt1bd0oJgv7hePaPpX7eAAWPTaE689rz6LHhjCiZ2uG+FjbyslXTeOOgZl2FTGooL8hxpidxpjvrdeHgQ1AW2A08IF12gfAVdbr0cCHxmEx0EREWgOXATONMQeMMQeBmUC2dayRMWaxcXyc+dArL1/3UCphbT9wnC9WFEYtf7s+8/fp0ITe7ZrYlJty8th9z63Jb+zFZ7het25cz+PfegGanHzVjq7Lal/lcoYqrI8RIpIJnAssAVoZY5yL5O8CnPP52wLum94WWGmB0gt8pBPgHt7lGisiuSKSu3dvYqzxrlSiG9Tl1KfXxB8oWv3deWFHnhjVlUZplbuCnxvdw+c1U++/kLN89K3Esrcp5CAhIg2AfwEPGmM8xr9ZNYCo/p4Fuocx5l1jTJYxJisjIyOaxVAq7rJsWuHVGFxPm2ownaDaMwbuuugMVj99WaVjTeunsvHZbBY8Mph3bnF149K9TWyHu/oSUpAQkTo4AsTfjTFfWMm7raYirH/3WOmFgHtdqJ2VFii9nY/0QPdQqtapa/Uf3NSvQ5xLogK5dcDpHu+dndfBBpul1UmmfbN0snu05q93nMczAUbOxXLr1FBGNwnwPrDBGPMnt0NTAOcIpTHAV27pt1mjnAYAxVaT0QxguIg0tTqshwMzrGOHRGSAda/bvPLydQ+lap2WjRwPm6psAfrNgxe5XteWPZrtsOixIcz/zWDuH9o54Hm3Djidfh2beaQ59xMJ5+c2uEtLxlyQ6fe4XdvAhiKUeRIDgVuBNSKy0kr7LTAB+ExE7gR+BK63jk0DRgJ5wDHgDgBjzAEReRZYZp033hhzwHo9DpgM1AOmW18EuIdStcb83wxmydb9XNCp6rOHU5OT+OWQTrwxJw9wW77Cptbifh2bMbLHacFPrGacHcztAmxU9PtrzuG6rMqjmSpM/OetVEXQIGGM+Q7//SRDfZxvgHv95DUJmOQjPReo1HNjjNnv6x5K1Satm6RFbTSL3R9IB3Rsxu3WHIGaKCnAN+z683z/jDo0r8/BY0XUSbY/SIw6pzUQ3ZqFzrhWKsHZ/ed/YacWvDEnj/PPbM6lXVvxxJdrOLe9zlMNhb+fxROjuvq9ZtKYLFb8VETDNHt3Clzxu2E08DFSym4aJJRKQJecleFaWM/uT4n9z2hO3vMjSLG2FP1iXPxnKVcX/n4Ud110hu8DQPMGdbk0yI5/kWgaxuqxVVFzplsqFSeh7q/gS/4LI32mu68g6uu5lJwkXO+j/dvbfYM7eeyB7Qw4KVHac7plo7So5JsofC2AWNNpkFCqik6WR7YV6V0XdiQ5SXjnlr6MH+17uOMr/9fLY9auU/4LI/n9tb2C3uPhy7rwzYMXx2SzmtdvPLfGD889L7MZX/8ytvs5xJsGCaVi5OdeHbpX93EsLJDd4zRuOz/TY3hlW2sUTZN6sWlSsEN299N8BrSaJtBCezWRBgmlqijUx+KTV3TzeO89JLKl24qhDww9i7du7sOgLrFfQeC1G3pHdF11HeJZVf/vEv/9ETWBdlwrVUWRzjAINJwyNSWJkT1bR5hz1QwOsCJpILU1SLRt4n/uRE2gNQmlbFR8vLTKedwYhXb9cAJZ7XzURy6Rvl9x2XRIKRWY+0Ni2dYDfs/zlqiL6kUy5HbS7VlRKIkKVTQDlTY3KWWjcJ773kthdLGGqp7boYl9BYpAuA+cbx8eRMcW9aNSlmqhhi+CpTUJpWLgvsGdAM+mpBSvNvzzMpux4JHBXNc3+PyHSIXyOAvnmXfrgNNrdYDo2roRo+LUdxQrGiSUCtPxk+FPnnN2Bndv08iVdmZGg0rntW+WHtMVPn3x16H++MjKS09k18DF/MIx/YGLaBajmc/xokFCqTD9YcYmj/fujUb/zN2OL94Df27q3yHuwSBcd19ceahn51aVA52qWbRPQqkwHTx20uN9admpGdf/Xb/b5zXen84TtdMaQm9u2jZhVHQLohKC1iSUCpP7Q7To2Enu/CA36DVlFY5Acpq1tlGHBF4DKJEDmIo9rUkoFSZx6/7tPX5mSNd8/2MRfU9vxtCuLfng5/240IYNhMIRzoM/0CQ/ldjs2jzKndYklApTJBOLy9y2sLzkrIyEnp2cmpLEWB/9DypxRTOua01CqTDVS00O+5o2TarXEtq/HdmVS87K4OaJSzzS3765DyLCGRm1d9hrbaNBQqkw1YlgL4Y+HRJj57dwPnEO9NEkNqKGzwlQlWlzk1IxoM38qrrSIKFUmFZtL4p3EWLGOVFs3KAz41wSFS/a3KRUGL7bvI/cHw/Guxhha1DX8ace7silhTlDWLxlf8xHY6nEoUFCqTDc8v6S4Cf5EO+5BxPHZPH16h1h79GcVieZQV0i219C1QwaJJQK0drC4pDPffDSziSJ8Omy7RQWHY9iqULTpkk9xl6sTUY1ne4noVQc5e89EvK5F3Zq4bFntXZcq2iK5u+XBgml3OwsPk55he+PYyfd1mgKxpnD+7dncVP/DrRpXLO3uFQ1lwYJpSx7DpVw/otz+P2Mja60L74v4Okp6wAoLQ+/Ln/2aY144eqeJCXwDGsVvo/u7BeVbWYTkfZJKGXZd8Sxuuu8TXt5bIRj74SHPlsFwOSF20gPY6Z1vDuqVXRd1DmDizpnxLsYMaE1CaUsznbdQ8dL+czHvhDHwthsqEfbRsFPUqoa0JqEUhZnkNhRXMIjn69m+4FjEeXzm8u6kJ6qf1qqZtCahFJ+vDEnL95FUCruggYJEZkkIntEZK1bWjMRmSkim61/m1rpIiKvi0ieiKwWkT5u14yxzt8sImPc0vuKyBrrmtfF2tPR3z2UipYpK3fEuwhKJZxQahKTgWyvtBxgtjGmMzDbeg8wAuhsfY0F3gbHAx94CugP9AOecnvovw3c7XZddpB7KBUVb83NtyWfuilaQVfxEY3xEkF/m40x84EDXsmjgQ+s1x8AV7mlf2gcFgNNRKQ1cBkw0xhzwBhzEJgJZFvHGhljFhtjDPChV16+7qGU7VbasGhf39Mdn3tuPf/0KuelVDjcd0u0W6S9a62MMTut17uAVtbrtoD7sJACKy1QeoGP9ED3ULXU/iMn+HLlDn4+MBOxcYrplysKefDTlVXO51+/uKDqhVEqwVS5XmzVAKI6KjzYPURkrIjkikju3r17o1kUFUcP/3MVz369nnU7DtmW5/4jJ2wJEErVVJEGid1WUxHWv3us9EKgvdt57ay0QOntfKQHukclxph3jTFZxpisjIzaMcGlNtpZXAJAaXnoy2MEsqagmL7PzYro2peu6enx/tKuWtFVNVOkQWIK4ByhNAb4yi39NmuU0wCg2GoymgEMF5GmVof1cGCGdeyQiAywRjXd5pWXr3uoWmrjrsOAfdXWK978LuJr/+88zyUZJngFDaVqilCGwH4MLAK6iEiBiNwJTACGichm4FLrPcA0YAuQB7wHjAMwxhwAngWWWV/jrTSscyZa1+QD0610f/dQtVyiLHnx7cODXK9bNKgbv4IoFUVBO66NMTf6OTTUx7kGuNdPPpOAST7Sc4EePtL3+7qHUlWx53AJV/95IR/e2a/KeXVsUZ83bjyXrfuO2lAypRKTrh2gqp3jXmsovT03n/KKCu4b0tnPFafMWLuLwqLjTPpua8j3G3VOa6au3unz2BW92oScj1LRZqJQzdYgoaqdW95fwsZns0mr41iV9aVvHEt7BwoS5RWGCmNYtGU/AH9f8lPQ+7x+47k0SkthUJeWXNV7N2XlFXywaBvrbRxdpZQdornpkAYJVS29MWczv7nsbPYcKvFIP3ayjOQkoW6K57LeN09czOIt3nNCA7vSrZYwrJtj9NKInq0jLLFS1ZOuH6CqpUPHywBY47bvdEWFoduTM+jyxDfM+8Fzvkw4AaJRWgqTbs+yp6BKVXMaJFRMLd16gC1ue0Wv21EcUTtqhXXNr9wmwj07db3r9ZhJS/1uQxrM6qcvY8jZOu9BKdAgoWLs+r8sYsgf5wEwd9MeRr3+HZ8u89zgZ8/hEl6ctiFg8Phx/zEyc6ZyqKTMlfbX/23zOOfM304D4K4Pcm0qvVK1j/ZJqLj4cf9R1u90dABv2OnZEdzv+dkADOzUggs7tSApSSgp9RzR9F3evpDuk5kzNeg5n99zPh8t/pGvdKlwpSrRIKFiZtqaU8NIL/nDXNdr597S3m6btBSAbRNGVapt2CU9NZmszGYhBx2lahttblIxM+7v3/tMn7rG9xwEpzsnL+OpKeuiUSQeHt4FgPsGd4pK/krFUlz2k1DK3ZyNu/ni+4LgJ3r5Zu2uiO85e6PftR2rrGNGfQBSkvVPQSlftLlJheXnkx2dwD/r0y7ImZ7u+dvyaBSnSqbefyHd2zSOdzGUSmj68UlFXSi7vvV9diYAZTYtAx7Mv8ddUClAXNGrDX+49pyY3F+p6kJrEirqrvrz/4Kes/+oo/P6mNcopmhwX9LD3Rs3nhv1eytV3WhNQkWspLSczJyp/PnbPJ/Hj5wo4y/z8kPO7515+Rw86nukUyD5L4wM+dxtE0b5DBBKKd80SKiIHbNWY317ru9A0OOpGbw4fWPI+U2YvtFjBnUoLurcguQkYeuLwQPFgkcGh5W3Ukqbm1QVLNvmWA/pyImyIGeG7vufisI6/15r6Kq4LYN59bltEYEvvi90pW2bMMqW8ilV22iQUCGb7jafwRjD3sMnKp1TWl5BSpJ4PLTdNaufyoEImpScHhtxtqt28sDQzgw4o7nrWO4Tl/K/vH2M7t0WgHGDOnF683Tq6PBWpSKmfz0qZO7LZ5woq2BR/n7X+7Ef5lJSWk7nx6fzp5k/+M2jbkrVfuX+3yVn8n9Z7QG4opfnst0tGtR1BQiATi0baIBQtUKfDk155f96kdHQ/m109S9Ihcx9NufJ8gqPmdL/Xb+b4uOlALwxJ4/r31lU6fore7XhZJljiOvCnCH0Pb0pZ1iT2XxZ8/RwAFKSPGslz4zuzj/u6k+nlg0j/a8oVaO0b5bO1ee2o1FaHdvz1iChQua+KKuv3dn6vzDb9XrpNs/9Gz6+ewCv33guJ615EHVTkvjXLy5g5q8u4ReDzvR5v4ZpdVj3zGVsfDbbIz2tTjIXdGoR6X9DKRUGDRIqJPl7j/Cm21DXG95dHNb155/p6Dtw1iRSrWan5CRhRI/TKp2/8slhANSvm+JaMqNFA/ur0kqpwDRIqJDM99rpLVJPjOpKanISDeqeGjNxTrsmXOhWM/j1sLNokp7qcd3S3w5lzsOX2FIGpVToNEiokCQn2bPT+q3nZ/LD8yMqjX66tu+ptaDuG1J5RdaWjdKi0t6qlApMh8CqkDw3dUPE1z55ebeg54zu3YaDx04ysFMLv8NnlVKxp0FChcTZl+DtZ33acvW5bbn1/aU+j696cjiN04PXAESEOwZ2rFIZlVL20yChquRP1/fmaIAZ16EECKVU4tI+CRWxX1p9B/Xr6mcNpWoq/etWEXto2Fmu1z88N4IFm/dy5weOTYnm/2Ywe49UXrZDKVW9aE2ihqqoMExbs5PyCvt3vT37tIZsmzDKo4M5NSWJoV1b0bBuCilJQofm6fQ9vant91ZKxZbWJGqol/+7ibfm5pMksOXFqq2AmrfniGfe1/Xye+7Sxy+t0r2UUolFaxLVxMrtRVz40hwOlZSGdP5b1h4PFQYyc6Zy/ouz+f03oe3t8O2mPczesBuA8gpDzr9Wu449Maor3ds08nttvdRk6qXqpj5K1RRak0ggRcdOUi81mboplR+yzi1A523ayxW92lQ6XlFhSLImvPkKJDuLS3hrbr4reLxx47k0r5/K6sJiJkzfyOXntObr1TsrXedu6v0XVtoXWilVsyV8kBCRbOA1IBmYaIyZEOci2cJYq+W5t+v3Hj+Ti8/K4MOf9/N73ZrC4kpBYsVPB7n6rYX8+aY+vD0vj7WFjsX3zj+jOVv3HWXXoZJK+fzy4xUe74MFCIAurXTVVaVqm4QOEiKSDPwZGAYUAMtEZIoxZn18S1Y1x0+W0/XJb3g0+2zXCqil1uqo83/Yy97DJ8hoWJeKCsPx0nKPPRHenb+F2y/IZPLCbUxcsAX3ful7//G9x33Gj+5O51YNKSktZ9+RE+TtOcITX66l4OBxv2Vr26QeN/XvwD+W/ERh0XHOPq0hD156FpecleFaaE8pVXuIMfaPfrGLiJwPPG2Mucx6/xiAMeZFf9dkZWWZ3NzcsO/10jcbWfHTQRz5W3snWN8ag3GlBft+VRioMMbxVQEijvzKKwwijprDgaMn2H3IMTzU+em8tLyCLfuOuvJp2bAue3zs/Baqdc9cpvMXlFIhE5Hlxpgs7/REf4q0Bba7vS8A+nufJCJjgbEAHTp0iOhGFRWOhzpW648AkgTieGU94B3vfS0t5IwdSUlCkkCynDrPGEhJFoxxBJF2Tesxc/1uurVuRLum9RCBlKQkio+Xsv/oSS7r3oqm6ams3F7Exl2HaZpehzMyGrD8x4Nc1LkFi7fsp3XjepyZUZ/7h3ame5vGiEBJaTkNdRE8pZSNEj1IhMQY8y7wLjhqEpHk8djIrraWKR50q06llN0S/alSCLR3e9/OSlNKKRUDiR4klgGdRaSjiKQCNwBT4lwmpZSqNRK6uckYUyYi9wEzcAyBnWSMWRfnYimlVK2R0EECwBgzDZgW73IopVRtlOjNTUoppeJIg4RSSim/NEgopZTyS4OEUkopvxJ6WY5IiMhe4Md4l8NNC2BfvAsRIi1rdGhZo0PLaq/TjTEZ3ok1LkgkGhHJ9bUeSiLSskaHljU6tKyxoc1NSiml/NIgoZRSyi8NEtH3brwLEAYta3RoWaNDyxoD2iehlFLKL61JKKWU8kuDhFJKKb80SERARNqLyLcisl5E1onIA1Z6MxGZKSKbrX+bWulni8giETkhIg975ZUtIptEJE9EchKxnP7yScSyuuWXLCIrROTrRC6riDQRkc9FZKOIbLC27E3Usv7KymOtiHwsImlxLuvNIrJaRNaIyEIR6eWWV9T+ruwsayz+tqrMGKNfYX4BrYE+1uuGwA9AN+D3QI6VngO8ZL1uCZwHPA887JZPMpAPnAGkAquAbglYTp/5JOL31C2/h4B/AF8n6s/fOvYBcJf1OhVokohlxbGV8FagnvX+M+D2OJf1AqCp9XoEsCQWf1c2lzXqf1tV/r/GuwA14Qv4ChgGbAJau/3wN3md97TXH975wAy3948BjyVaOf3lk4jfUyutHTAbGEIUgoSNP//G1oNXqsHvqnO/+WY4thj4GhieCGW10psChdbrmP5dVaWs/vKJ1e9DKF/a3FRFIpIJnAssAVoZY3Zah3YBrYJc7vzDcyqw0mxXxXL6yycqbCjrq8AjQEU0yueuimXtCOwF/mo1jU0UkfqJWFZjTCHwMvATsBMoNsb8N4HKeicw3Xods78rqHJZ/eWTMDRIVIGINAD+BTxojDnkfsw4PhYkxPhiu8oZKB+7VLWsInI5sMcYszwa5fO6V1W/rylAH+BtY8y5wFEcTRQJV1arbX00jsDWBqgvIrckQllFZDCOB++j0ShPIHaVNRZ/W5HSIBEhEamD44f6d2PMF1bybhFpbR1vDewJkk0h0N7tfTsrLdHK6S8fW9lU1oHAlSKyDfgEGCIif0vQshYABcYY5yfHz3EEjUQs66XAVmPMXmNMKfAFjnb2uJZVRM4BJgKjjTH7reSo/13ZWNaY/G1VhQaJCIiIAO8DG4wxf3I7NAUYY70eg6N9MZBlQGcR6SgiqcANVh4JVc4A+djGrrIaYx4zxrQzxmTi+H7OMcbY+onXxrLuAraLSBcraSiwPhHLiqOZaYCIpFt5DgU2xLOsItIBR7C61Rjzg9v5Uf27srOssfjbqrJ4d4pUxy/gQhzVyNXASutrJNAcR4fpZmAW0Mw6/zQcnxoPAUXW60bWsZE4RjTkA48nYjn95ZOIZfXKcxDRGd1k58+/N5Br5fUl1giYBC3rM8BGYC3wEVA3zmWdCBx0OzfXLa+o/V3ZWVZ/+dhd3qp86bIcSiml/NLmJqWUUn5pkFBKKeWXBgmllFJ+aZBQSinllwYJpZRSfmmQUEop5ZcGCVXriWO57nEJUI65IpIV5JwHRSQ9VmVSSoOEUtAEqBQkRCQl9kUJ6kFAg4SKGQ0SSsEE4EwRWSkiy0RkgYhMwVoiQ0S+FJHl1qYwY50XicgREXleRFaJyGIRaWWlXyeOjXlWich8fzcVkXoi8ok4Nhv6N1DP7djbIpJr3fMZK+1+HIvrfSsi31ppw8WxSdD3IvJPa6E4pWyjM65VrWct0fy1MaaHiAwCpgI9jDFbrePNjDEHRKQejnWBLjHG7BcRA1xpjPmPiPweOGSMeU5E1gDZxphCEWlijCnyc9+HrPv83Fr87XtggDEm1+2eyTiWebjfGLPaWrgwyxizT0Ra4FgPaIQx5qiIPIpjqYzx0fpeqdpHaxJKVbbUGSAs94vIKmAxjtVFO1vpJ3FsvgOwHMi0Xv8PmCwid+PYJc2fi4G/ARhjVuNYv8fpehH5HlgBdMex65m3AVb6/0RkJY4F5U4P4f+nVMgSsc1VqXg76nxh1SwuBc43xhwTkbmAc2/nUnOqKl6O9fdkjLlHRPoDo4DlItLXuC0NHYyIdAQeBs4zxhwUkclu9/Q4FZhpjLkxjP+bUmHRmoRScBjH/sK+NAYOWgHibByf3gMSkTONMUuMMU/i2HmuvZ9T5wM3Wdf0AM6x0hvhCFTFVj/HCD9lXQwMFJFOVh71ReSsYOVTKhxak1C1ntW/8D8RWQscB3a7Hf4GuEdENuDYv3hxCFn+QUQ64/ikPxtY5ee8t3FsXboBx94My63yrBKRFTiW5d6Oo/nK6V3gGxHZYYwZLCK3Ax+LSF3r+BM4lshWyhbaca2UUsovbW5SSinllzY3KRVlInIZ8JJX8lZjzNXxKI9S4dDmJqWUUn5pc5NSSim/NEgopZTyS4OEUkopvzRIKKWU8uv/A8F2K4f1KHhRAAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","source":["As you can see, common table expressions (CTEs) let you shift a lot of your data cleaning into SQL. **That's an especially good thing in the case of BigQuery, because it is vastly faster than doing the work in Pandas.**"],"metadata":{"id":"PZq7HuBsLQye"}},{"cell_type":"markdown","source":["### Joining data"],"metadata":{"id":"qbVO4_UU0fTP"}},{"cell_type":"markdown","source":["When our data lives across different tables, how do we analyze it? By\n","JOINing the tables together. A `JOIN` combines rows in the left table with\n","corresponding rows in the right table, where the meaning of “corresponding” is based on how we specify the join.\n","\n","GitHub is the most popular place to collaborate on software projects. A GitHub **repository** (or **repo**) is a collection of files associated with a specific project. Most repos on GitHub are shared under a specific legal license, which determines the legal restrictions on how they are used.  **For our example, we're going to look at how many different files have been released under each license.** \n","\n","We'll work with two tables in the database.  The first table is the `licenses` table, which provides the name of each GitHub repo (in the `repo_name` column) and its corresponding license.  Here's a view of the first five rows."],"metadata":{"id":"xkwgZ5D_NDae"}},{"cell_type":"code","source":["# Construct a reference to the \"github_repos\" dataset\n","dataset_ref = client.dataset(\"github_repos\", project=\"bigquery-public-data\")\n","\n","# API request - fetch the dataset\n","dataset = client.get_dataset(dataset_ref)\n","\n","# Construct a reference to the \"licenses\" table\n","licenses_ref = dataset_ref.table(\"licenses\")\n","\n","# API request - fetch the table\n","licenses_table = client.get_table(licenses_ref)\n","\n","# Preview the first five lines of the \"licenses\" table\n","client.list_rows(licenses_table, max_results=5).to_dataframe()"],"metadata":{"id":"TYTZKEmlLUTY","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"0d8e7ac5-7733-4868-c626-ea1b623b3510","executionInfo":{"status":"ok","timestamp":1677904979614,"user_tz":-480,"elapsed":1014,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":32,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                                           repo_name       license\n","0             autarch/Dist-Zilla-Plugin-Test-TidyAll  artistic-2.0\n","1                           thundergnat/Prime-Factor  artistic-2.0\n","2                       kusha-b-k/Turabian_Engin_Fan  artistic-2.0\n","3  onlinepremiumoutlet/onlinepremiumoutlet.github.io  artistic-2.0\n","4                       huangyuanlove/LiaoBa_Service  artistic-2.0"],"text/html":["\n","  <div id=\"df-6a859134-c8ae-4f4d-9706-e650f540dbdd\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>repo_name</th>\n","      <th>license</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>autarch/Dist-Zilla-Plugin-Test-TidyAll</td>\n","      <td>artistic-2.0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>thundergnat/Prime-Factor</td>\n","      <td>artistic-2.0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>kusha-b-k/Turabian_Engin_Fan</td>\n","      <td>artistic-2.0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>onlinepremiumoutlet/onlinepremiumoutlet.github.io</td>\n","      <td>artistic-2.0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>huangyuanlove/LiaoBa_Service</td>\n","      <td>artistic-2.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6a859134-c8ae-4f4d-9706-e650f540dbdd')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-6a859134-c8ae-4f4d-9706-e650f540dbdd button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-6a859134-c8ae-4f4d-9706-e650f540dbdd');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":32}]},{"cell_type":"markdown","source":["The second table is the `sample_files` table, which provides, among other information, the GitHub repo that each file belongs to (in the `repo_name` column).  The first several rows of this table are printed below."],"metadata":{"id":"1Xxrz8BxNMsF"}},{"cell_type":"code","source":["# Construct a reference to the \"sample_files\" table\n","files_ref = dataset_ref.table(\"sample_files\")\n","\n","# API request - fetch the table\n","files_table = client.get_table(files_ref)\n","\n","# Preview the first five lines of the \"sample_files\" table\n","client.list_rows(files_table, max_results=5).to_dataframe()"],"metadata":{"id":"gzyzOy7DNPti","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"44ff3cc0-15f3-41dc-95f0-a02ad50647b1","executionInfo":{"status":"ok","timestamp":1677904980085,"user_tz":-480,"elapsed":473,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":33,"outputs":[{"output_type":"execute_result","data":{"text/plain":["  repo_name                ref  \\\n","0   EOL/eol  refs/heads/master   \n","1   np/ling  refs/heads/master   \n","2   np/ling  refs/heads/master   \n","3   np/ling  refs/heads/master   \n","4   np/ling  refs/heads/master   \n","\n","                                                path   mode  \\\n","0                           generate/vendor/railties  40960   \n","1  tests/success/merger_seq_inferred.t/merger_seq...  40960   \n","2                       fixtures/sequence/lettype.ll  40960   \n","3               fixtures/failure/wrong_order_seq3.ll  40960   \n","4                             issues/sequence/keep.t  40960   \n","\n","                                         id  \\\n","0  0338c33fb3fda57db9e812ac7de969317cad4959   \n","1  dd4bb3d5ecabe5044d3fa5a36e0a9bf7ca878209   \n","2  8fdf536def2633116d65b92b3b9257bcf06e3e45   \n","3  c2509ae1196c4bb79d7e60a3d679488ca4a753e9   \n","4  5721de3488fb32745dfc11ec482e5dd0331fecaf   \n","\n","                                 symlink_target  \n","0             /usr/share/rails-ruby1.8/railties  \n","1  ../../../fixtures/all/merger_seq_inferred.ll  \n","2                             ../all/lettype.ll  \n","3                    ../all/wrong_order_seq3.ll  \n","4                                     ../keep.t  "],"text/html":["\n","  <div id=\"df-22f56145-b4ee-44f4-bdd1-fa3f106c3e5b\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>repo_name</th>\n","      <th>ref</th>\n","      <th>path</th>\n","      <th>mode</th>\n","      <th>id</th>\n","      <th>symlink_target</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>EOL/eol</td>\n","      <td>refs/heads/master</td>\n","      <td>generate/vendor/railties</td>\n","      <td>40960</td>\n","      <td>0338c33fb3fda57db9e812ac7de969317cad4959</td>\n","      <td>/usr/share/rails-ruby1.8/railties</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>np/ling</td>\n","      <td>refs/heads/master</td>\n","      <td>tests/success/merger_seq_inferred.t/merger_seq...</td>\n","      <td>40960</td>\n","      <td>dd4bb3d5ecabe5044d3fa5a36e0a9bf7ca878209</td>\n","      <td>../../../fixtures/all/merger_seq_inferred.ll</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>np/ling</td>\n","      <td>refs/heads/master</td>\n","      <td>fixtures/sequence/lettype.ll</td>\n","      <td>40960</td>\n","      <td>8fdf536def2633116d65b92b3b9257bcf06e3e45</td>\n","      <td>../all/lettype.ll</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>np/ling</td>\n","      <td>refs/heads/master</td>\n","      <td>fixtures/failure/wrong_order_seq3.ll</td>\n","      <td>40960</td>\n","      <td>c2509ae1196c4bb79d7e60a3d679488ca4a753e9</td>\n","      <td>../all/wrong_order_seq3.ll</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>np/ling</td>\n","      <td>refs/heads/master</td>\n","      <td>issues/sequence/keep.t</td>\n","      <td>40960</td>\n","      <td>5721de3488fb32745dfc11ec482e5dd0331fecaf</td>\n","      <td>../keep.t</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-22f56145-b4ee-44f4-bdd1-fa3f106c3e5b')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-22f56145-b4ee-44f4-bdd1-fa3f106c3e5b button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-22f56145-b4ee-44f4-bdd1-fa3f106c3e5b');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":33}]},{"cell_type":"markdown","source":["Next, we write a query that uses information in both tables to determine how many files are released in each license."],"metadata":{"id":"SIb2OHmnNV0H"}},{"cell_type":"code","source":["# Query to determine the number of files per license, sorted by number of files\n","query = \"\"\"\n","        SELECT L.license, COUNT(1) AS number_of_files\n","        FROM `bigquery-public-data.github_repos.sample_files` AS sf\n","        INNER JOIN `bigquery-public-data.github_repos.licenses` AS L \n","            ON sf.repo_name = L.repo_name\n","        GROUP BY L.license\n","        ORDER BY number_of_files DESC\n","        \"\"\"\n","\n","# Set up the query (cancel the query if it would use too much of \n","# your quota, with the limit set to 10 GB)\n","safe_config = bigquery.QueryJobConfig(maximum_bytes_billed=10**10)\n","query_job = client.query(query, job_config=safe_config)\n","\n","# API request - run the query, and convert the results to a pandas DataFrame\n","file_count_by_license = query_job.to_dataframe()"],"metadata":{"id":"3WWRJmdWNYR4","executionInfo":{"status":"ok","timestamp":1677904984960,"user_tz":-480,"elapsed":4877,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":34,"outputs":[]},{"cell_type":"markdown","source":["It's a big query, and so we'll investigate each piece separately.\n","\n","![](https://i.imgur.com/QeufD01.png)\n","    \n","We'll begin with the **JOIN** (highlighted in blue above).  This specifies the sources of data and how to join them. We use **ON** to specify that we combine the tables by matching the values in the `repo_name` columns in the tables.\n","\n","Next, we'll talk about **SELECT** and **GROUP BY** (highlighted in yellow).  The **GROUP BY** breaks the data into a different group for each license, before we **COUNT** the number of rows in the `sample_files` table that corresponds to each license.  (Remember that you can count the number of rows with `COUNT(1)`.) \n","\n","Finally, the **ORDER BY** (highlighted in purple) sorts the results so that licenses with more files appear first.\n","\n","It was a big query, but it gave us a nice table summarizing how many files have been committed under each license:  "],"metadata":{"id":"WA0TW4FpNdQ9"}},{"cell_type":"code","source":["# Print the DataFrame\n","file_count_by_license"],"metadata":{"id":"4hGOBl6rNfve","colab":{"base_uri":"https://localhost:8080/","height":520},"outputId":"afee0fb3-3f66-4509-9e2e-35b69ceaef07","executionInfo":{"status":"ok","timestamp":1677904985447,"user_tz":-480,"elapsed":4,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":35,"outputs":[{"output_type":"execute_result","data":{"text/plain":["         license  number_of_files\n","0            mit         20560894\n","1        gpl-2.0         16608922\n","2     apache-2.0          7201141\n","3        gpl-3.0          5107676\n","4   bsd-3-clause          3465437\n","5       agpl-3.0          1372100\n","6       lgpl-2.1           799664\n","7   bsd-2-clause           692357\n","8       lgpl-3.0           582277\n","9        mpl-2.0           457000\n","10       cc0-1.0           449149\n","11       epl-1.0           322255\n","12     unlicense           208602\n","13  artistic-2.0           147391\n","14           isc           118332"],"text/html":["\n","  <div id=\"df-5499ecef-d984-449c-852d-651da9cbe573\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>license</th>\n","      <th>number_of_files</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>mit</td>\n","      <td>20560894</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>gpl-2.0</td>\n","      <td>16608922</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>apache-2.0</td>\n","      <td>7201141</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>gpl-3.0</td>\n","      <td>5107676</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>bsd-3-clause</td>\n","      <td>3465437</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>agpl-3.0</td>\n","      <td>1372100</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>lgpl-2.1</td>\n","      <td>799664</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>bsd-2-clause</td>\n","      <td>692357</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>lgpl-3.0</td>\n","      <td>582277</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>mpl-2.0</td>\n","      <td>457000</td>\n","    </tr>\n","    <tr>\n","      <th>10</th>\n","      <td>cc0-1.0</td>\n","      <td>449149</td>\n","    </tr>\n","    <tr>\n","      <th>11</th>\n","      <td>epl-1.0</td>\n","      <td>322255</td>\n","    </tr>\n","    <tr>\n","      <th>12</th>\n","      <td>unlicense</td>\n","      <td>208602</td>\n","    </tr>\n","    <tr>\n","      <th>13</th>\n","      <td>artistic-2.0</td>\n","      <td>147391</td>\n","    </tr>\n","    <tr>\n","      <th>14</th>\n","      <td>isc</td>\n","      <td>118332</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5499ecef-d984-449c-852d-651da9cbe573')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-5499ecef-d984-449c-852d-651da9cbe573 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-5499ecef-d984-449c-852d-651da9cbe573');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":35}]},{"cell_type":"markdown","source":["There are a few more types of JOIN, along with how to use UNIONs to pull information from multiple tables. We'll work with the [Hacker News](https://www.kaggle.com/hacker-news/hacker-news) dataset. We begin by reviewing the first several rows of the `comments` table."],"metadata":{"id":"bSlj8eo5PZ85"}},{"cell_type":"code","source":["# Construct a reference to the \"hacker_news\" dataset\n","dataset_ref = client.dataset(\"hacker_news\", project=\"bigquery-public-data\")\n","\n","# API request - fetch the dataset\n","dataset = client.get_dataset(dataset_ref)\n","\n","# Construct a reference to the \"comments\" table\n","table_ref = dataset_ref.table(\"comments\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)\n","\n","# Preview the first five lines of the table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"id":"ZC-UGEr3Paq0","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"6ce34c40-23bf-4ee8-8ae0-024e54d06be6","executionInfo":{"status":"ok","timestamp":1677904985901,"user_tz":-480,"elapsed":457,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":36,"outputs":[{"output_type":"execute_result","data":{"text/plain":["        id    by author        time                   time_ts  text   parent  \\\n","0  9734136  None   None  1434565400 2015-06-17 18:23:20+00:00  None  9733698   \n","1  4921158  None   None  1355496966 2012-12-14 14:56:06+00:00  None  4921100   \n","2  7500568  None   None  1396261158 2014-03-31 10:19:18+00:00  None  7499385   \n","3  8909635  None   None  1421627275 2015-01-19 00:27:55+00:00  None  8901135   \n","4  9256463  None   None  1427204705 2015-03-24 13:45:05+00:00  None  9256346   \n","\n","   deleted  dead  ranking  \n","0     True  <NA>        0  \n","1     True  <NA>        0  \n","2     True  <NA>        0  \n","3     True  <NA>        0  \n","4     True  <NA>        0  "],"text/html":["\n","  <div id=\"df-797c191c-3d70-43b0-92ba-ba2076521507\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>by</th>\n","      <th>author</th>\n","      <th>time</th>\n","      <th>time_ts</th>\n","      <th>text</th>\n","      <th>parent</th>\n","      <th>deleted</th>\n","      <th>dead</th>\n","      <th>ranking</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>9734136</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1434565400</td>\n","      <td>2015-06-17 18:23:20+00:00</td>\n","      <td>None</td>\n","      <td>9733698</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>4921158</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1355496966</td>\n","      <td>2012-12-14 14:56:06+00:00</td>\n","      <td>None</td>\n","      <td>4921100</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>7500568</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1396261158</td>\n","      <td>2014-03-31 10:19:18+00:00</td>\n","      <td>None</td>\n","      <td>7499385</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>8909635</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1421627275</td>\n","      <td>2015-01-19 00:27:55+00:00</td>\n","      <td>None</td>\n","      <td>8901135</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>9256463</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>1427204705</td>\n","      <td>2015-03-24 13:45:05+00:00</td>\n","      <td>None</td>\n","      <td>9256346</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-797c191c-3d70-43b0-92ba-ba2076521507')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-797c191c-3d70-43b0-92ba-ba2076521507 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-797c191c-3d70-43b0-92ba-ba2076521507');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":36}]},{"cell_type":"code","source":["# Construct a reference to the \"stories\" table\n","table_ref = dataset_ref.table(\"stories\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)\n","\n","# Preview the first five lines of the table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"id":"GG9M1WJYPgrg","colab":{"base_uri":"https://localhost:8080/","height":337},"outputId":"a59e6946-7647-464d-b9e7-db0082100e75","executionInfo":{"status":"ok","timestamp":1677904986190,"user_tz":-480,"elapsed":292,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":37,"outputs":[{"output_type":"execute_result","data":{"text/plain":["        id      by  score        time                   time_ts  \\\n","0  6988445  cflick      0  1388454902 2013-12-31 01:55:02+00:00   \n","1  7047571     Rd2      1  1389562985 2014-01-12 21:43:05+00:00   \n","2  9157712     mo0      1  1425657937 2015-03-06 16:05:37+00:00   \n","3  8127403    ad11      1  1407052667 2014-08-03 07:57:47+00:00   \n","4  6933158    emyy      1  1387432701 2013-12-19 05:58:21+00:00   \n","\n","                                               title  \\\n","0                                           Appshare   \n","1                                   Java in startups   \n","2  Show HN: Discover what songs were used in YouT...   \n","3               My poker project, what do you think?   \n","4  Christmas Crafts Ideas - Easy and Simple Famil...   \n","\n","                                                 url  \\\n","0                  http://chadflick.ws/appshare.html   \n","1                                                      \n","2                               http://www.mooma.sh/   \n","3                                                      \n","4  http://www.winxdvd.com/resource/christmas-craf...   \n","\n","                                                text  deleted  dead  \\\n","0      Did facebook or angrybirds pay you?  We will!     <NA>  True   \n","1  Hello, hacker news!<p>Have any of you used jav...     <NA>  True   \n","2  The user can paste a media url(currently only ...     <NA>  True   \n","3  Hi guys, what do you think about my poker proj...     <NA>  True   \n","4  There are some free Christmas craft ideas to m...     <NA>  True   \n","\n","   descendants  author  \n","0         <NA>  cflick  \n","1         <NA>     Rd2  \n","2         <NA>     mo0  \n","3         <NA>    ad11  \n","4         <NA>    emyy  "],"text/html":["\n","  <div id=\"df-08455135-feb5-4767-a695-83f24a1863f7\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>by</th>\n","      <th>score</th>\n","      <th>time</th>\n","      <th>time_ts</th>\n","      <th>title</th>\n","      <th>url</th>\n","      <th>text</th>\n","      <th>deleted</th>\n","      <th>dead</th>\n","      <th>descendants</th>\n","      <th>author</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>6988445</td>\n","      <td>cflick</td>\n","      <td>0</td>\n","      <td>1388454902</td>\n","      <td>2013-12-31 01:55:02+00:00</td>\n","      <td>Appshare</td>\n","      <td>http://chadflick.ws/appshare.html</td>\n","      <td>Did facebook or angrybirds pay you?  We will!</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>cflick</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>7047571</td>\n","      <td>Rd2</td>\n","      <td>1</td>\n","      <td>1389562985</td>\n","      <td>2014-01-12 21:43:05+00:00</td>\n","      <td>Java in startups</td>\n","      <td></td>\n","      <td>Hello, hacker news!&lt;p&gt;Have any of you used jav...</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>Rd2</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>9157712</td>\n","      <td>mo0</td>\n","      <td>1</td>\n","      <td>1425657937</td>\n","      <td>2015-03-06 16:05:37+00:00</td>\n","      <td>Show HN: Discover what songs were used in YouT...</td>\n","      <td>http://www.mooma.sh/</td>\n","      <td>The user can paste a media url(currently only ...</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>mo0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>8127403</td>\n","      <td>ad11</td>\n","      <td>1</td>\n","      <td>1407052667</td>\n","      <td>2014-08-03 07:57:47+00:00</td>\n","      <td>My poker project, what do you think?</td>\n","      <td></td>\n","      <td>Hi guys, what do you think about my poker proj...</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>ad11</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>6933158</td>\n","      <td>emyy</td>\n","      <td>1</td>\n","      <td>1387432701</td>\n","      <td>2013-12-19 05:58:21+00:00</td>\n","      <td>Christmas Crafts Ideas - Easy and Simple Famil...</td>\n","      <td>http://www.winxdvd.com/resource/christmas-craf...</td>\n","      <td>There are some free Christmas craft ideas to m...</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>True</td>\n","      <td>&lt;NA&gt;</td>\n","      <td>emyy</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-08455135-feb5-4767-a695-83f24a1863f7')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-08455135-feb5-4767-a695-83f24a1863f7 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-08455135-feb5-4767-a695-83f24a1863f7');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":37}]},{"cell_type":"markdown","source":["The query below pulls information from the `stories` and `comments` tables to create a table showing all stories posted on January 1, 2012, along with the corresponding number of comments.  We use a **LEFT JOIN** so that the results include stories that didn't receive any comments."],"metadata":{"id":"w-36FLhDPd2K"}},{"cell_type":"code","source":["# Query to select all stories posted on January 1, 2012, with number of comments\n","join_query = \"\"\"\n","             WITH c AS\n","             (\n","             SELECT parent, COUNT(*) as num_comments\n","             FROM `bigquery-public-data.hacker_news.comments` \n","             GROUP BY parent\n","             )\n","             SELECT s.id as story_id, s.by, s.title, c.num_comments\n","             FROM `bigquery-public-data.hacker_news.stories` AS s\n","             LEFT JOIN c\n","             ON s.id = c.parent\n","             WHERE EXTRACT(DATE FROM s.time_ts) = '2012-01-01'\n","             ORDER BY c.num_comments DESC\n","             \"\"\"\n","\n","# Run the query, and return a pandas DataFrame\n","join_result = client.query(join_query).result().to_dataframe()\n","join_result.head()"],"metadata":{"id":"WU9bl9oYPlUj","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"81460959-194f-48d3-cb84-845cd93db4c0","executionInfo":{"status":"ok","timestamp":1677904992253,"user_tz":-480,"elapsed":6065,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":38,"outputs":[{"output_type":"execute_result","data":{"text/plain":["   story_id           by                                              title  \\\n","0   3412900  whoishiring              Ask HN: Who is Hiring? (January 2012)   \n","1   3412901  whoishiring  Ask HN: Freelancer? Seeking freelancer? (Janua...   \n","2   3412643     jemeshsu                                       Avoid Apress   \n","3   3412891   Brajeshwar  There's no shame in code that is simply \"good ...   \n","4   3414012    ramanujam  Impress.js - a Prezi like implementation using...   \n","\n","   num_comments  \n","0           154  \n","1            97  \n","2            30  \n","3            27  \n","4            27  "],"text/html":["\n","  <div id=\"df-ec9d878c-98c4-49da-a91d-1acae017914d\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>story_id</th>\n","      <th>by</th>\n","      <th>title</th>\n","      <th>num_comments</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>3412900</td>\n","      <td>whoishiring</td>\n","      <td>Ask HN: Who is Hiring? (January 2012)</td>\n","      <td>154</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>3412901</td>\n","      <td>whoishiring</td>\n","      <td>Ask HN: Freelancer? Seeking freelancer? (Janua...</td>\n","      <td>97</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3412643</td>\n","      <td>jemeshsu</td>\n","      <td>Avoid Apress</td>\n","      <td>30</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3412891</td>\n","      <td>Brajeshwar</td>\n","      <td>There's no shame in code that is simply \"good ...</td>\n","      <td>27</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>3414012</td>\n","      <td>ramanujam</td>\n","      <td>Impress.js - a Prezi like implementation using...</td>\n","      <td>27</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ec9d878c-98c4-49da-a91d-1acae017914d')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-ec9d878c-98c4-49da-a91d-1acae017914d button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-ec9d878c-98c4-49da-a91d-1acae017914d');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":38}]},{"cell_type":"markdown","source":["Since the results are ordered by the `num_comments` column, stories without comments appear at the end of the DataFrame.  (Remember that **NaN** stands for \"not a number\".)"],"metadata":{"id":"s3F7dqryPnua"}},{"cell_type":"code","source":["# None of these stories received any comments\n","join_result.tail()"],"metadata":{"id":"EbbScrGpPqTy","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"99740a56-7760-435d-c710-604d6f64ff3c","executionInfo":{"status":"ok","timestamp":1677904992253,"user_tz":-480,"elapsed":4,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":39,"outputs":[{"output_type":"execute_result","data":{"text/plain":["     story_id               by  \\\n","439   3413342       jonsteiman   \n","440   3412327           arroyo   \n","441   3413203    norris_tony44   \n","442   3412940         julelara   \n","443   3412632  UsedCarFleetCom   \n","\n","                                                 title  num_comments  \n","439       The List You Can't Miss: Top 5 Blogs of 2011          <NA>  \n","440               Can your business card be followed?           <NA>  \n","441              10 Popular iPhone Games you Must Play          <NA>  \n","442  Washington Redskins vs Philadelphia Eagles liv...          <NA>  \n","443                                     Used Car fleet          <NA>  "],"text/html":["\n","  <div id=\"df-949eecda-10bd-4506-8e4d-c3a5df391c1a\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>story_id</th>\n","      <th>by</th>\n","      <th>title</th>\n","      <th>num_comments</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>439</th>\n","      <td>3413342</td>\n","      <td>jonsteiman</td>\n","      <td>The List You Can't Miss: Top 5 Blogs of 2011</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","    <tr>\n","      <th>440</th>\n","      <td>3412327</td>\n","      <td>arroyo</td>\n","      <td>Can your business card be followed?</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","    <tr>\n","      <th>441</th>\n","      <td>3413203</td>\n","      <td>norris_tony44</td>\n","      <td>10 Popular iPhone Games you Must Play</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","    <tr>\n","      <th>442</th>\n","      <td>3412940</td>\n","      <td>julelara</td>\n","      <td>Washington Redskins vs Philadelphia Eagles liv...</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","    <tr>\n","      <th>443</th>\n","      <td>3412632</td>\n","      <td>UsedCarFleetCom</td>\n","      <td>Used Car fleet</td>\n","      <td>&lt;NA&gt;</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-949eecda-10bd-4506-8e4d-c3a5df391c1a')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-949eecda-10bd-4506-8e4d-c3a5df391c1a button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-949eecda-10bd-4506-8e4d-c3a5df391c1a');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":39}]},{"cell_type":"markdown","source":["As you've seen, JOINs horizontally combine results from different tables. If you instead would like to vertically concatenate columns, you can do so with a `UNION`. \n","\n","Next, we write a query to select all usernames corresponding to users who wrote stories or comments on January 1, 2014.  We use **UNION DISTINCT** (instead of **UNION ALL**) to ensure that each user appears in the table at most once."],"metadata":{"id":"nZ65-rWPPsPf"}},{"cell_type":"code","source":["# Query to select all users who posted stories or comments on January 1, 2014\n","union_query = \"\"\"\n","              SELECT c.by\n","              FROM `bigquery-public-data.hacker_news.comments` AS c\n","              WHERE EXTRACT(DATE FROM c.time_ts) = '2014-01-01'\n","              UNION DISTINCT\n","              SELECT s.by\n","              FROM `bigquery-public-data.hacker_news.stories` AS s\n","              WHERE EXTRACT(DATE FROM s.time_ts) = '2014-01-01'\n","              \"\"\"\n","\n","# Run the query, and return a pandas DataFrame\n","union_result = client.query(union_query).result().to_dataframe()\n","union_result.head()"],"metadata":{"id":"W7SueEUcPt5P","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"a5e8c04e-f14d-4861-8fb5-fa66ba171313","executionInfo":{"status":"ok","timestamp":1677904995197,"user_tz":-480,"elapsed":2947,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":40,"outputs":[{"output_type":"execute_result","data":{"text/plain":["            by\n","0       vidarh\n","1  tlarkworthy\n","2          jbl\n","3       dmgrow\n","4      maurorm"],"text/html":["\n","  <div id=\"df-bdf3cbf9-68f3-48c4-874b-fd44d872be77\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>by</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>vidarh</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>tlarkworthy</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>jbl</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>dmgrow</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>maurorm</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bdf3cbf9-68f3-48c4-874b-fd44d872be77')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-bdf3cbf9-68f3-48c4-874b-fd44d872be77 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-bdf3cbf9-68f3-48c4-874b-fd44d872be77');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":40}]},{"cell_type":"markdown","source":["To get the number of users who posted on January 1, 2014, we need only take the length of the DataFrame."],"metadata":{"id":"pHWr22nyPwL4"}},{"cell_type":"code","source":["# Number of users who posted stories or comments on January 1, 2014\n","len(union_result)"],"metadata":{"id":"fXlfo6uOPyWw","colab":{"base_uri":"https://localhost:8080/"},"outputId":"a37e1355-28aa-4e72-bbbc-3c1053690f0f","executionInfo":{"status":"ok","timestamp":1677904995197,"user_tz":-480,"elapsed":6,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":41,"outputs":[{"output_type":"execute_result","data":{"text/plain":["2282"]},"metadata":{},"execution_count":41}]},{"cell_type":"markdown","source":["### Nested and Repeated data"],"metadata":{"id":"qPwco53PO6YV"}},{"cell_type":"markdown","source":["So far, you've worked with many types of data, including numeric types (integers, floating point values), strings, and the DATETIME type. In this tutorial, you'll learn how to query nested and repeated data. These are the most complex data types that you can find in BigQuery datasets!"],"metadata":{"id":"X1GK-qpkPAvU"}},{"cell_type":"markdown","source":["We'll work with the Google Analytics Sample dataset. It contains information tracking the behavior of visitors to the Google Merchandise store, an e-commerce website that sells Google branded items.\n","\n","We begin by printing the first few rows of the `ga_sessions_20170801` table. This table tracks visits to the website on August 1, 2017. The table has many nested fields from table preview:"],"metadata":{"id":"3ExcmDBjP2QN"}},{"cell_type":"code","source":["# Construct a reference to the \"google_analytics_sample\" dataset\n","dataset_ref = client.dataset(\"google_analytics_sample\", project=\"bigquery-public-data\")\n","\n","# Construct a reference to the \"ga_sessions_20170801\" table\n","table_ref = dataset_ref.table(\"ga_sessions_20170801\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)\n","\n","# Preview the first five lines of the table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"id":"HyksohjiP8iE","colab":{"base_uri":"https://localhost:8080/","height":2676},"executionInfo":{"status":"ok","timestamp":1677905005497,"user_tz":-480,"elapsed":809,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}},"outputId":"0ab1773b-e1f4-4238-ee23-a8a9b07745f5"},"execution_count":42,"outputs":[{"output_type":"execute_result","data":{"text/plain":["   visitorId  visitNumber     visitId  visitStartTime      date  \\\n","0       <NA>            1  1501591568      1501591568  20170801   \n","1       <NA>            2  1501589647      1501589647  20170801   \n","2       <NA>            1  1501616621      1501616621  20170801   \n","3       <NA>            1  1501601200      1501601200  20170801   \n","4       <NA>            1  1501615525      1501615525  20170801   \n","\n","                                              totals  \\\n","0  {'visits': 1, 'hits': 1, 'pageviews': 1, 'time...   \n","1  {'visits': 1, 'hits': 1, 'pageviews': 1, 'time...   \n","2  {'visits': 1, 'hits': 1, 'pageviews': 1, 'time...   \n","3  {'visits': 1, 'hits': 1, 'pageviews': 1, 'time...   \n","4  {'visits': 1, 'hits': 1, 'pageviews': 1, 'time...   \n","\n","                                       trafficSource  \\\n","0  {'referralPath': None, 'campaign': '(not set)'...   \n","1  {'referralPath': '/analytics/web/', 'campaign'...   \n","2  {'referralPath': '/analytics/web/', 'campaign'...   \n","3  {'referralPath': '/analytics/web/', 'campaign'...   \n","4  {'referralPath': '/analytics/web/', 'campaign'...   \n","\n","                                              device  \\\n","0  {'browser': 'Chrome', 'browserVersion': 'not a...   \n","1  {'browser': 'Chrome', 'browserVersion': 'not a...   \n","2  {'browser': 'Chrome', 'browserVersion': 'not a...   \n","3  {'browser': 'Firefox', 'browserVersion': 'not ...   \n","4  {'browser': 'Chrome', 'browserVersion': 'not a...   \n","\n","                                          geoNetwork  \\\n","0  {'continent': 'Europe', 'subContinent': 'South...   \n","1  {'continent': 'Asia', 'subContinent': 'Souther...   \n","2  {'continent': 'Europe', 'subContinent': 'North...   \n","3  {'continent': 'Americas', 'subContinent': 'Nor...   \n","4  {'continent': 'Americas', 'subContinent': 'Nor...   \n","\n","                           customDimensions  \\\n","0                                        []   \n","1           [{'index': 4, 'value': 'APAC'}]   \n","2           [{'index': 4, 'value': 'EMEA'}]   \n","3  [{'index': 4, 'value': 'North America'}]   \n","4  [{'index': 4, 'value': 'North America'}]   \n","\n","                                                hits        fullVisitorId  \\\n","0  [{'hitNumber': 1, 'time': 0, 'hour': 5, 'minut...  3418334011779872055   \n","1  [{'hitNumber': 1, 'time': 0, 'hour': 5, 'minut...  2474397855041322408   \n","2  [{'hitNumber': 1, 'time': 0, 'hour': 12, 'minu...  5870462820713110108   \n","3  [{'hitNumber': 1, 'time': 0, 'hour': 8, 'minut...  9397809171349480379   \n","4  [{'hitNumber': 1, 'time': 0, 'hour': 12, 'minu...  6089902943184578335   \n","\n","  userId clientId channelGrouping  socialEngagementType  \n","0   None     None  Organic Search  Not Socially Engaged  \n","1   None     None        Referral  Not Socially Engaged  \n","2   None     None        Referral  Not Socially Engaged  \n","3   None     None        Referral  Not Socially Engaged  \n","4   None     None        Referral  Not Socially Engaged  "],"text/html":["\n","  <div id=\"df-7cc5b604-e790-45dd-ad22-50796c10062e\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>visitorId</th>\n","      <th>visitNumber</th>\n","      <th>visitId</th>\n","      <th>visitStartTime</th>\n","      <th>date</th>\n","      <th>totals</th>\n","      <th>trafficSource</th>\n","      <th>device</th>\n","      <th>geoNetwork</th>\n","      <th>customDimensions</th>\n","      <th>hits</th>\n","      <th>fullVisitorId</th>\n","      <th>userId</th>\n","      <th>clientId</th>\n","      <th>channelGrouping</th>\n","      <th>socialEngagementType</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>&lt;NA&gt;</td>\n","      <td>1</td>\n","      <td>1501591568</td>\n","      <td>1501591568</td>\n","      <td>20170801</td>\n","      <td>{'visits': 1, 'hits': 1, 'pageviews': 1, 'time...</td>\n","      <td>{'referralPath': None, 'campaign': '(not set)'...</td>\n","      <td>{'browser': 'Chrome', 'browserVersion': 'not a...</td>\n","      <td>{'continent': 'Europe', 'subContinent': 'South...</td>\n","      <td>[]</td>\n","      <td>[{'hitNumber': 1, 'time': 0, 'hour': 5, 'minut...</td>\n","      <td>3418334011779872055</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>Organic Search</td>\n","      <td>Not Socially Engaged</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>&lt;NA&gt;</td>\n","      <td>2</td>\n","      <td>1501589647</td>\n","      <td>1501589647</td>\n","      <td>20170801</td>\n","      <td>{'visits': 1, 'hits': 1, 'pageviews': 1, 'time...</td>\n","      <td>{'referralPath': '/analytics/web/', 'campaign'...</td>\n","      <td>{'browser': 'Chrome', 'browserVersion': 'not a...</td>\n","      <td>{'continent': 'Asia', 'subContinent': 'Souther...</td>\n","      <td>[{'index': 4, 'value': 'APAC'}]</td>\n","      <td>[{'hitNumber': 1, 'time': 0, 'hour': 5, 'minut...</td>\n","      <td>2474397855041322408</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>Referral</td>\n","      <td>Not Socially Engaged</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>&lt;NA&gt;</td>\n","      <td>1</td>\n","      <td>1501616621</td>\n","      <td>1501616621</td>\n","      <td>20170801</td>\n","      <td>{'visits': 1, 'hits': 1, 'pageviews': 1, 'time...</td>\n","      <td>{'referralPath': '/analytics/web/', 'campaign'...</td>\n","      <td>{'browser': 'Chrome', 'browserVersion': 'not a...</td>\n","      <td>{'continent': 'Europe', 'subContinent': 'North...</td>\n","      <td>[{'index': 4, 'value': 'EMEA'}]</td>\n","      <td>[{'hitNumber': 1, 'time': 0, 'hour': 12, 'minu...</td>\n","      <td>5870462820713110108</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>Referral</td>\n","      <td>Not Socially Engaged</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>&lt;NA&gt;</td>\n","      <td>1</td>\n","      <td>1501601200</td>\n","      <td>1501601200</td>\n","      <td>20170801</td>\n","      <td>{'visits': 1, 'hits': 1, 'pageviews': 1, 'time...</td>\n","      <td>{'referralPath': '/analytics/web/', 'campaign'...</td>\n","      <td>{'browser': 'Firefox', 'browserVersion': 'not ...</td>\n","      <td>{'continent': 'Americas', 'subContinent': 'Nor...</td>\n","      <td>[{'index': 4, 'value': 'North America'}]</td>\n","      <td>[{'hitNumber': 1, 'time': 0, 'hour': 8, 'minut...</td>\n","      <td>9397809171349480379</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>Referral</td>\n","      <td>Not Socially Engaged</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>&lt;NA&gt;</td>\n","      <td>1</td>\n","      <td>1501615525</td>\n","      <td>1501615525</td>\n","      <td>20170801</td>\n","      <td>{'visits': 1, 'hits': 1, 'pageviews': 1, 'time...</td>\n","      <td>{'referralPath': '/analytics/web/', 'campaign'...</td>\n","      <td>{'browser': 'Chrome', 'browserVersion': 'not a...</td>\n","      <td>{'continent': 'Americas', 'subContinent': 'Nor...</td>\n","      <td>[{'index': 4, 'value': 'North America'}]</td>\n","      <td>[{'hitNumber': 1, 'time': 0, 'hour': 12, 'minu...</td>\n","      <td>6089902943184578335</td>\n","      <td>None</td>\n","      <td>None</td>\n","      <td>Referral</td>\n","      <td>Not Socially Engaged</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7cc5b604-e790-45dd-ad22-50796c10062e')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-7cc5b604-e790-45dd-ad22-50796c10062e button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-7cc5b604-e790-45dd-ad22-50796c10062e');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":42}]},{"cell_type":"markdown","source":["Now we'll work with the `hits` column as an example of data that is both nested and repeated. Since:\n","\n","- `hits` is a STRUCT (contains nested data) and is repeated,\n","- `hitNumber`, `page`, and `type` are all nested inside the `hits` column, and\n","- `pagePath` is nested inside the `page` field,\n","\n","we can query these fields with the following syntax:"],"metadata":{"id":"lXdO2vZOQcl8"}},{"cell_type":"code","source":["# Query to determine most popular landing point on the website\n","query = \"\"\"\n","        SELECT hits.page.pagePath as path,\n","            COUNT(hits.page.pagePath) as counts\n","        FROM `bigquery-public-data.google_analytics_sample.ga_sessions_20170801`, \n","            UNNEST(hits) as hits\n","        WHERE hits.type=\"PAGE\" and hits.hitNumber=1\n","        GROUP BY path\n","        ORDER BY counts DESC\n","        \"\"\"\n","\n","# Run the query, and return a pandas DataFrame\n","result = client.query(query).result().to_dataframe()\n","result.head()"],"metadata":{"id":"EuN6fzHQQqMj","colab":{"base_uri":"https://localhost:8080/","height":206},"executionInfo":{"status":"ok","timestamp":1677905010813,"user_tz":-480,"elapsed":2924,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}},"outputId":"4589e6aa-fd97-44f8-9c5b-e53c9be75907"},"execution_count":43,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                                          path  counts\n","0                                        /home    1257\n","1       /google+redesign/shop+by+brand/youtube     587\n","2  /google+redesign/apparel/mens/mens+t+shirts     117\n","3                                 /signin.html      78\n","4                                 /basket.html      35"],"text/html":["\n","  <div id=\"df-52dd7cae-3376-415e-96c6-dbf91a37fbb8\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>path</th>\n","      <th>counts</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>/home</td>\n","      <td>1257</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>/google+redesign/shop+by+brand/youtube</td>\n","      <td>587</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>/google+redesign/apparel/mens/mens+t+shirts</td>\n","      <td>117</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>/signin.html</td>\n","      <td>78</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>/basket.html</td>\n","      <td>35</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-52dd7cae-3376-415e-96c6-dbf91a37fbb8')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-52dd7cae-3376-415e-96c6-dbf91a37fbb8 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-52dd7cae-3376-415e-96c6-dbf91a37fbb8');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":43}]},{"cell_type":"markdown","source":["### Analytic Function (Optional)"],"metadata":{"id":"jRJR1mPdmhwD"}},{"cell_type":"markdown","source":["You can also define analytic functions, which also operate on a set of rows like aggregation function. However, unlike aggregate functions, analytic functions return a (potentially different) value for each row in the original table. Analytic functions allow us to perform complex calculations with relatively straightforward syntax. For instance, we can quickly calculate moving averages and running totals, among other quantities.\n","\n","We'll work with the [San Francisco Open Data](https://www.kaggle.com/datasf/san-francisco) dataset."],"metadata":{"id":"pWi_mgaHRr4d"}},{"cell_type":"code","source":["# Construct a reference to the \"san_francisco\" dataset\n","dataset_ref = client.dataset(\"san_francisco\", project=\"bigquery-public-data\")\n","\n","# API request - fetch the dataset\n","dataset = client.get_dataset(dataset_ref)\n","\n","# Construct a reference to the \"bikeshare_trips\" table\n","table_ref = dataset_ref.table(\"bikeshare_trips\")\n","\n","# API request - fetch the table\n","table = client.get_table(table_ref)\n","\n","# Preview the first five lines of the table\n","client.list_rows(table, max_results=5).to_dataframe()"],"metadata":{"id":"toMZH_UbRvHH","colab":{"base_uri":"https://localhost:8080/","height":337},"outputId":"0ab883e6-7f19-4626-e63e-e9d161737f10","executionInfo":{"status":"ok","timestamp":1677905142622,"user_tz":-480,"elapsed":769,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":44,"outputs":[{"output_type":"execute_result","data":{"text/plain":["   trip_id  duration_sec                start_date  \\\n","0  1235850          1540 2016-06-11 08:19:00+00:00   \n","1  1219337          6324 2016-05-29 12:49:00+00:00   \n","2   793762        115572 2015-06-04 09:22:00+00:00   \n","3   453845         54120 2014-09-15 16:53:00+00:00   \n","4  1245113          5018 2016-06-17 20:08:00+00:00   \n","\n","                  start_station_name  start_station_id  \\\n","0  San Jose Diridon Caltrain Station                 2   \n","1  San Jose Diridon Caltrain Station                 2   \n","2  San Jose Diridon Caltrain Station                 2   \n","3  San Jose Diridon Caltrain Station                 2   \n","4  San Jose Diridon Caltrain Station                 2   \n","\n","                   end_date                   end_station_name  \\\n","0 2016-06-11 08:45:00+00:00  San Jose Diridon Caltrain Station   \n","1 2016-05-29 14:34:00+00:00  San Jose Diridon Caltrain Station   \n","2 2015-06-05 17:28:00+00:00  San Jose Diridon Caltrain Station   \n","3 2014-09-16 07:55:00+00:00  San Jose Diridon Caltrain Station   \n","4 2016-06-17 21:32:00+00:00  San Jose Diridon Caltrain Station   \n","\n","   end_station_id  bike_number zip_code subscriber_type  \n","0               2          124    15206        Customer  \n","1               2          174    55416        Customer  \n","2               2          190    95391        Customer  \n","3               2          127       81        Customer  \n","4               2          153    95070        Customer  "],"text/html":["\n","  <div id=\"df-48ccaf1a-0d12-4b18-b35c-e730180c2be8\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trip_id</th>\n","      <th>duration_sec</th>\n","      <th>start_date</th>\n","      <th>start_station_name</th>\n","      <th>start_station_id</th>\n","      <th>end_date</th>\n","      <th>end_station_name</th>\n","      <th>end_station_id</th>\n","      <th>bike_number</th>\n","      <th>zip_code</th>\n","      <th>subscriber_type</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1235850</td>\n","      <td>1540</td>\n","      <td>2016-06-11 08:19:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>2016-06-11 08:45:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>124</td>\n","      <td>15206</td>\n","      <td>Customer</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1219337</td>\n","      <td>6324</td>\n","      <td>2016-05-29 12:49:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>2016-05-29 14:34:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>174</td>\n","      <td>55416</td>\n","      <td>Customer</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>793762</td>\n","      <td>115572</td>\n","      <td>2015-06-04 09:22:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>2015-06-05 17:28:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>190</td>\n","      <td>95391</td>\n","      <td>Customer</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>453845</td>\n","      <td>54120</td>\n","      <td>2014-09-15 16:53:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>2014-09-16 07:55:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>127</td>\n","      <td>81</td>\n","      <td>Customer</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>1245113</td>\n","      <td>5018</td>\n","      <td>2016-06-17 20:08:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>2016-06-17 21:32:00+00:00</td>\n","      <td>San Jose Diridon Caltrain Station</td>\n","      <td>2</td>\n","      <td>153</td>\n","      <td>95070</td>\n","      <td>Customer</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-48ccaf1a-0d12-4b18-b35c-e730180c2be8')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-48ccaf1a-0d12-4b18-b35c-e730180c2be8 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-48ccaf1a-0d12-4b18-b35c-e730180c2be8');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":44}]},{"cell_type":"markdown","source":["Each row of the table corresponds to a different bike trip, and we can use an analytic function to **calculate the cumulative number of trips for each date in 2015.**"],"metadata":{"id":"nY8VBY-8Rx8w"}},{"cell_type":"code","source":["# Query to count the (cumulative) number of trips per day\n","num_trips_query = \"\"\"\n","                  WITH trips_by_day AS\n","                  (\n","                  SELECT DATE(start_date) AS trip_date,\n","                      COUNT(*) as num_trips\n","                  FROM `bigquery-public-data.san_francisco.bikeshare_trips`\n","                  WHERE EXTRACT(YEAR FROM start_date) = 2015\n","                  GROUP BY trip_date\n","                  )\n","                  SELECT *,\n","                      SUM(num_trips) \n","                          OVER (\n","                               ORDER BY trip_date\n","                               ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW\n","                               ) AS cumulative_trips\n","                  FROM trips_by_day\n","                  \"\"\"\n","\n","# Run the query, and return a pandas DataFrame\n","num_trips_result = client.query(num_trips_query).result().to_dataframe()\n","num_trips_result.head()"],"metadata":{"id":"_aHYi_0IR0WG","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"854c92df-a3cf-4770-c5f4-2a710159ae73","executionInfo":{"status":"ok","timestamp":1677905148241,"user_tz":-480,"elapsed":4330,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":45,"outputs":[{"output_type":"execute_result","data":{"text/plain":["    trip_date  num_trips  cumulative_trips\n","0  2015-01-13       1368             10709\n","1  2015-04-06       1281             91635\n","2  2015-04-25        405            111722\n","3  2015-06-18       1352            166602\n","4  2015-07-14       1358            192161"],"text/html":["\n","  <div id=\"df-01822dfe-c121-4eb1-9ad2-4f81718d7a67\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trip_date</th>\n","      <th>num_trips</th>\n","      <th>cumulative_trips</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2015-01-13</td>\n","      <td>1368</td>\n","      <td>10709</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2015-04-06</td>\n","      <td>1281</td>\n","      <td>91635</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2015-04-25</td>\n","      <td>405</td>\n","      <td>111722</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>2015-06-18</td>\n","      <td>1352</td>\n","      <td>166602</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>2015-07-14</td>\n","      <td>1358</td>\n","      <td>192161</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-01822dfe-c121-4eb1-9ad2-4f81718d7a67')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-01822dfe-c121-4eb1-9ad2-4f81718d7a67 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-01822dfe-c121-4eb1-9ad2-4f81718d7a67');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":45}]},{"cell_type":"markdown","source":["The query uses a [common table expression (CTE)](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#with_clause) to first calculate the daily number of trips.  Then, we use **SUM()** as an aggregate function.\n","- Since there is no **PARTITION BY** clause, the entire table is treated as a single partition.\n","- The **ORDER BY** clause orders the rows by date, where earlier dates appear first. \n","- By setting the **window frame** clause to `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`, we ensure that all rows up to and including the current date are used to calculate the (cumulative) sum. See https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts#def_window_frame for more details.\n","\n","The next query **tracks the stations where each bike began (in `start_station_id`) and ended (in `end_station_id`) the day on October 25, 2015.**"],"metadata":{"id":"ShH0iValR196"}},{"cell_type":"code","source":["# Query to track beginning and ending stations on October 25, 2015, for each bike\n","start_end_query = \"\"\"\n","                  SELECT bike_number,\n","                      TIME(start_date) AS trip_time,\n","                      FIRST_VALUE(start_station_id)\n","                          OVER (\n","                               PARTITION BY bike_number\n","                               ORDER BY start_date\n","                               ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING\n","                               ) AS first_station_id,\n","                      LAST_VALUE(end_station_id)\n","                          OVER (\n","                               PARTITION BY bike_number\n","                               ORDER BY start_date\n","                               ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING\n","                               ) AS last_station_id,\n","                      start_station_id,\n","                      end_station_id\n","                  FROM `bigquery-public-data.san_francisco.bikeshare_trips`\n","                  WHERE DATE(start_date) = '2015-10-25' \n","                  \"\"\"\n","\n","# Run the query, and return a pandas DataFrame\n","start_end_result = client.query(start_end_query).result().to_dataframe()\n","start_end_result.head()"],"metadata":{"id":"ivG9NWgZR7QY","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"7eb8b339-b399-4a8e-8ec6-725301140127","executionInfo":{"status":"ok","timestamp":1677905155783,"user_tz":-480,"elapsed":4186,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":46,"outputs":[{"output_type":"execute_result","data":{"text/plain":["   bike_number trip_time  first_station_id  last_station_id  start_station_id  \\\n","0          230  22:41:00                22               22                22   \n","1          601  15:50:00                68               67                68   \n","2          601  23:27:00                68               67                50   \n","3          604  08:56:00                70               66                70   \n","4          604  12:34:00                70               66                39   \n","\n","   end_station_id  \n","0              22  \n","1              50  \n","2              67  \n","3              39  \n","4              67  "],"text/html":["\n","  <div id=\"df-c205929d-c797-4058-86b5-e87437f7d90d\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>bike_number</th>\n","      <th>trip_time</th>\n","      <th>first_station_id</th>\n","      <th>last_station_id</th>\n","      <th>start_station_id</th>\n","      <th>end_station_id</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>230</td>\n","      <td>22:41:00</td>\n","      <td>22</td>\n","      <td>22</td>\n","      <td>22</td>\n","      <td>22</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>601</td>\n","      <td>15:50:00</td>\n","      <td>68</td>\n","      <td>67</td>\n","      <td>68</td>\n","      <td>50</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>601</td>\n","      <td>23:27:00</td>\n","      <td>68</td>\n","      <td>67</td>\n","      <td>50</td>\n","      <td>67</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>604</td>\n","      <td>08:56:00</td>\n","      <td>70</td>\n","      <td>66</td>\n","      <td>70</td>\n","      <td>39</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>604</td>\n","      <td>12:34:00</td>\n","      <td>70</td>\n","      <td>66</td>\n","      <td>39</td>\n","      <td>67</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c205929d-c797-4058-86b5-e87437f7d90d')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-c205929d-c797-4058-86b5-e87437f7d90d button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-c205929d-c797-4058-86b5-e87437f7d90d');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":46}]},{"cell_type":"markdown","source":["The query uses both **FIRST_VALUE()** and **LAST_VALUE()** as analytic functions.\n","- The **PARTITION BY** clause breaks the data into partitions based on the `bike_number` column.  Since this column holds unique identifiers for the bikes, this ensures the calculations are performed separately for each bike.\n","- The **ORDER BY** clause puts the rows within each partition in chronological order.\n","- Since the **window frame** clause is `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING`, for each row, its entire partition is used to perform the calculation.  (_This ensures the calculated values for rows in the same partition are identical._)"],"metadata":{"id":"iegKJa1nR_rZ"}},{"cell_type":"markdown","source":["You can check https://cloud.google.com/bigquery/docs/reference/standard-sql/introduction and https://googleapis.dev/python/bigquery/latest/index.html for more details."],"metadata":{"id":"4XUIvA2v2A30"}},{"cell_type":"markdown","source":["## Data Wrangling with Pandas"],"metadata":{"id":"Sd-M02wrbNle"}},{"cell_type":"markdown","metadata":{"id":"LU-ma6F6bdkB"},"source":["### `Series` objects\n","The `Pandas` library contains these useful data structures:\n","* `Series` objects, that we will discuss now. A `Series` object is 1D array, similar to a column in a spreadsheet (with a column name and row labels).\n","* `DataFrame` objects. This is a 2D table, similar to a spreadsheet (with column names and row labels)."]},{"cell_type":"markdown","source":["#### Creating a `Series`"],"metadata":{"id":"oXl6kdLvoydm"}},{"cell_type":"markdown","metadata":{"id":"q6En2jWCbdkC"},"source":["Let's start by creating our first `Series` object!"]},{"cell_type":"code","execution_count":47,"metadata":{"id":"Twbix6NpbdkC","outputId":"066f5c2f-7b91-4805-bdd3-4efbd3a8712c","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905179704,"user_tz":-480,"elapsed":313,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    2\n","1   -1\n","2    3\n","3    5\n","dtype: int64"]},"metadata":{},"execution_count":47}],"source":["s = pd.Series([2,-1,3,5])\n","s"]},{"cell_type":"markdown","metadata":{"id":"6ERNyjvMbdkE"},"source":["Arithmetic operations on `Series` are also possible, and they apply *elementwise*, just like for `ndarray`s in NumPy:"]},{"cell_type":"code","execution_count":48,"metadata":{"id":"qCPOAd6tbdkF","outputId":"24a64a15-58d9-47d2-a4aa-58a6cbd625a7","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905183363,"user_tz":-480,"elapsed":301,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    1002\n","1    1999\n","2    3003\n","3    4005\n","dtype: int64"]},"metadata":{},"execution_count":48}],"source":["s + [1000,2000,3000,4000]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"WOQxfgJEbdkF","outputId":"9fc0dc18-d336-4e04-9664-902864c5c456","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677380049273,"user_tz":-480,"elapsed":295,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    1002\n","1     999\n","2    1003\n","3    1005\n","dtype: int64"]},"metadata":{},"execution_count":20}],"source":["s + 1000 # Broadcasting"]},{"cell_type":"markdown","source":["#### Index labels"],"metadata":{"id":"9nIlakAapJfv"}},{"cell_type":"markdown","metadata":{"id":"05v47g-TbdkG"},"source":["Each item in a `Series` object has a unique identifier called the *index label*. By default, it is simply the rank of the item in the `Series` (starting at `0`) but you can also set the index labels manually:"]},{"cell_type":"code","execution_count":49,"metadata":{"id":"uToGC_H1bdkG","outputId":"8e5a154a-5a2c-4956-a670-4b204f92da32","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905203549,"user_tz":-480,"elapsed":269,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["alice       68\n","bob         83\n","charles    112\n","darwin      68\n","dtype: int64"]},"metadata":{},"execution_count":49}],"source":["s2 = pd.Series([68, 83, 112, 68], index=[\"alice\", \"bob\", \"charles\", \"darwin\"])\n","s2"]},{"cell_type":"markdown","metadata":{"id":"YHll54B1bdkH"},"source":["You can then use the `Series` just like a `dict`:"]},{"cell_type":"code","execution_count":50,"metadata":{"id":"k_5q4EuqbdkH","outputId":"3adf8fa3-028b-402a-aa9f-6c55e56c1120","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905210229,"user_tz":-480,"elapsed":281,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["83"]},"metadata":{},"execution_count":50}],"source":["s2[\"bob\"]"]},{"cell_type":"markdown","metadata":{"id":"mmMlzPbLbdkH"},"source":["You can still access the items by integer location, like in a regular array:"]},{"cell_type":"code","execution_count":51,"metadata":{"id":"xhH_OoLQbdkI","outputId":"2637cebc-e32b-4621-c6f8-92e101599205","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905214184,"user_tz":-480,"elapsed":2,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["83"]},"metadata":{},"execution_count":51}],"source":["s2[1]"]},{"cell_type":"markdown","metadata":{"id":"TFDcqmL2bdkI"},"source":["To make it clear when you are accessing, **it is recommended to always use the `loc` attribute when accessing by label, and the `iloc` attribute when accessing by integer location**:"]},{"cell_type":"code","execution_count":52,"metadata":{"id":"86J9jGtfbdkI","outputId":"cf8ec0dc-c6fd-4156-8a68-cf32a17c3a5c","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905219801,"user_tz":-480,"elapsed":304,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["83"]},"metadata":{},"execution_count":52}],"source":["s2.loc[\"bob\"]"]},{"cell_type":"code","execution_count":53,"metadata":{"id":"TMjzFBctbdkI","outputId":"09be2422-436b-4815-a22a-2fbd212f6718","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905220221,"user_tz":-480,"elapsed":2,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["83"]},"metadata":{},"execution_count":53}],"source":["s2.iloc[1]"]},{"cell_type":"markdown","metadata":{"id":"tYYk0kN2bdkJ"},"source":["Slicing a `Series` also slices the index labels:"]},{"cell_type":"code","execution_count":54,"metadata":{"id":"pGJh6BLRbdkK","outputId":"74106f73-afe9-48da-fdae-fe414020a090","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905222079,"user_tz":-480,"elapsed":306,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["bob         83\n","charles    112\n","dtype: int64"]},"metadata":{},"execution_count":54}],"source":["s2.iloc[1:3]"]},{"cell_type":"markdown","source":["#### Initialize from `dict`"],"metadata":{"id":"0Run8DDKpZCR"}},{"cell_type":"markdown","metadata":{"id":"hzw9bWbBbdkM"},"source":["You can create a `Series` object from a `dict`. The keys will be used as index labels:"]},{"cell_type":"code","execution_count":55,"metadata":{"id":"u-QkmNSSbdkM","outputId":"0ec86122-fb72-4d28-a5fe-91224cf2286b","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905239401,"user_tz":-480,"elapsed":417,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["alice     68\n","bob       83\n","colin     86\n","darwin    68\n","dtype: int64"]},"metadata":{},"execution_count":55}],"source":["weights = {\"alice\": 68, \"bob\": 83, \"colin\": 86, \"darwin\": 68}\n","s3 = pd.Series(weights)\n","s3"]},{"cell_type":"markdown","metadata":{"id":"SM3qtn08bdkM"},"source":["When an operation involves multiple `Series` objects, `pandas` automatically aligns items by matching index labels."]},{"cell_type":"code","execution_count":56,"metadata":{"id":"5AI1dciMbdkM","outputId":"b2f0de77-465e-4227-a66e-db29a96c99a3","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677905239848,"user_tz":-480,"elapsed":2,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Index(['alice', 'bob', 'charles', 'darwin'], dtype='object')\n","Index(['alice', 'bob', 'colin', 'darwin'], dtype='object')\n"]},{"output_type":"execute_result","data":{"text/plain":["alice      136.0\n","bob        166.0\n","charles      NaN\n","colin        NaN\n","darwin     136.0\n","dtype: float64"]},"metadata":{},"execution_count":56}],"source":["print(s2.keys())\n","print(s3.keys())\n","\n","s2 + s3"]},{"cell_type":"markdown","metadata":{"id":"6JMZev9LbdkN"},"source":["The resulting `Series` contains the union of index labels from `s2` and `s3`. Since `\"colin\"` is missing from `s2` and `\"charles\"` is missing from `s3`, these items have a `NaN` result value. (ie. Not-a-Number means *missing*).\n","\n","Automatic alignment is very handy when working with data that may come from various sources with varying structure and missing items"]},{"cell_type":"markdown","source":["#### Initialize with a scalar"],"metadata":{"id":"1eEUnpU7prjd"}},{"cell_type":"markdown","metadata":{"id":"H57kzHKabdkN"},"source":["You can also initialize a `Series` object using a scalar and a list of index labels: all items will be set to the scalar."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fz5q8tyGbdkN","outputId":"70b47999-f01f-42a2-c5ac-cc6ba0eb688d","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677380232112,"user_tz":-480,"elapsed":281,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["life          42\n","universe      42\n","everything    42\n","dtype: int64"]},"metadata":{},"execution_count":29}],"source":["meaning = pd.Series(42, [\"life\", \"universe\", \"everything\"])\n","meaning"]},{"cell_type":"markdown","metadata":{"id":"lQUC1nqObdkO"},"source":["Pandas makes it easy to plot `Series` data using matplotlib:"]},{"cell_type":"code","execution_count":null,"metadata":{"scrolled":true,"id":"IQFkp_wlbdkO","outputId":"ec870a9a-7edd-4269-f7b6-e9b82da56ad8","colab":{"base_uri":"https://localhost:8080/","height":265},"executionInfo":{"status":"ok","timestamp":1677380275026,"user_tz":-480,"elapsed":905,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 432x288 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmg0lEQVR4nO3dd3jV5f3/8ec7mwwIJGGHGYbsERmCVNwDxbrrrvqlVlyt46t2t7baatW6f0rdmziwVnHiwIEm7E3YYSaMkEH2/fuDo19KGSE5yZ1z8npcV66c8THndYTrlQ/3uT/3bc45REQk9EX4DiAiIsGhQhcRCRMqdBGRMKFCFxEJEyp0EZEwEeXrhVNTU123bt18vbyISEjKyckpcM6l7e85b4XerVs3srOzfb28iEhIMrO1B3pOQy4iImFChS4iEiZU6CIiYUKFLiISJlToIiJhQoUuIhImVOgiImHC2zx08aukvIpvV29n+ZYiBnVOZljXZGKjIn3HEpF6UKE3E5XVNcxbv5OZuQV8lbuN2et2UFXzf2vhx0VHMKJ7CmMzUjiqZyr9OrQkIsI8JhaRw6VCD1POOZZvKWZmbgFf5hYwa9U2SiqqMYOBnVrxP+N6MDYjlT7tk5i7bucPx/3l3aUAtI6P5qiMVMZmpDKmZypdUuI9vyMRORQVehjZsHM3XwaK+cvcbRQUlwPQPTWBHw/rxNiMVEb1SCE5PuY//rvj+7Xj+H7tANiyq4yvVhYwc8U2vswt4N/zNwGQ3qYFYzNSOapnKkf1TCElMbZx35yIHJL52oIuMzPTaS2X+iksreTrVQWBs+ttrC4oASA1MYYxGak/fHVKblGnn++cY2V+SaDgC/h61TaKyqoA6NehJWN77Sn3Ed3bEB+jcwORxmBmOc65zP0+p0IPHWWV1eSs3fHD8MiCDYU4BwkxkYzskcKYwBBJ73aJmAV//LuquoYFGwr5auU2Zq4oIGftDiqqa4iONIZ1af3DL5DBnVsRFakJVCINQYUeoqprHIs2Fv5Q4NlrdlBeVUNUhDG0S/IPBT44PZloDwW6u6Ka7LXbf8i3aOMunIOk2KjAL5gUxmakktG2YX7BiDRHByt0/Tu5idlUuJuPlmzly8AQR+HuSgD6tk/i4lFdGZuRypHd25AY6/+PrkVMJEf3SuPoXnuWZt5RUsHXq7b9UPAfLdkCQNuk2D3j7xmp9G2fREQjlXtkhNGrbaJm60izoTP0JqRwdyVH//UTdpVV0Sm5BWMy9gyjHNUzlbSk0PsQcv320j3j77nb+Cq3gG0lFY2e4Zg+aTxxSSYxURoCkvCgM/QQ8c78jewqq+KFK0cyJiMl5Icp0tvEc36bLpx/ZBdqahxLNxexbntpo73+ii1F/P3D5dz46hwevGCoxvUl7KnQm5Cp2Xn0aZcUFmW+r4gIo1/HlvTr2LLRXvPkAe1pERPJnf9eQnzMAv529iANv0hYU6E3Eblbi5i7fie/OvWIsCtzn646ugfF5VU88NEKEmOj+N3p/fT/V8KWCr2JmJqTR2SEcebQTr6jhJ0bjutFUVkV/5y5mqS4KG46sY/vSCINolaDimaWbGZZZrbUzJaY2eh9njcze9DMcs1svpkNa5i44amquoY3Zm9gfJ+2IfnhZ1NnZvz6tCO44Mh0Hvokl8c/W+k7kkiDqO0Z+j+A6c65c8wsBth3YY9TgF6Br5HAY4HvUgufr8gnv6icczM7+44StsyMP/94IMXlVdz93lISY6O4eFRX37FEguqQhW5mrYBxwOUAzrkKYN/5ZxOB59yeOZDfBM7oOzjnNgU5b1jKysmjTUIM4/u09R0lrEVGGPefP4TdFdX8ZtpCEmOjNMQlYaU2Qy7dgXzgaTObY2ZTzCxhn2M6Aev3up8XeOw/mNkkM8s2s+z8/Pw6hw4nO0oq+GjxVs4c0klzpRtBdGQEj1w0jFHdU7hp6jw+WLTZdySRoKlNg0QBw4DHnHNDgRLgtrq8mHPuCedcpnMuMy0trS4/IuxMm7uBiuoaDbc0orjoSJ68LJOBnVpx7UtzmLmiwHckkaCoTaHnAXnOuVmB+1nsKfi9bQDS97rfOfCYHMLUnDwGdGrJER0ab362QGJsFM/89Eh6pCXwP89lk7N2u+9IIvV2yEJ3zm0G1pvZ93O9jgMW73PY28Clgdkuo4BCjZ8f2uKNu1i0cRfnDNPZuQ/J8TE8f+VI2reK4/Knv2PhhkLfkUTqpbaDttcBL5rZfGAI8Bczu9rMrg48/y6wCsgFngSuCXbQcJSVk0dMZAQTh+iDOV/SkmJ54aqRJMVGcelT35K7tch3JJE60+JcnlRU1TDqro8Z1aMNj1403HecZm91QQnnPv41URHG1KtHk95GW+5J03Swxbk0rcKTT5ZuZXtJBecOTz/0wdLguqcm8PyVI9hdWc1FU2axZVeZ70gih02F7klWTh5tk2I5uleq7ygScESHljzz0yPZVlzOxVNmsd3Dcr8i9aFC9yC/qJwZy7by42GdtKRrEzO0S2umXHYk67aXctlT37KrrNJ3JJFaU5t48NacDVTXOA23NFGje6bw2MXDWLJpF1c9k83uimrfkURqRYXeyJxzTM1Zz9AuyWS0TfQdRw7g2L7teOCCIWSv3c7PXsihvEqlLk2fCr2Rzc8rZPmWYs4ZrrnnTd2EQR25+6xBfL48nxtfmUtVdY3vSCIHpUJvZFk5ecRGRXD64I6+o0gtnHdkOr+Z0I/3Fm7mf19fQE2Nn2m+IrWhDS4aUVllNdPmbuDkAe1pGRftO47U0pVju1NcVsX9Hy0nMTaS35/RX7seSZOkQm9EHy7ewq6yKn0YGoKuPy6D4vJKnvxiNYlxUdxyUl/fkUT+iwq9EU3NyaNjqzhG90zxHUUOk5lxx6lHUFxexSMzVpIQG8U1x2T4jiXyH1TojWRzYRkzV+QzeXwGkdp5PiSZGXeeOZCS8mr+Nn0ZSbFRXDK6m+9YIj9QoTeS12fnUePQ7JYQFxlh/P28wZRWVPGbaYuIj4nibP2ZShOhWS6NwDlHVk4eI7q3oWvKvps9SaiJjozg4QuHcVTPFG7Jmsf0hdr1SJoGFXojyFm7g9UFJTo7DyNx0ZE8eWkmg9OTuf7lOXy+XFsqin8q9EaQlZNHfEwkpw3s4DuKBFFCbBTPXD6Cnm0TmfR8Nt+t0a5H4pcKvYGVVlTxzvxNnDqwAwmx+sgi3LSKj+a5K0bQsVULrnj6O3K3FvuOJM2YCr2BTV+4meLyKs7VcEvYSkuK5fmrRhITFcHkF2drMS/xRoXewKZm59GlTTwjurfxHUUaUKfkFtx//hCWby3it9MW+o4jzZQKvQGt317K16u2cc7wzrpUvBkY1zuN68ZnMDUnj6nZ633HkWZIhd6AXp+dhxmap9yM3HB8b0b3SOE30xaybLM2nJbGpUJvIDU1e+aej+mZSqfkFr7jSCOJjDD+8ZMhJMZG8/MXcygpr/IdSZoRFXoD+Wb1NvJ27Nbc82aobVIcD/5kCGsKSrjjzQU4pyV3pXGo0BtIVk4eSbFRnNS/ve8o4sFRPVP5xfG9mTZ3Iy9/q/F0aRwq9AZQXF7Fews2M2FwR1rERPqOI55MHp/BuN5p/P5fi1i4odB3HGkGVOgN4N/zN7K7sppzMzXc0pxFRBj3nzeYNvExXPvSbHaVVfqOJGFOhd4Apmbn0SMtgaHpyb6jiGcpibE8dOFQ1u/YzW2vz9d4ujQoFXqQrS4oIXvtDs4dnq655wLAkd3acMtJfXh3wWae+3qt7zgSxmpV6Ga2xswWmNlcM8vez/PHmFlh4Pm5Zvbb4EcNDVk564kwOGtYJ99RpAmZdHQPjuvbljv/vZh563f6jiNh6nDO0Mc754Y45zIP8PwXgeeHOOf+GIxwoaa6xvF6zgZ+1DuNdi3jfMeRJiQisDFG26Q4Jr80m8JSjadL8GnIJYhm5haweVcZ52gTaNmP5PgYHr5wKFt2lXFz1jyNp0vQ1bbQHfCBmeWY2aQDHDPazOaZ2Xtm1n9/B5jZJDPLNrPs/Pzw2xAgKyeP5Phoju/X1ncUaaKGdmnNbaccwYeLt/DPmat9x5EwU9tCH+ucGwacAkw2s3H7PD8b6OqcGww8BLy1vx/inHvCOZfpnMtMS0ura+YmqbC0kvcXbWbi4I7ERmnuuRzYFWO6cVL/dtz93lJy1u7wHUfCSK0K3Tm3IfB9K/AmMGKf53c554oDt98Fos0sNchZm7S352+koqpGwy1ySGbG384ZTIfkOK59aTbbSyp8R5IwcchCN7MEM0v6/jZwIrBwn2PaW2COnpmNCPzcbcGP23RlZa+nb/skBnRq6TuKhIBWLaJ59MLhbCuu4JevzaWmRuPpUn+1OUNvB8w0s3nAt8C/nXPTzexqM7s6cMw5wMLAMQ8CF7hm9InP8i1FzMsr1LrnclgGdm7FbyYcwafL8nn885W+40gYOOQml865VcDg/Tz++F63HwYeDm600JGVk0dUhHHmUM09l8Nz8aiuzFq9nXvfX8bwLq0Z2SPFdyQJYZq2WE+V1TW8MXsD4/u2JTUx1nccCTFmxl1nDaRrSgLXvTyHguJy35EkhKnQ6+mzZfkUFJdrE2ips6S4aB65cBiFuyu58ZW5VGs8XepIhV5PWTl5pCbGML6v5p5L3fXr2JI/nNGfmbkFPPxJru84EqJU6PWwvaSCj5du4cwhnYiO1P9KqZ/zj0znrKGdeODj5XyZW+A7joQgtVA9vDVnA5XVjnO07rkEgZlx548H0DMtkRtemcPWXWW+I0mIUaHXw9ScPAZ2akXf9pp7LsERHxPFYxcNo6S8mutenkNVdY3vSBJCVOh1tGhjIUs27dKuRBJ0vdolceeZA5i1ejv3f7TcdxwJISr0OpqanUdMZARnDO7oO4qEobOHd+b8zHQembGSGcu2+o4jIUKFXgcVVTVMm7uBE/q1Izk+xnccCVN/mNifvu2T+OWrc9m4c7fvOBICVOh18PGSLewordSHodKg4qIjefSiYVRU1XDdy3Oo1Hi6HIIKvQ6ycvJo1zKWcb3CawlgaXp6pCVy99mDyFm7g3veX+Y7jjRxKvTDtLWojE+X53PWsM5ERmghLml4pw/uyMWjuvDE56v4cPEW33GkCVOhH6Y3Z2+gusZxji71l0b069P6MaBTS256bS7rt5f6jiNNlAr9MDjnmJqTx7AuyfRMS/QdR5qRuOhIHrlwGM7BtS/NpqJK4+ny31Toh2FeXiG5W4s5N1O7Eknj65qSwD3nDmJeXiF/eXeJ7zjSBKnQD8PU7PXERUdw2qAOvqNIM3XygA78dEw3nvlqDe8t2OQ7jjQxKvRaKqus5u15Gzm5f3taxkX7jiPN2O2nHMGQ9GRuzZrPmoIS33GkCVGh19L7izZTVFal4RbxLiYqgocvHEpEhDH5pdmUVVb7jiRNhAq9lrJy8uiU3ILR2iJMmoDOreO577zBLNq4iz+9s9h3HGkiVOi1sHHnbmbmFnD28M5EaO65NBHHHdGOn43rwYuz1jFt7gbfcaQJUKHXwhuz83AOzhmmuefStNx8Uh8yu7bmjjcWsDK/2Hcc8UyFfgjOObJy8hjZvQ1dUuJ9xxH5D9GRETx04VBioyOZ/OJsdldoPL05U6EfwndrdrBmW6k+DJUmq0OrFtx33mCWbSnid28v9B1HPFKhH8LL364jISaSUwe29x1F5ICO6dOWycdk8Fp2Hlk5eb7jiCcq9INYtrmIaXM38JMRXYiPifIdR+Sgbjy+F6N6tOHXby1g+ZYi33HEAxX6Qfxt+lISYqOYPD7DdxSRQ4qKjODBC4aSGBvNNS/OpqS8ynckaWQq9AOYtWobHy/dyjXHZNA6QbsSSWho2zKOBy8Ywsr8Yn791kKcc74jSSNSoe+Hc467py+lfcs4fjqmm+84IoflqIxUbjyuN2/O2cCr3633HUcaUa0K3czWmNkCM5trZtn7ed7M7EEzyzWz+WY2LPhRG8/7izYzZ91OfnFCL+KiI33HETls1x6bwdG9Uvnt24tYvHGX7zjSSA7nDH28c26Icy5zP8+dAvQKfE0CHgtGOB+qqmv42/RlZLRN5GxdSCQhKjLCuP/8IbSOj2byS7MpKqv0HUkaQbCGXCYCz7k9vgGSzSwk15h9LTuPVQUl3HpSH6IiNSIloSs1MZYHLxjKuu2l3PbGAo2nNwO1bSwHfGBmOWY2aT/PdwL2HqzLCzz2H8xskpllm1l2fn7+4adtYKUVVTzw0XIyu7bmhH7tfMcRqbeRPVK46cTe/Hv+Jl74Zq3vONLAalvoY51zw9gztDLZzMbV5cWcc0845zKdc5lpaWl1+REN6ukv17C1qJzbTumLmRbhkvBw9biejO+Txp/eWcKCvELfcaQB1arQnXMbAt+3Am8CI/Y5ZAOw97XxnQOPhYztJRU8/ulKTujXjsxubXzHEQmaiAjjvvOGkJoYwzUv5VC4W+Pp4eqQhW5mCWaW9P1t4ERg3wUj3gYuDcx2GQUUOudCan+shz/JpaSiiltP6uM7ikjQtU6I4aELh7FpZxm3Zs3TeHqYqs0ZejtgppnNA74F/u2cm25mV5vZ1YFj3gVWAbnAk8A1DZK2gazfXsrz36zh3OHp9GqX5DuOSIMY3rU1t53Sl/cXbeGpL9f4jiMN4JALlDjnVgGD9/P443vddsDk4EZrPPd9uJwIM35xQm/fUUQa1JVjuzNr9XbuencJQ7skM6xLa9+RJIia/by8RRsLeWvuBq4Y2532reJ8xxFpUGbGvecMpn2rOK57aQ47Syt8R5IgavaF/tfpy2gZF83VP+rpO4pIo2gVH82jFw0jv6icm16bR02NxtPDRbMu9C9zC/h8eT7Xjs+gVYto33FEGs2gzsn86rQj+HjpVp74YpXvOBIkzbbQa2ocd7+3lE7JLbhkdFffcUQa3aWju3LawA7c8/4yvluz3XccCYJmW+jvLtzEgg2F/PKE3lqAS5olM+PusweS3roF1740m23F5b4jST01y0KvrK7hnveX0bd9EmcO/a8VCkSajaS4aB65aBg7Siu58dW5Gk8Pcc2y0F/+dh1rt5Xyvyf3JTJCl/hL89a/Yyt+f3p/vlhRwCMzcn3HkXpodoVeXF7Fgx+vYGT3NhzTp+mtJyPiw09GpHPmkI7c/9FyvlpZ4DuO1FGzK/QpX6yioLhCC3CJ7MXM+POPB9I9NYHrX57L1qIy35GkDppVoecXlfPk56s4dWB7huoKOZH/kBAbxaMXDae4vJIbXp5LtcbTQ06zKvSHP1lBWVUNN5+oBbhE9qdP+yT+NHEAX6/axj8+Wu47jhymZlPoawpKeHHWOi44Mp0eaYm+44g0WedmpnPu8M48NCOXz5c3vY1o5MCaTaHf+8EyoiMjuOG4Xr6jiDR5f5w4gN5tk7jx1blsLtR4eqhoFoU+P28n78zfxFVHd6dtSy3AJXIoLWIieeSiYZRVVnPdy7Opqq7xHUlqIewL3bk9l/i3SYhh0rgevuOIhIyMtoncddZAvluzg3s/0Hh6KAj7Qv98RQFfrdzGdcdmkBSnBbhEDsfEIZ24cGQXHv9sJTOWbfUdRw4hrAv9+wW40tu04MKRXXzHEQlJv53Qjz7tkrjjjQUUl1f5jiMHEdaF/va8jSzZtIubT+xDbJQW4BKpi7joSO46eyCbd5Xx9w+W+Y4jBxG2hV5eVc29Hyyjf8eWnD6oo+84IiFtWJfWXDKqK898tYZ563f6jiMHELaF/uI368jbsZvbTulLhBbgEqm3W07qQ9ukWG5/Y4FmvTRRYVnou8oqeeiTFYzNSOXoXlqASyQYkuKi+cMZ/Vm8aRdPfbnadxzZj7As9Cc+W8WO0kr+9+S+vqOIhJWT+rfn+CPacf+HK1i/vdR3HNlH2BX61l1lTJm5itMHd2Rg51a+44iEFTPjjxP7E2Hw67cW4pwW8GpKwq7QH/h4BdU1jptP7O07ikhY6pjcgptP6sNny/P51/xNvuPIXsKq0FfmF/Pqd+u5aGRXuqYk+I4jErYuHd2NwZ1b8cd/LaKwtNJ3HAkIq0K/Z/oy4qIiuPbYDN9RRMJaZITxl7MGsqO0krveW+I7jgSETaHPXreD6Ys2M2lcT1ITY33HEQl7/Tu24sqx3Xnlu/V8u3q77zhCmBS6c467311KamIMVx3d3XcckWbjxuN70bl1C25/Yz7lVdW+4zR7tS50M4s0szlm9s5+nrvczPLNbG7g66rgxjy4Gcu28u2a7dxwXC8SYqMa86VFmrX4mCjuPHMAK/NLePzTVb7jNHuHc4Z+A3CwwbJXnXNDAl9T6pmr1qprHH99bxndUuK5YIQW4BJpbMf0acvpgzvyyIxccrcW+47TrNWq0M2sM3Aa0GhFXVtvzM5j2ZYibjmpL9GRYTGCJBJyfjuhH3HREfzqzQWam+5RbRvwAeBW4GALOJxtZvPNLMvM0vd3gJlNMrNsM8vOz6//XoVlldXc9+FyBnduxakD29f754lI3aQlxXLHqUcwa/V2pmbn+Y7TbB2y0M1sArDVOZdzkMP+BXRzzg0CPgSe3d9BzrknnHOZzrnMtLT6r7Hy3Ndr2FRYxv+e0hczLcAl4tN5memM6NaGP7+7hILict9xmqXanKGPAc4wszXAK8CxZvbC3gc457Y5577/E5wCDA9qyv0oLK3kkRkr+VHvNI7qmdrQLycihxARYfzlrAGUVlTxp3cW+47TLB2y0J1ztzvnOjvnugEXAJ845y7e+xgz67DX3TM4+IenQfHoZ7nsKtMCXCJNSUbbJH5+TAbT5m7ks+X1H1aVw1PnTxHN7I9mdkbg7vVmtsjM5gHXA5cHI9yBbNy5m6e/XMOPh3SiX8eWDflSInKYrjmmJz3SEvj1WwvYXaG56Y3psArdOfepc25C4PZvnXNvB27f7pzr75wb7Jwb75xb2hBhv/fAR8vBwS9O0AJcIk1NXHQkf/nxQNZv380DHy/3HadZCbl5fsu3FJGVk8clo7uS3ibedxwR2Y9RPVI4L7MzU75YzeKNu3zHaTZCrtC37iono20ik8drAS6RpuyOU48guUU0t7+5gOoazU1vDCFX6GN7pfL+jeNokxDjO4qIHERyfAy/Pb0f89bv5Pmv1/iO0yyEXKEDmnMuEiLOGNyRo3ulcs/7y9hUuNt3nLAXkoUuIqHBzPjzmQOpdo7fTVvkO07YU6GLSIPqkhLPDcf15oPFW5i+cLPvOGFNhS4iDe6qo7vTt30Sv397EUVl2rKuoajQRaTBRUdGcPfZg9hSVMa97y/zHSdsqdBFpFEMSU/m0lFdee6btcxZt8N3nLCkQheRRnPzSX1olxTH7W8soLL6YKtxS12o0EWk0STFRfOHif1ZurmIKV+s9h0n7KjQRaRRndS/PSf2a8c/Pl7Oum2lvuOEFRW6iDS6P0zsT1REBL96S1vWBZMKXUQaXYdWLbj5xN58saKAaXM3+o4TNlToIuLFJaO7MTg9mT+9s5idpRW+44QFFbqIeBEZYdx91kB27q7kL+82+CZnzYIKXUS8OaJDS646ujuvZefx9cptvuOEPBW6iHh143G9SW/Tgl+9uYCySm1ZVx8qdBHxqkVMJHeeOZBVBSU8+ulK33FCmgpdRLz7Ue80Jg7pyGOf5pK7tch3nJClQheRJuE3E/oRHxPF7W8soEZb1tWJCl1EmoTUxFjuOLUv363ZwavZ633HCUkqdBFpMs7LTGdk9zbc9e4SPly8RVeRHiYVuog0GWbGX88eREpiLP/zXDZnP/YV36zSdMbaUqGLSJPSLTWBD34xjrvOGsiGnbu54IlvuPSpb1m4odB3tCbPfP2TJjMz02VnZ3t5bREJDWWV1Tz71Roe/XQlhbsrmTCoAzed2IfuqQm+o3ljZjnOucz9PqdCF5GmrnB3JU9+vop/zlxNRXUN52V25vrjetGhVQvf0RrdwQq91kMuZhZpZnPM7J39PBdrZq+aWa6ZzTKzbvXIKyLyH1q1iObmk/rw+a3juWRUV7Jy8jjmnk+5690l7CjRwl7fO5wx9BuAA62gcyWwwzmXAdwP/LW+wURE9pWWFMvvz+jPJzcdw2kDO/DEF6sY97cZPPTxCkrKq3zH865WhW5mnYHTgCkHOGQi8GzgdhZwnJlZ/eOJiPy39Dbx3Hf+EKbfMI5RPVP4+4fL+dE9M3jmy9WUVzXf9WBqe4b+AHArcKBdXTsB6wGcc1VAIZCy70FmNsnMss0sOz8///DTiojspU/7JJ68NJPXf34UPdMS+f2/FnPc3z/j9Zw8qpvh1aaHLHQzmwBsdc7l1PfFnHNPOOcynXOZaWlp9f1xIiIADO/amlcmjeLZK0bQqkU0N02dxyn/+JwPFm1uVhcn1eYMfQxwhpmtAV4BjjWzF/Y5ZgOQDmBmUUArQFcDiEijMTN+1DuNf107locvHEpltWPS8zmc9dhXzWat9UMWunPududcZ+dcN+AC4BPn3MX7HPY2cFng9jmBY5rPr0URaTIiIowJgzr+cHHSpp1l/OTJ5nFxUp2vFDWzP5rZGYG7/wRSzCwX+CVwWzDCiYjUVXRkBD8Z0YVPbzmGO07ty/y8nUx4aCaTX5rNqvxi3/EahC4sEpFmYVfZ/12cVF4Vuhcn6UpREZGA/KJyHpmRy4uz1mJmXH5UN37+o560TojxHa1WgnKlqIhIONj74qQJgzrw5BerOOUfX7B+e6nvaPWmQheRZim9TTz3nTeEaZPHsLuymoumzGLLrjLfsepFhS4izdqgzsk8e8UIthWXc/GUWWwP4bVhVOgi0uwNSU9mymVHsm57KZc99S27yip9R6oTFbqICDC6ZwqPXTyMJZt2ceUz37G7IvTWhFGhi4gEHNu3HQ9cMISctTuY9Hx2yC30pUIXEdnLhEEdufusQXyxooAbXp5LVfWB1iRselToIiL7OO/IdH4zoR/TF23m1tfnUxMiKzdG+Q4gItIUXTm2OyXlVdz34XISY6P4wxn9aerbPKjQRUQO4LpjMygur+KJz1eRFBfFLSf19R3poFToIiIHYGbcfkpfisqqeGTGShJio7jmmAzfsQ5IhS4ichBmxp1nDqC0ooq/TV9GUmwUl4zu5jvWfqnQRUQOITLCuPfcwZSUV/ObaYuIj4ni7OGdfcf6L5rlIiJSC9GRETx84VCO6pnCLVnzmL5wk+9I/0WFLiJSS3HRkTx5aSaD05O57uU5fLa8aW12r0IXETkMCbFRPHP5CDLaJvGz57P5bs1235F+oEIXETlMreKjef7KEXRMbsEVT3/HgrymsVepCl1EpA5SE2N54cqRtGwRzaVPzWLFliLfkVToIiJ11TG5BS9eNZKoyAgumjKLddv87nqkQhcRqYduqQm8cOVIKqpruHDKN2wu9LfrkQpdRKSe+rRP4tmfjmBnaSUXTfmGbcXlXnKo0EVEgmBwejJTLsskb8duLvW065EKXUQkSEb1SOHxS4azfEsRVzz9HaUVVY36+ip0EZEgGt+nLQ+cP5TZ63bws+dzGnXXIxW6iEiQnTaoA3efvWfXo+temtNoux6p0EVEGsB5men87vR+fLB4C7dkNc6uR1ptUUSkgfx0zJ5dj+79YDkJsZH8aeKABt316JCFbmZxwOdAbOD4LOfc7/Y55nLgHmBD4KGHnXNTghtVRCT0TB6fQVF5Ff/vs1UkxEZx28l9G6zUa3OGXg4c65wrNrNoYKaZveec+2af4151zl0b/IgiIqHLzLjt5L4Ul+0p9ZZx0Uwe3zC7Hh2y0J1zDigO3I0OfIXGFtgiIk2AmfGniQMoKa/inveXkRATyeVjugf9dWr1oaiZRZrZXGAr8KFzbtZ+DjvbzOabWZaZpR/g50wys2wzy87Pb1rrCIuINKSICOOecwdz+uCOdEmJb5DXsD0n4LU82CwZeBO4zjm3cK/HU4Bi51y5mf0MON85d+zBflZmZqbLzs6uW2oRkWbKzHKcc5n7e+6wpi0653YCM4CT93l8m3Pu+8ULpgDD65BTRETq4ZCFbmZpgTNzzKwFcAKwdJ9jOux19wxgSRAziohILdRmlksH4Fkzi2TPL4DXnHPvmNkfgWzn3NvA9WZ2BlAFbAcub6jAIiKyf4c1hh5MGkMXETl8QRtDFxGRpkuFLiISJlToIiJhQoUuIhImvH0oamb5wNo6/uepQEEQ4zQ14fz+9N5CVzi/v1B6b12dc2n7e8JbodeHmWUf6FPecBDO70/vLXSF8/sLl/emIRcRkTChQhcRCROhWuhP+A7QwML5/em9ha5wfn9h8d5CcgxdRET+W6ieoYuIyD5U6CIiYSLkCt3MTjazZWaWa2a3+c4TLGaWbmYzzGyxmS0ysxt8Zwq2wM5Xc8zsHd9Zgs3MkgO7dS01syVmNtp3pmAxs18E/k4uNLOXAxvHhywze8rMtprZ3pv0tDGzD81sReB7a58Z6yqkCj2whO8jwClAP+AnZtbPb6qgqQJucs71A0YBk8PovX3vBsJ3rfx/ANOdc32BwYTJ+zSzTsD1QKZzbgAQCVzgN1W9PcM+m/QAtwEfO+d6AR8H7oeckCp0YASQ65xb5ZyrAF4BJnrOFBTOuU3OudmB20XsKYROflMFj5l1Bk5jz45WYcXMWgHjgH8COOcqArt7hYsooIWZRQHxwEbPeerFOfc5e/Zt2NtE4NnA7WeBMxszU7CEWqF3AtbvdT+PMCq975lZN2AosL/NuEPVA8CtQI3nHA2hO5APPB0YUppiZgm+QwWDc24DcC+wDtgEFDrnPvCbqkG0c85tCtzeDLTzGaauQq3Qw56ZJQKvAzc653b5zhMMZjYB2Oqcy/GdpYFEAcOAx5xzQ4ESQvSf7PsKjCVPZM8vrY5Agpld7DdVw3J75nKH5HzuUCv0DUD6Xvc7Bx4LC2YWzZ4yf9E594bvPEE0BjjDzNawZ5jsWDN7wW+koMoD8pxz3/+LKos9BR8OjgdWO+fynXOVwBvAUZ4zNYQt3++NHPi+1XOeOgm1Qv8O6GVm3c0shj0fzrztOVNQmJmxZwx2iXPuPt95gsk5d7tzrrNzrht7/sw+cc6FzVmec24zsN7M+gQeOg5Y7DFSMK0DRplZfODv6HGEyQe++3gbuCxw+zJgmscsdVabTaKbDOdclZldC7zPnk/bn3LOLfIcK1jGAJcAC8xsbuCxO5xz7/qLJIfhOuDFwInGKuCnnvMEhXNulpllAbPZMxNrDiF+mbyZvQwcA6SaWR7wO+Bu4DUzu5I9y3qf5y9h3enSfxGRMBFqQy4iInIAKnQRkTChQhcRCRMqdBGRMKFCFxEJEyp0EZEwoUIXEQkT/x81wYwtxc24QAAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}],"source":["temperatures = [4.4,5.1,6.1,6.2,6.1,6.1,5.7,5.2,4.7,4.1,3.9,3.5]\n","s4 = pd.Series(temperatures, name=\"Temperature\")\n","s4.plot()\n","plt.show()"]},{"cell_type":"markdown","source":["You can easily convert it to NumPy array by dicarding the index. "],"metadata":{"id":"jiCCsIfv4LoD"}},{"cell_type":"code","source":["s4.to_numpy()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"8FykEnJJ4HQl","outputId":"8977fff5-2546-4a2c-c6b9-68c3cb951801","executionInfo":{"status":"ok","timestamp":1677380287463,"user_tz":-480,"elapsed":282,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([4.4, 5.1, 6.1, 6.2, 6.1, 6.1, 5.7, 5.2, 4.7, 4.1, 3.9, 3.5])"]},"metadata":{},"execution_count":31}]},{"cell_type":"markdown","metadata":{"id":"JLC98XwVbdkO"},"source":["There are *many* options for plotting your data. It is not necessary to list them all here: if you need a particular type of plot (histograms, pie charts, etc.), just look for it in the excellent [Visualization](http://pandas.pydata.org/pandas-docs/stable/visualization.html) section of pandas' documentation, and look at the example code."]},{"cell_type":"markdown","source":["### Handling time"],"metadata":{"id":"5Dl3wpxgqIpm"}},{"cell_type":"markdown","metadata":{"id":"jwz5rnEsbdkO"},"source":["\n","Many datasets have timestamps, and pandas is awesome at manipulating such data:\n","* it can represent periods (such as 2016Q3) and frequencies (such as \"monthly\")\n","* it can convert periods to actual timestamps, and *vice versa*\n","* it can resample data and aggregate values any way you like\n","* it can handle timezones."]},{"cell_type":"markdown","source":["#### Time range"],"metadata":{"id":"uc065Y3XqNIO"}},{"cell_type":"markdown","source":["Let's start by creating a time series using `pd.date_range()`. This returns a `DatetimeIndex` containing one datetime per hour for 12 hours starting on March 6th 2023 at 5:30pm."],"metadata":{"id":"x5qxWaoRqLzG"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"becHbUssbdkO","outputId":"a991f954-d881-4056-9d55-3dff47422939","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677380416195,"user_tz":-480,"elapsed":287,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["DatetimeIndex(['2023-03-06 17:30:00', '2023-03-06 18:30:00',\n","               '2023-03-06 19:30:00', '2023-03-06 20:30:00',\n","               '2023-03-06 21:30:00', '2023-03-06 22:30:00',\n","               '2023-03-06 23:30:00', '2023-03-07 00:30:00',\n","               '2023-03-07 01:30:00', '2023-03-07 02:30:00',\n","               '2023-03-07 03:30:00', '2023-03-07 04:30:00'],\n","              dtype='datetime64[ns]', freq='H')"]},"metadata":{},"execution_count":32}],"source":["dates = pd.date_range('2023/03/06 5:30pm', periods=12, freq='H')\n","dates"]},{"cell_type":"markdown","metadata":{"id":"Zhslg2XVbdkP"},"source":["This `DatetimeIndex` may be used as an index in a `Series`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ojYFJeizbdkP","outputId":"c2cf0b79-4b42-4d0a-8140-7e49264f902f","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677380446948,"user_tz":-480,"elapsed":294,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["2023-03-06 17:30:00    4.4\n","2023-03-06 18:30:00    5.1\n","2023-03-06 19:30:00    6.1\n","2023-03-06 20:30:00    6.2\n","2023-03-06 21:30:00    6.1\n","2023-03-06 22:30:00    6.1\n","2023-03-06 23:30:00    5.7\n","2023-03-07 00:30:00    5.2\n","2023-03-07 01:30:00    4.7\n","2023-03-07 02:30:00    4.1\n","2023-03-07 03:30:00    3.9\n","2023-03-07 04:30:00    3.5\n","Freq: H, dtype: float64"]},"metadata":{},"execution_count":33}],"source":["temp_series = pd.Series(temperatures, dates)\n","temp_series"]},{"cell_type":"markdown","metadata":{"id":"O_vVwZqWbdkP"},"source":["Let's plot this series:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"gOa_KJ_JbdkP","outputId":"dd35e6d4-ffe9-487e-e57d-f30cf04f8174","colab":{"base_uri":"https://localhost:8080/","height":360},"executionInfo":{"status":"ok","timestamp":1677380458904,"user_tz":-480,"elapsed":693,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 432x288 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAWoAAAFXCAYAAACLPASQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAf2UlEQVR4nO3dfZRcdZ3n8feXJDghwQCijRIljI4ia5AxDTgHj5tGEZDgw66OsKv4NCfruCjsYY6E3XHUcRCcM/iwe4Z1GXRGZ2fsdRhdWUAeRokcXQl0AtqGAKIGJGuCPMYggtHP/nFvm0qnuruquPd3f3XzeZ1TJ133Vtfn/r5d/e3KrXt/NyRhZmb52qfpDTAzs9m5UZuZZc6N2swsc27UZmaZc6M2M8vc/Dqe9OCDD9ayZcv6/r7HHnuMRYsWVb9BDWc5z3nO23vyBs1av379A5Ke2XWlpMpvK1as0CBuuOGGgb4v9yznOc95e0/eoFnAhGboqd71YWaWOTdqM7PMuVGbmWXOjdrMLHNu1GZmmXOjNjPLnBu1mVnm3KjNzDLnRm1mlrlaTiG3NJatuWrGdecu38k7Zli/+aJT69okM6uB31GbmWXOjdrMLHM97fqIiAOAy4CXAALeJek7NW7XUGr7rojU42t7Pc161es+6k8D10h6U0TsC+xX4zaZmVmHORt1RCwBXgm8A0DSk8CT9W6WmZlNiWIa1FkeEHE0cClwO/BSYD1wtqTHpj1uNbAaYGRkZMX4+HjfG7Njxw4WL17c9/cNoo6syS2PzrhuZCFse7z7uuWHLnFeBnmzSfnadN5w5w2aNTY2tl7SaLd1vTTqUeAm4HhJ6yLi08B2SR+c6XtGR0c1MTHR94auXbuWlStX9v19g6gja659qhdPdv8PTF37cJ1XnZSvTecNd96gWRExY6Pu5aiP+4D7JK0r718OvKzvrTAzs4HM2aglbQV+EhEvKhe9imI3iJmZJdDrUR/vA/6hPOLjR8A769skMzPr1FOjlnQb0HXfiZmZ1ctnJpqZZc6N2swsc27UZmaZc6M2M8uc56M2K3kSKMuV31GbmWXOjdrMLHNu1GZmmXOjNjPLnBu1mVnm3KjNzDLnRm1mljk3ajOzzLlRm5llzo3azCxzbtRmZplzozYzy5wbtZlZ5tyozcwy50ZtZpY5N2ozs8z5wgFmDfGFCqxXfkdtZpY5N2ozs8y1eteH/2tpZm3QU6OOiM3Az4FfAzsljda5UWZmtks/76jHJD1Q25aYmVlX3kdtZpa5kDT3gyJ+DDwMCPgfki7t8pjVwGqAkZGRFePj431vzI4dO1i8eHHf3zeTyS2PzrhuZCFse7z7uuWHLnGe81qXN5uqf/f25rxBs8bGxtbPtFu510Z9qKQtEfEs4HrgfZJunOnxo6OjmpiY6HtD165dy8qVK/v+vpnM9WHixZPd9/wM+mGi85yXc95sqv7d25vzBs2KiBkbdU+7PiRtKf+9H/gKcGzfW2FmZgOZs1FHxKKI2H/qa+A1wPfr3jAzMyv0ctTHCPCViJh6/D9KuqbWrTIzs9+as1FL+hHw0gTbYmZmXfjwPDOzzLlRm5llzo3azCxzbtRmZplr9ex5ZraLZ5McXn5HbWaWOTdqM7PMuVGbmWXOjdrMLHNu1GZmmXOjNjPLnBu1mVnm3KjNzDLnRm1mlrnkZyb67Cgzs/74HbWZWebcqM3MMudGbWaWOTdqM7PMuVGbmWXOjdrMLHNu1GZmmXOjNjPLnBu1mVnmej4zMSLmARPAFkmr6tskM2sDn4VcnX7eUZ8NbKprQ8zMrLueGnVELAVOBS6rd3PMzGy6kDT3gyIuBy4E9gf+pNuuj4hYDawGGBkZWTE+Pt71uSa3PDpjzshC2PZ493XLD10y53Y2meU85zmv2bzZ7Nixg8WLF1f+vFVmjY2NrZc02m3dnI06IlYBr5X03ohYyQyNutPo6KgmJia6rptrv9XFk913mw+y3ypllvOc57xm82azdu1aVq5cWfnzVpkVETM26l52fRwPvC4iNgPjwAkR8T/73gozMxvInI1a0vmSlkpaBpwOfEPSW2vfMjMzAxq4cICZWR0GORxwWA4F7KtRS1oLrK1lS8zMrCufmWhmljk3ajOzzLlRm5llzo3azCxzbtRmZplzozYzy5wbtZlZ5tyozcwy50ZtZpY5N2ozs8y5UZuZZc6TMpmZ9Sn19SD9jtrMLHNu1GZmmXOjNjPLnBu1mVnm3KjNzDLnRm1mljk3ajOzzLlRm5llzo3azCxzbtRmZplzozYzy5wbtZlZ5uZs1BHxOxFxc0R8NyI2RsRHUmyYmZkVepk97wngBEk7ImIB8K2I+Jqkm2reNjMzo4dGLUnAjvLugvKmOjfKzMx26WkfdUTMi4jbgPuB6yWtq3WrzMzst6J4w9zjgyMOAL4CvE/S96etWw2sBhgZGVkxPj7e9Tkmtzw64/OPLIRtj3dft/zQJT1vZxNZznOe84YvL6exjY2NrZc02m1dX40aICL+DPiFpL+a6TGjo6OamJjoum6uKyNcPNl9b8wgV0ZImeU85zlv+PJyGltEzNioeznq45nlO2kiYiFwInDHXN9nZmbV6OWoj2cDn4+IeRSN/UuSrqx3s8zMbEovR318D/j9BNtiZmZd+MxEM7PMuVGbmWXOjdrMLHNu1GZmmXOjNjPLnBu1mVnm3KjNzDLnRm1mljk3ajOzzLlRm5llzo3azCxzbtRmZplzozYzy5wbtZlZ5tyozcwy50ZtZpY5N2ozs8y5UZuZZc6N2swsc27UZmaZc6M2M8ucG7WZWebcqM3MMudGbWaWOTdqM7PMzdmoI+K5EXFDRNweERsj4uwUG2ZmZoX5PTxmJ3CupA0RsT+wPiKul3R7zdtmZmb08I5a0k8lbSi//jmwCTi07g0zM7NCSOr9wRHLgBuBl0jaPm3damA1wMjIyIrx8fGuzzG55dEZn39kIWx7vPu65Ycu6Xk7m8hynvOcN3x5OY1tbGxsvaTRbut6btQRsRj4JnCBpC/P9tjR0VFNTEx0XbdszVUzft+5y3dy8WT3vTGbLzq1p+1sKst5znPe8OXlNLaImLFR93TUR0QsAP4Z+Ie5mrSZmVWrl6M+AvgssEnSJ+rfJDMz69TLO+rjgbcBJ0TEbeXttTVvl5mZleY8PE/St4BIsC1mZtaFz0w0M8ucG7WZWebcqM3MMudGbWaWOTdqM7PMuVGbmWXOjdrMLHNu1GZmmXOjNjPLnBu1mVnm3KjNzDLnRm1mljk3ajOzzLlRm5llzo3azCxzbtRmZplzozYzy5wbtZlZ5tyozcwy50ZtZpY5N2ozs8y5UZuZZc6N2swsc27UZmaZm7NRR8TnIuL+iPh+ig0yM7Pd9fKO+u+Ak2veDjMzm8GcjVrSjcBDCbbFzMy6CElzPyhiGXClpJfM8pjVwGqAkZGRFePj410fN7nl0RlzRhbCtse7r1t+6JI5t7PJLOc5z3nDl5fT2MbGxtZLGu22rrJG3Wl0dFQTExNd1y1bc9WM33fu8p1cPDm/67rNF53aS3RjWc5znvOGLy+nsUXEjI3aR32YmWXOjdrMLHO9HJ73ReA7wIsi4r6IeHf9m2VmZlO670jpIOmMFBtiZmbdedeHmVnm3KjNzDLnRm1mljk3ajOzzLlRm5llzo3azCxzbtRmZplzozYzy5wbtZlZ5tyozcwy50ZtZpY5N2ozs8y5UZuZZc6N2swsc27UZmaZc6M2M8ucG7WZWebcqM3MMudGbWaWOTdqM7PMuVGbmWXOjdrMLHNu1GZmmXOjNjPLnBu1mVnmemrUEXFyRNwZEXdHxJq6N8rMzHaZs1FHxDzgr4FTgCOBMyLiyLo3zMzMCr28oz4WuFvSjyQ9CYwDr693s8zMbEpImv0BEW8CTpb0R+X9twHHSTpr2uNWA6vLuy8C7hxgew4GHhjg+waRMst5znPe3pM3aNZhkp7ZbcX8p7Y9u0i6FLj0qTxHRExIGq1ok7LJcp7znLf35NWR1cuujy3AczvuLy2XmZlZAr006luA34uIwyNiX+B04Ip6N8vMzKbMuetD0s6IOAu4FpgHfE7Sxpq25yntOsk4y3nOc97ek1d51pwfJpqZWbN8ZqKZWebcqM3MMudGbZaJiDgoIg5qejvaImU9685qbB91RATFWY+Hlou2ADerpg2KiJHOLEnb6sgps5KOrcxs7fjaXM+IeB7wl8CrgEeAAJ4OfANYI2lzTbnJXi8p81LWM2lWE406Il4DXAL8gF3HZC8FXgC8V9J1FWYdDXwGWDIt65Eya0NVWWVesrGVeUfT7vG1vZ7fAT4FXC7p1+WyecCbgXMkvbzivKNJO77UecnqmfRnJyn5DdgELOuy/HBgU8VZt1Gc8j59+cuB7w7z2PaS8bW9nj8YZN0Qja+19UyZVdkp5H2aD9zXZfkWYEHFWYskrZu+UNJNEbGo4ixIOzZo//jaXs/1EXEJ8HngJ+Wy5wJvB26tIS/1+Npcz2RZTTXqzwG3RMQ4uw/wdOCzFWd9LSKuAr4wLetM4JqKsyDt2KD942t7Pc8E3g18hN33wV9BO8bX5nomy2ryw8QjgdcxbYCSbq8h6xSKqVmnZ11ddVaZl2xsZV7bx9fqeqbWwOul1fVMofEzE6cOaZH0UKMbUoM2jw3Sj6+N9YyI+RTvyt7A7o3sq8BnJf2qoU0bSinrmTSriUbdcVjLCcCj1HsIzRLgfIq/6COAgPspinmRpEeqyirzko2tzGv7+Npezy9SHAHxeXbti19KsZ/zIElvqTgv9fhaW8+kP7uqP3Xt8dPS7wBvAeZ1LJtHsd/xpoqzrgXOAw7pWHYIsAa4bpjHtpeMr+31vGuQdUM0vtbWM2lW1YXqcYApD6G5c5B1wzC2vWR8ba/nTRTH3e7TsWyf8o/TuhaMr7X1TJnV1Cnk6yPikog4LiKeU96OKw91qfoQmnsi4gPlmVFAcZZURJzHrk+hq5RybND+8bW9nqcDbwK2RcRdEXEXsBX4N+W6qqUeX5vrmSyrqX3U+1LshO/8JPg+4P9Q7IR/osKsAyn+m9W5j2wbxSE0H1fFH0ylHFuZ1/bxtbqe07KfASDpwRozUr9eWl3PVFmNH/VhZoWIOETS1pnuW39S1rPurMZnz4uIVbPdrzjrZbPdryEv2djK52/7+FpdT/Y8SaKOE15+q4HXS5vrWWtW440aOGaO+1X64znuVy3l2KD942t1PSWdOtv9GqR+vbS2nnVnedeHWcMi0k/j2mYp65kqq8lTyI+g+2mlm2rIWgKcPC3rWlV8sH1HXrKxlXltH19r6xmJp3EtM1O/XlpZz5RZjez6KA/NGac4y+zm8hbAFyNiTcVZZwIbgJXAfuVtjOKwrzOrzCrzko2tzGv7+FpdT+DTwKslnSLpj8rbycCJ5bpKNfB6aXM902VVfcB5jweK3wUs6LJ8X2o44QU4oMvyA6nnzK9kY9tLxtf2ev4AmD/D+O5uwfhaW8+UWU1Nc/ob4DnAPdOWP7tcV6WgOHaz2zZExVlTz5tqbND+8bW9nqmncU09vjbXM1lWU436HODrEfEDdg3weRT7ds6qOOsCYENEXDct60TgoxVnQdqxQfvHlzovaT0lXRgRX6WYxvUPysVbgH+veqZxTf16aW09U2Y1+WHiPuz5aektKq89VnHWgcBJ7PlhxsNVZ5V5ycZW5rV9fK2uZ2oNvF5aXc8Umm7USPpNeZrwS4DNSjDXcEQcVGdOk2Mr81s1vjbXszyi5ZMUuwLeD3yQYn7ju4C3q6YjW6ZtQ62vl5R5KeuZ9GdX9c78HnfCv4HifP+fUhx2tQ74OsUcDqdVnHU8xQVSNwLHAdcDP6T4b9gfDPPY9pLxtb2eNwKnAWdQ7Ic/nWLf7WnA11swvtbWM2lW1YXqcYC3UsxJeziwHXhRufwwYKLirJuB5RT7kB4AXlEufxnw7WEe214yvtbXs+Pru6et29CC8bW2nimzmvowEZUTlkTEvZLuLJfdM/Xf3AotkDRZZv1M0rfKrA0RsbDiLMrnTjU2aP/42l7PeR1ff2Laun1ryEs9vjbXM1lWY3N9dPySvatj2TyqL2bnGM+ftq6OX4SUY4P2j6/t9fzriFgMIOmSqYUR8QLgX2rISz2+NtczXVbV//Xo8b8MxwC/02X5MuCtFWe9Dtivy/LnAx8Y5rHtJeNrdT1T3xp4vbS6nqlunpTJLBMRsUrSlTPdt/6krGfdWY1PcxoRH57tfsVZq2e7X0Peh2e7X0Ne28eXOi9pPUk8jWsDr5c217PWrMYbNbB+jvtVmn7Kah2nsHZKOTZo//haXU9JH5rtfg1Sv15aW8+6s7zrw6xhqadxbbuIOBaQpFsi4kiKKVbvkHR1guwvSKp+FsImGnVEfAL4Z0nfTpB1EMWcEP+PYqKU/0xxTOcm4GOq4TTWiBgD/i3FBC2/pjhT6TJJd1edVeadRHFiSOcv+lclXVNH3izb8WeS/ryG5z2JYp7fr0va3LH8XZI+V3FWAG+mmEjocuAEiiZ6B/AZSZVOBFVO43oGxVSu95WLl1KcPDEu6aIq8+bYlndK+tsanvcIitfmOkk7OpafXPVrNCI+BJxCMY/R9RQn2dxAMbfItZIuqDDriumLKKZw/QaApNdVltVQo/4ZxZk8zwT+F/BFSbfWlHU1MAk8HXhx+fWXKH5wL5X0+orzLqQ4QePrFM3zxxSN+r0Ufxj+qeK8TwEvBL7A7r/oZ1JMA3p2lXlzbMu9kp5X8XN+DHgFxZzGpwGfkvTfynUbJFV63b2IuAR4FsWhY9uBp1FcMftUYFvV9YyIu4B/JelX05bvC2yU9HtV5s2xLXX8/N4P/EeKN0ZHA2dL+mq5ro6f32SZ8zRgK7BU0vbymO11ko6qMGsDcDtwGcUf9gC+SPFHFknfrCqrkUNNKM/ooWgwH6Q4vfQO4EPACyvOuq38N4At3dZVnDfZ8fV8yrOvKObf/X4NeV3n9C3HW8d8zdtnuP0c2FlHPSnn/AUOAK4GPtn5Oqrj5wcsAB4E9u34WX6vhrw7gMO6LD8MuLOGvO/NcJsEnqjp57e4/HoZMEHRrOv6+d3a7evy/m0VZ+0D/CeKd+5Hl8t+VPWYpObOTBSApLsopjr8aEQcRfFfwKspprCsyj7l7F37A4sjYpmkzRHxDOo54P43HZPOPIfy7CVJD5f/ra7aLyPiGEm3TFt+DPDLGvIeAY6RtG36ioj4yZ4Pf8rmS9oJIOmRiDgNuDQi/ol6fn5TWb+KiFskPVne3xkRdcx/fQ5pp3EdoZjJbvouvwD+bw15+6jc3VH+3q0ELo+Iw6jnw8QnI2I/Sb8AVkwtjOJyYJX+/FTsBvtk+Vr8ZETcT01TRzfVqPf4AUma+ss+/eylp+pCinctUJzZdllECDgS+EjFWQAfA24t/0v7IsorLUfEM4Hv1pD3DuC/R8T+7Nr18Vzg0XJd1b5A8W5vj0YN/GMNeT+MiH+t8r+RKqY2fXdE/AXF5wBV2xoRiyXtUHFZJQAi4hDgyarDJF0TES8k3TSuV1K8w71t+oqIWFtD3raIOHoqT9KOiFhFMen+8hryXinpiTKrszEvAN5eQx6S7gPeHBGnUvzeVa6pfdSL1fGhQoK8eRRj3RkR8yn2YW2R9NOa8g4CfpdiopZH6sjoknkIHb/oKufHGHZT80FIerzLukMlbdnzu2rZjkXAIkn31/Dcrb0KeUQspdgltsfrMSKOVw0HFKSsZ6qs7A7Pi4gjJN0x9yOHK6vuvIhYoD0/kDpY0gPOyzcvmrkKedI/DIkbZyuvQl75Tu8KdtDf28asuvIoDge6j2IKyeuAZR3r6pgm03nV5m3qzOhYfjiwqYa81wB3A1+jOFrhMuCactlrWpCXrJ4psxrZRx0R/3WmVRSf7A9lVhN5wF8CJ0naGBFvAq6PiLdJuol6PqxxXrXms+uzhU5bKParVu3TwKvVcTw6QEQcTvFB/ouHPC9lPZNlNfVh4juBc4Enuqw7Y4izmsjbV9JGAEmXR8Qm4MvliRR1/FfWedVKfRXy1H8YUue18irkTX2Y+A3gTyXtcThQRPxY0uHDmNVQ3gSwSh0f1pQf4FwJPF/S/s7LN698/hfT/RTyyq9CHhHnA39IcSbk9ObyJUkXDnNemZmynkmymmrUBwG/VHGsY2uyGsp7NfAzSd+dtnwJcJYqPGXWedXnNSFlI2sir42yO+rDzMx2l8M0p2ZmNgs3ajOzzLlRmzUoIr4cEW+N8iKp9tSkrGfKrEYadUScFREHl1+/ICJujIhHImJdRFR6/n/KLOc5bwDHUUyHe29EfCki3hjFFKe1SP2HoYE/RCnrmS6r6jODejyjZ2PH11cBbyy/Xkk5LegwZjnPeQPk3Vr++3TgbRQngfwM+FvqOXNvC8UFER6imJf9jZRTudZxayAvWT2TZtVVsDkGeGfH17dMW1fpnL8ps5znvAHy9jgtHXgG8B7gGzXk3Vr+m+oPQ+q8ZPVMmdXUPurLI+LvIuJ3ga9ExDkRcVhEvBO4d4iznOe8fu0xi6SkByV9RtIJNeRNzQW/XdLfS3otcASwDljTgryU9UyXVfVftD7+Gr2D4of1AMXVQW6nmMt5yTBnOc95Od+AG9uc19abT3gxa1j4KuSVSlnPVFmNNeo2FtN5zhsgK/lVyF3PIcxqolG3tZjOc94AeUmvQu56VlfPpD+7Jva3AHcBC7os35eKr5ydMst5zhsgL/VVyF3PIcxqaj7q31BcofueacufTcVXCk6c5Tzn9esc0l6F3PUcwqymGvU5tLCYznNev5T+KuTn4HoOXVaTHybuQ6IXZ8os5zmvgvzVki6t8fldz2HLqnof0VPY37O6jVnOc94AeZVfRDez8bW2nnVl5TR73ntamuU85/Wrjovozsb1zDwrp0Y99MV0nvMqclriPNcz86xszkyMiKWSul2teKiznOe8OZ47gDdTzIlxOXACxckhdwCfkVTHkRHTt8H1zD2riUbd1mI6z3kD5F0CPIviuOLtwNOAK4BTgW2Szq44z/UcxqyGGnUri+k85w2QNylpeUQsALYCz5b0ZETMp/hg6qiK81zPIcxK9snrtE9GJ8t/FwAPUk4kTnFcd9XzUSfLcp7zBsi7tePra6atu60F42ttPVNmNfVh4k4AFefI3yLpyfL+Tqo/WylllvOc16+tUV6mStLJUwsj4hDgyRryXM8hzGqqUbeymM5zXr8knSJpjwnoKebBXlV1Hq7nUGZlc9QHQEQsAhZJur9NWc5z3hzPvQQ4md3P3LtW0iNVZ82yDa5nxllNnkLeumI6z3kDZJ0JfAi4rsyBYhrQE4GPSPpCDZmu55BlNbLroxzgBoorO+9X3saA9eW6ocxynvMG8F+AFZL+WNJflLf3AKPAn1Yd5noOaVaVn0z2egPuBA7osvxA4K5hzXKe8wbIu4su12IEllDPfM2u5xBmNTXNaVBenXia31D96aUps5znvH5dAGyIiOvYfRrQE4GP1pDneg5hVlONupXFdJ7z+iXp8xFxBXASu/bhrgXOl/Rw1Xm4nkOZ1eSHiQey+wCnPmCo/MWZMst5zqsgf5WkK2t8ftdzyLKyOTyvDcV0nvMqytsg6WUJ81zPzLNymub0z1ua5Tzn9Sv1NKCuZ+ZZOTXqoS+m85xXkf+QOM/1zDwrp0Y99MV0nvMGERHHRsQx5ddHAq+IiNfWmTmN65l5VlNHfRARxwKSdEvHAA+WdPUwZznPeX1mfQg4BZgfEdcDxwE3AGsi4vclXVBDpus5bFlNfJjYOUCgc4AnUnwaXEsx685ynvMGyJsEjqaYp3krsFTS9ohYCKxT9fNRu55DmFXpWUG93oBJYB7F6aTbgaeXyxdSw3zUqbKc57wB8m7t9nV5/7YWjK+19UyZ1dh81JJ+LekXwA8lbQeQ9Dg1zEedMMt5zuvXkxGxX/n1iqmF5URGbRhfm+uZLKupRt3KYjrPeQN4ZdnE0O7XD1wAvL2GPNdzCLOa2kf9NElPdFl+MMV1xyaHMct5zsud6zmcsjkz0WxvFBFHAZdSnF79NeA8ladWR8TNko5tcvuGTcp6psxqaj7qoyLipoj4SURcGsVcAFPrbh7WLOc5bwCXAB8GllNMm/mtiHh+uW5B1WGu55BmVfnJZB+fln6L4ooPBwB/AmwEnt/t09NhynKe8wbI++60+2PAD4CXAxtaML7W1jNpVtWFym6A6X8RnOe8vvKYNvk8cFSZ+WAbxtfWeibNqrpQ2Q2wgV8E5zmvj7x/B7y8y/LnAX/TgvG1tp5Js6ouVHYDTP+L4DznZXtzPYfz5qM+zBpUHk98PvAG4FkUl626H/gqcJFqujJ4W6WsZ8qspo76WBIRF0XEHRHxUEQ8GBGbymUHDGuW85w3gC8BDwMrJR0k6RkU+3EfLtdVyvUczqymzkxsZTGd57wBLJP0cUlbpxZI2irp48BhNeS5nsOY1cT+FuDOQdblnuU85w2Qdx3wAWCkY9kIcB7wLy0YX2vrmTKrqXfU90TEByJiZGpBRIxExHnsulLxMGY5z3n9egvwDOCb5a6BhyiuZH0Q8Ic15LmeQ5jV1FwfBwJrgNdT7IQH2AZcAXxc0kPDmOU85+XO9RxOPurDrGERcQTFfBE3SXqsY/nJkq5pbsuGU8p6pspq7JqJEXFERLwqIhZNW37yMGc5z3l9Zr2f4nCu9wEbI+L1Has/VnVemel6DllWpTvy+9gJ/37gTuB/A5uB13esq/oU8mRZznPeAHmTwOLy62XABHB2ef/WFoyvtfVMmlV1obIbYPpfBOc5r5+8jdPuLwauAT5BfZficj2HLKupq5DvI2kHgKTNEbESuDwiDgNiiLOc57x+bYuIoyXdVmbuiIhVwOcops+smus5hFlN7aPeFhFHT90pf5CrgIOpqZiJspznvH6dSXEF69+StFPSmcAra8hzPYcwq6nD85ZSXPRya5d1x0v69jBmOc95uXM9h5MPzzMzy1xjh+eZmVlv3KjNzDLnRm1mljk3ajOzzP1/mULCySuKwSYAAAAASUVORK5CYII=\n"},"metadata":{"needs_background":"light"}}],"source":["temp_series.plot(kind=\"bar\")\n","\n","plt.grid(True)\n","plt.show()"]},{"cell_type":"markdown","source":["### Periods"],"metadata":{"id":"yilRzJ7zqsYa"}},{"cell_type":"markdown","metadata":{"id":"mh0jD10ZbdkU"},"source":["The `pd.period_range()` function returns a `PeriodIndex` instead of a `DatetimeIndex`. For example, let's get all quarters in 2022 and 2023:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MuvcVUi7bdkU","outputId":"b45030fe-3a47-41a3-ec73-48e98bf1712c","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677380508261,"user_tz":-480,"elapsed":287,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["PeriodIndex(['2022Q1', '2022Q2', '2022Q3', '2022Q4', '2023Q1', '2023Q2',\n","             '2023Q3', '2023Q4'],\n","            dtype='period[Q-DEC]')"]},"metadata":{},"execution_count":36}],"source":["quarters = pd.period_range('2022Q1', periods=8, freq='Q')\n","quarters"]},{"cell_type":"markdown","metadata":{"id":"s0U-6hwvbdkU"},"source":["Adding a number `N` to a `PeriodIndex` shifts the periods by `N` times the `PeriodIndex`'s frequency:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"msf61zGDbdkV","outputId":"e4eee169-865c-425f-839b-5d3c40ffa88b","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677380518451,"user_tz":-480,"elapsed":335,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["PeriodIndex(['2022Q4', '2023Q1', '2023Q2', '2023Q3', '2023Q4', '2024Q1',\n","             '2024Q2', '2024Q3'],\n","            dtype='period[Q-DEC]')"]},"metadata":{},"execution_count":37}],"source":["quarters + 3"]},{"cell_type":"markdown","metadata":{"id":"svb-1SlLbdkW"},"source":["Pandas also provides many other time-related functions that we recommend you check out in the [documentation](http://pandas.pydata.org/pandas-docs/stable/timeseries.html)"]},{"cell_type":"markdown","source":["### `DataFrame` objects"],"metadata":{"id":"tO8nGEYPq8TM"}},{"cell_type":"markdown","metadata":{"id":"_3B8znu4bdkX"},"source":["A DataFrame object represents a spreadsheet, with cell values, column names and row index labels. You can define expressions to compute columns based on other columns, create pivot-tables, group rows, draw graphs, etc. You can see `DataFrame` as dictionaries of `Series`."]},{"cell_type":"markdown","source":["#### Creating a `DataFrame`"],"metadata":{"id":"FUNENUdHrE4G"}},{"cell_type":"code","source":["#%unload_ext google.colab.data_table #cloab only"],"metadata":{"id":"aGJi63XEEgbS"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["#from google.colab import data_table\n","#data_table.disable_dataframe_formatter()"],"metadata":{"id":"rv7Yv0DOsLF2"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["You can create a DataFrame by passing a dictionary of `Series` objects:"],"metadata":{"id":"ge2vV_g9nKT3"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"YYuXxk5IbdkX","outputId":"d5f07370-0b19-4c86-d705-6d47f452d9a8","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677380862380,"user_tz":-480,"elapsed":433,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight  birthyear  children    hobby\n","alice        68       1985       NaN   Biking\n","bob          83       1984       3.0  Dancing\n","charles     112       1992       0.0      NaN"],"text/html":["\n","  <div id=\"df-0ede3346-56bc-4e16-87e7-1a89795feb12\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","      <th>hobby</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>1985</td>\n","      <td>NaN</td>\n","      <td>Biking</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>1984</td>\n","      <td>3.0</td>\n","      <td>Dancing</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>1992</td>\n","      <td>0.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0ede3346-56bc-4e16-87e7-1a89795feb12')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-0ede3346-56bc-4e16-87e7-1a89795feb12 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-0ede3346-56bc-4e16-87e7-1a89795feb12');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":43}],"source":["people_dict = {\n","    \"weight\": pd.Series([68, 83, 112], index=[\"alice\", \"bob\", \"charles\"]),\n","    \"birthyear\": pd.Series([1984, 1985, 1992], index=[\"bob\", \"alice\", \"charles\"], name=\"year\"),\n","    \"children\": pd.Series([0, 3], index=[\"charles\", \"bob\"]),\n","    \"hobby\": pd.Series([\"Biking\", \"Dancing\"], index=[\"alice\", \"bob\"]),\n","}\n","people = pd.DataFrame(people_dict)\n","people"]},{"cell_type":"markdown","metadata":{"id":"q5PqT_JjbdkX"},"source":["A few things to note:\n","* the `Series` were automatically aligned based on their index,\n","* missing values are represented as `NaN`,\n","* `Series` names are ignored (the name `\"year\"` was dropped),\n","* `DataFrame`s are displayed nicely in Jupyter notebooks!"]},{"cell_type":"markdown","source":["#### Subsets - Accessing columns"],"metadata":{"id":"D96FglZluzN7"}},{"cell_type":"markdown","metadata":{"id":"Z7-ImAbLbdkX"},"source":["You can access columns by using the column name or fancy indexing. They are returned as `Series` objects:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Wzc_2C8fbdkX","outputId":"cd95a181-decb-4ce5-c103-44b08034f516","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677380865928,"user_tz":-480,"elapsed":276,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["alice      1985\n","bob        1984\n","charles    1992\n","Name: birthyear, dtype: int64"]},"metadata":{},"execution_count":44}],"source":["people[\"birthyear\"]"]},{"cell_type":"markdown","metadata":{"id":"EorAvWrRbdkX"},"source":["You can also get multiple columns at once:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1MlqI9oLbdkX","outputId":"ca1253ba-782b-4f26-ecf3-67cb1e805efa","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677380867112,"user_tz":-480,"elapsed":3,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         birthyear    hobby\n","alice         1985   Biking\n","bob           1984  Dancing\n","charles       1992      NaN"],"text/html":["\n","  <div id=\"df-4a35d01e-6795-4769-b22e-d49dd8767e67\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>birthyear</th>\n","      <th>hobby</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>1985</td>\n","      <td>Biking</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>1984</td>\n","      <td>Dancing</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>1992</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4a35d01e-6795-4769-b22e-d49dd8767e67')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-4a35d01e-6795-4769-b22e-d49dd8767e67 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-4a35d01e-6795-4769-b22e-d49dd8767e67');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":45}],"source":["people[[\"birthyear\", \"hobby\"]]"]},{"cell_type":"markdown","metadata":{"id":"BR_u72S8bdkY"},"source":["Another convenient way to create a `DataFrame` is to pass all the values to the constructor as an `ndarray`, or a list of lists, and specify the column names and row index labels separately:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7W8qDSb0bdkY","outputId":"327a1395-6ca5-40e7-e354-7a5e6cebedf1","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677380869657,"user_tz":-480,"elapsed":2,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         birthyear  children    hobby  weight\n","alice         1985       NaN   Biking      68\n","bob           1984       3.0  Dancing      83\n","charles       1992       0.0      NaN     112"],"text/html":["\n","  <div id=\"df-9bf3aad4-8870-47b5-9fc7-85f328bb19ec\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","      <th>hobby</th>\n","      <th>weight</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>1985</td>\n","      <td>NaN</td>\n","      <td>Biking</td>\n","      <td>68</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>1984</td>\n","      <td>3.0</td>\n","      <td>Dancing</td>\n","      <td>83</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>1992</td>\n","      <td>0.0</td>\n","      <td>NaN</td>\n","      <td>112</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9bf3aad4-8870-47b5-9fc7-85f328bb19ec')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-9bf3aad4-8870-47b5-9fc7-85f328bb19ec button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-9bf3aad4-8870-47b5-9fc7-85f328bb19ec');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":46}],"source":["values = [\n","            [1985, np.nan, \"Biking\",   68],\n","            [1984, 3,      \"Dancing\",  83],\n","            [1992, 0,      np.nan,    112]\n","         ]\n","d3 = pd.DataFrame(\n","        values,\n","        columns=[\"birthyear\", \"children\", \"hobby\", \"weight\"],\n","        index=[\"alice\", \"bob\", \"charles\"]\n","     )\n","d3"]},{"cell_type":"markdown","source":["#### Multi-index (optional)"],"metadata":{"id":"RYU0DulBtPwf"}},{"cell_type":"markdown","source":["You can also create multi-index datafram as follows:"],"metadata":{"id":"1hA9uEAiqUlg"}},{"cell_type":"code","source":["df = pd.DataFrame(\n","    {\n","        \"a\" : [4 ,5, 6],\n","        \"b\" : [7, 8, 9],\n","        \"c\" : [10, 11, 12]\n","     },\n","    index = pd.MultiIndex.from_tuples(\n","        [('d',1),('d',2),('e',2)], names=['n','v']\n","    )\n",")\n","df"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":175},"id":"xsBbZelxn8tC","outputId":"d35413b2-9598-4a53-8811-1a782a338ad0","executionInfo":{"status":"ok","timestamp":1677381098004,"user_tz":-480,"elapsed":278,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["     a  b   c\n","n v          \n","d 1  4  7  10\n","  2  5  8  11\n","e 2  6  9  12"],"text/html":["\n","  <div id=\"df-1af6cfdb-4adf-429e-a48a-f4a9bc1ece05\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th></th>\n","      <th>a</th>\n","      <th>b</th>\n","      <th>c</th>\n","    </tr>\n","    <tr>\n","      <th>n</th>\n","      <th>v</th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th rowspan=\"2\" valign=\"top\">d</th>\n","      <th>1</th>\n","      <td>4</td>\n","      <td>7</td>\n","      <td>10</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>5</td>\n","      <td>8</td>\n","      <td>11</td>\n","    </tr>\n","    <tr>\n","      <th>e</th>\n","      <th>2</th>\n","      <td>6</td>\n","      <td>9</td>\n","      <td>12</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1af6cfdb-4adf-429e-a48a-f4a9bc1ece05')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-1af6cfdb-4adf-429e-a48a-f4a9bc1ece05 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-1af6cfdb-4adf-429e-a48a-f4a9bc1ece05');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":48}]},{"cell_type":"markdown","source":["If all columns are tuples of the same size, then they are understood as a multi-index. The same goes for row index labels. For example:"],"metadata":{"id":"ZJG34ozirdgh"}},{"cell_type":"code","source":["d5 = pd.DataFrame(\n","  {\n","    (\"public\", \"birthyear\"):\n","        {(\"Paris\",\"alice\"):1985, (\"Paris\",\"bob\"): 1984, (\"London\",\"charles\"): 1992},\n","    (\"public\", \"hobby\"):\n","        {(\"Paris\",\"alice\"):\"Biking\", (\"Paris\",\"bob\"): \"Dancing\"},\n","    (\"private\", \"weight\"):\n","        {(\"Paris\",\"alice\"):68, (\"Paris\",\"bob\"): 83, (\"London\",\"charles\"): 112},\n","    (\"private\", \"children\"):\n","        {(\"Paris\", \"alice\"):np.nan, (\"Paris\",\"bob\"): 3, (\"London\",\"charles\"): 0}\n","  }\n",")\n","d5"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":175},"id":"sge2O2qvrDop","outputId":"a2dc4c3c-6df0-43df-ca4b-eb019ce66207","executionInfo":{"status":"ok","timestamp":1677381157579,"user_tz":-480,"elapsed":282,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                  public          private         \n","               birthyear    hobby  weight children\n","Paris  alice        1985   Biking      68      NaN\n","       bob          1984  Dancing      83      3.0\n","London charles      1992      NaN     112      0.0"],"text/html":["\n","  <div id=\"df-a3331ede-d7af-433b-8997-b448c3e485ca\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead tr th {\n","        text-align: left;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr>\n","      <th></th>\n","      <th></th>\n","      <th colspan=\"2\" halign=\"left\">public</th>\n","      <th colspan=\"2\" halign=\"left\">private</th>\n","    </tr>\n","    <tr>\n","      <th></th>\n","      <th></th>\n","      <th>birthyear</th>\n","      <th>hobby</th>\n","      <th>weight</th>\n","      <th>children</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th rowspan=\"2\" valign=\"top\">Paris</th>\n","      <th>alice</th>\n","      <td>1985</td>\n","      <td>Biking</td>\n","      <td>68</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>1984</td>\n","      <td>Dancing</td>\n","      <td>83</td>\n","      <td>3.0</td>\n","    </tr>\n","    <tr>\n","      <th>London</th>\n","      <th>charles</th>\n","      <td>1992</td>\n","      <td>NaN</td>\n","      <td>112</td>\n","      <td>0.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a3331ede-d7af-433b-8997-b448c3e485ca')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-a3331ede-d7af-433b-8997-b448c3e485ca button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-a3331ede-d7af-433b-8997-b448c3e485ca');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":49}]},{"cell_type":"markdown","source":["You can now get a DataFrame containing all the \"public\" columns very simply:"],"metadata":{"id":"EeN5brxLrmHy"}},{"cell_type":"code","source":["d5[\"public\"]"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":143},"id":"m4yGqoUOrlnZ","outputId":"aba5f802-02b7-4518-a3d7-ddba6208c7f6","executionInfo":{"status":"ok","timestamp":1677381161247,"user_tz":-480,"elapsed":294,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                birthyear    hobby\n","Paris  alice         1985   Biking\n","       bob           1984  Dancing\n","London charles       1992      NaN"],"text/html":["\n","  <div id=\"df-74c60beb-0732-4325-be08-ec011c593260\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th></th>\n","      <th>birthyear</th>\n","      <th>hobby</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th rowspan=\"2\" valign=\"top\">Paris</th>\n","      <th>alice</th>\n","      <td>1985</td>\n","      <td>Biking</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>1984</td>\n","      <td>Dancing</td>\n","    </tr>\n","    <tr>\n","      <th>London</th>\n","      <th>charles</th>\n","      <td>1992</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-74c60beb-0732-4325-be08-ec011c593260')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-74c60beb-0732-4325-be08-ec011c593260 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-74c60beb-0732-4325-be08-ec011c593260');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":50}]},{"cell_type":"markdown","metadata":{"id":"IT41nGg7bdkc"},"source":["It is noted that most methods return modified copies in pandas."]},{"cell_type":"markdown","source":["#### Subsets - Accessing rows"],"metadata":{"id":"ZWxiDR0QtNXX"}},{"cell_type":"markdown","metadata":{"id":"HHUt1_d1bdkc"},"source":["\n","Let's go back to the `people` `DataFrame`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"75UxPZPAbdkc","outputId":"52dc74fd-6cc3-4463-c7bc-d1cdf992da13","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677381192460,"user_tz":-480,"elapsed":291,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight  birthyear  children    hobby\n","alice        68       1985       NaN   Biking\n","bob          83       1984       3.0  Dancing\n","charles     112       1992       0.0      NaN"],"text/html":["\n","  <div id=\"df-6f00d101-fe5d-4d6e-bd4b-73a50953bdcf\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","      <th>hobby</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>1985</td>\n","      <td>NaN</td>\n","      <td>Biking</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>1984</td>\n","      <td>3.0</td>\n","      <td>Dancing</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>1992</td>\n","      <td>0.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6f00d101-fe5d-4d6e-bd4b-73a50953bdcf')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-6f00d101-fe5d-4d6e-bd4b-73a50953bdcf button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-6f00d101-fe5d-4d6e-bd4b-73a50953bdcf');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":51}],"source":["people"]},{"cell_type":"markdown","metadata":{"id":"86z3de-Ibdkc"},"source":["**The `loc` attribute lets you access rows instead of columns.** The result is a `Series` object in which the `DataFrame`'s column names are mapped to row index labels:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"2ii7IFnIbdkc","outputId":"1e40a94e-28cc-4c8c-8b07-96a4ceea4f48","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677381221528,"user_tz":-480,"elapsed":288,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["weight        112\n","birthyear    1992\n","children      0.0\n","hobby         NaN\n","Name: charles, dtype: object"]},"metadata":{},"execution_count":52}],"source":["people.loc[\"charles\"]"]},{"cell_type":"markdown","metadata":{"id":"M9n8cJsDbdkd"},"source":["You can also access rows by integer location using the `iloc` attribute:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"u2T-r9f2bdkd","outputId":"f77cc170-822b-487c-f34b-6c90626dd225","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677381257598,"user_tz":-480,"elapsed":295,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["weight        112\n","birthyear    1992\n","children      0.0\n","hobby         NaN\n","Name: charles, dtype: object"]},"metadata":{},"execution_count":53}],"source":["people.iloc[2]"]},{"cell_type":"markdown","metadata":{"id":"h9wxoGC2bdkd"},"source":["You can also get a slice of rows, and this returns a `DataFrame` object:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"PUAGAfWmbdkd","outputId":"e3774828-3f39-4dcc-bcd9-50f514a43ca6","colab":{"base_uri":"https://localhost:8080/","height":112},"executionInfo":{"status":"ok","timestamp":1677381262085,"user_tz":-480,"elapsed":391,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight  birthyear  children    hobby\n","bob          83       1984       3.0  Dancing\n","charles     112       1992       0.0      NaN"],"text/html":["\n","  <div id=\"df-657a53d5-6e38-444e-8ad2-e73453feae18\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","      <th>hobby</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>1984</td>\n","      <td>3.0</td>\n","      <td>Dancing</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>1992</td>\n","      <td>0.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-657a53d5-6e38-444e-8ad2-e73453feae18')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-657a53d5-6e38-444e-8ad2-e73453feae18 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-657a53d5-6e38-444e-8ad2-e73453feae18');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":54}],"source":["people.iloc[1:3]"]},{"cell_type":"markdown","metadata":{"id":"agVv1ZEKbdkd"},"source":["Finally, you can pass a boolean array to get the matching rows. This is most useful when combined with boolean expressions:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Uh00Rgg1bdkd","outputId":"88b02931-3345-4809-b4f1-7660d73e0e54","colab":{"base_uri":"https://localhost:8080/","height":112},"executionInfo":{"status":"ok","timestamp":1677381277973,"user_tz":-480,"elapsed":291,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       weight  birthyear  children    hobby\n","alice      68       1985       NaN   Biking\n","bob        83       1984       3.0  Dancing"],"text/html":["\n","  <div id=\"df-b502c8b0-a828-4f7e-ae93-e1fd4614a936\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","      <th>hobby</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>1985</td>\n","      <td>NaN</td>\n","      <td>Biking</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>1984</td>\n","      <td>3.0</td>\n","      <td>Dancing</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b502c8b0-a828-4f7e-ae93-e1fd4614a936')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-b502c8b0-a828-4f7e-ae93-e1fd4614a936 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-b502c8b0-a828-4f7e-ae93-e1fd4614a936');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":55}],"source":["people[people[\"birthyear\"] < 1990]"]},{"cell_type":"markdown","source":["You can also accessing columns by specifiying the second axis:"],"metadata":{"id":"KVvkXiAksSd6"}},{"cell_type":"code","source":["people.iloc[:,2]"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"i4uSRaCfsaGp","outputId":"3f279f3a-36c2-4697-c1c9-54a942d8af22","executionInfo":{"status":"ok","timestamp":1677381286779,"user_tz":-480,"elapsed":284,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["alice      NaN\n","bob        3.0\n","charles    0.0\n","Name: children, dtype: float64"]},"metadata":{},"execution_count":56}]},{"cell_type":"markdown","source":["#### Adding and removing columns"],"metadata":{"id":"CDlCJay-t4lO"}},{"cell_type":"markdown","metadata":{"id":"KuXwobq6bdke"},"source":["You can generally treat `DataFrame` objects like dictionaries of `Series`, so the following work fine:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"9n9f6-N_bdke","outputId":"23297b51-5253-43f1-de45-ad4f538d8002","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677381329559,"user_tz":-480,"elapsed":470,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight  birthyear  children    hobby\n","alice        68       1985       NaN   Biking\n","bob          83       1984       3.0  Dancing\n","charles     112       1992       0.0      NaN"],"text/html":["\n","  <div id=\"df-d0c11ce8-ab58-4c7b-9749-6dfba08b94d5\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","      <th>hobby</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>1985</td>\n","      <td>NaN</td>\n","      <td>Biking</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>1984</td>\n","      <td>3.0</td>\n","      <td>Dancing</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>1992</td>\n","      <td>0.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d0c11ce8-ab58-4c7b-9749-6dfba08b94d5')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-d0c11ce8-ab58-4c7b-9749-6dfba08b94d5 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-d0c11ce8-ab58-4c7b-9749-6dfba08b94d5');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":57}],"source":["people"]},{"cell_type":"code","source":["people[\"age\"] = 2023 - people[\"birthyear\"]  # adds a new column \"age\"\n","people[\"over 30\"] = people[\"age\"] > 30      # adds another column \"over 30\"\n","birthyears = people.pop(\"birthyear\")\n","people.drop(columns=['children'], inplace=True) # drop a column inplace\n","people"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":143},"id":"Y-qN7BgDtlh9","outputId":"31064531-093c-4d69-963c-52ef688e2b86","executionInfo":{"status":"ok","timestamp":1677381366681,"user_tz":-480,"elapsed":314,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight    hobby  age  over 30\n","alice        68   Biking   38     True\n","bob          83  Dancing   39     True\n","charles     112      NaN   31     True"],"text/html":["\n","  <div id=\"df-375686f2-f35c-4e33-80a1-8d69e4782276\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>hobby</th>\n","      <th>age</th>\n","      <th>over 30</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>Biking</td>\n","      <td>38</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>Dancing</td>\n","      <td>39</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>NaN</td>\n","      <td>31</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-375686f2-f35c-4e33-80a1-8d69e4782276')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-375686f2-f35c-4e33-80a1-8d69e4782276 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-375686f2-f35c-4e33-80a1-8d69e4782276');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":58}]},{"cell_type":"code","execution_count":null,"metadata":{"id":"CmdxWxvqbdke","outputId":"91be6303-25cd-40b4-91ee-87b7ba45bcf1","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677381370590,"user_tz":-480,"elapsed":286,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["alice      1985\n","bob        1984\n","charles    1992\n","Name: birthyear, dtype: int64"]},"metadata":{},"execution_count":59}],"source":["birthyears"]},{"cell_type":"markdown","metadata":{"id":"1_VwIo7cbdke"},"source":["When you add a new column, it must have the same number of rows. Missing rows are filled with NaN, and extra rows are ignored:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"4xPE-3XQbdke","outputId":"2de94b28-711b-447f-d280-079d400fa4ea","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677381397671,"user_tz":-480,"elapsed":303,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight    hobby  age  over 30  pets\n","alice        68   Biking   38     True   NaN\n","bob          83  Dancing   39     True   0.0\n","charles     112      NaN   31     True   5.0"],"text/html":["\n","  <div id=\"df-c432952b-d204-4b10-af51-8b993275392a\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>hobby</th>\n","      <th>age</th>\n","      <th>over 30</th>\n","      <th>pets</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>Biking</td>\n","      <td>38</td>\n","      <td>True</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>Dancing</td>\n","      <td>39</td>\n","      <td>True</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>NaN</td>\n","      <td>31</td>\n","      <td>True</td>\n","      <td>5.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c432952b-d204-4b10-af51-8b993275392a')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-c432952b-d204-4b10-af51-8b993275392a button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-c432952b-d204-4b10-af51-8b993275392a');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":60}],"source":["people[\"pets\"] = pd.Series({\"bob\": 0, \"charles\": 5, \"eugene\":1})  # alice is missing, eugene is ignored\n","people"]},{"cell_type":"markdown","metadata":{"id":"qmOZF02Ebdkf"},"source":["When adding a new column, it is added at the end (on the right) by default. You can also insert a column anywhere else using the `insert()` method:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"GEIn8gb1bdkf","outputId":"6b311f7f-30da-4f56-d324-3c1397d0963a","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677381421921,"user_tz":-480,"elapsed":301,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight  height    hobby  age  over 30  pets\n","alice        68     172   Biking   38     True   NaN\n","bob          83     181  Dancing   39     True   0.0\n","charles     112     185      NaN   31     True   5.0"],"text/html":["\n","  <div id=\"df-d6ad43f6-9828-4a95-9665-0dff47035d7b\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>height</th>\n","      <th>hobby</th>\n","      <th>age</th>\n","      <th>over 30</th>\n","      <th>pets</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>172</td>\n","      <td>Biking</td>\n","      <td>38</td>\n","      <td>True</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>181</td>\n","      <td>Dancing</td>\n","      <td>39</td>\n","      <td>True</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>185</td>\n","      <td>NaN</td>\n","      <td>31</td>\n","      <td>True</td>\n","      <td>5.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d6ad43f6-9828-4a95-9665-0dff47035d7b')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-d6ad43f6-9828-4a95-9665-0dff47035d7b button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-d6ad43f6-9828-4a95-9665-0dff47035d7b');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":61}],"source":["people.insert(1, \"height\", [172, 181, 185])\n","people"]},{"cell_type":"markdown","metadata":{"id":"YAQ4_fffbdkf"},"source":["You can also create new columns by calling the `assign()` method. Note that this returns a new `DataFrame` object, the original is not modified"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"qEDE3MLTbdkf","outputId":"90d51ffb-c8f3-4e35-a578-61df54faa2e3","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677381440325,"user_tz":-480,"elapsed":281,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight  height    hobby  age  over 30  pets        bmi  has_pets\n","alice        68     172   Biking   38     True   NaN  22.985398     False\n","bob          83     181  Dancing   39     True   0.0  25.335002     False\n","charles     112     185      NaN   31     True   5.0  32.724617      True"],"text/html":["\n","  <div id=\"df-75ed80f8-b0b0-4351-910e-96b66a3ad0f0\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>height</th>\n","      <th>hobby</th>\n","      <th>age</th>\n","      <th>over 30</th>\n","      <th>pets</th>\n","      <th>bmi</th>\n","      <th>has_pets</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>172</td>\n","      <td>Biking</td>\n","      <td>38</td>\n","      <td>True</td>\n","      <td>NaN</td>\n","      <td>22.985398</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>181</td>\n","      <td>Dancing</td>\n","      <td>39</td>\n","      <td>True</td>\n","      <td>0.0</td>\n","      <td>25.335002</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>185</td>\n","      <td>NaN</td>\n","      <td>31</td>\n","      <td>True</td>\n","      <td>5.0</td>\n","      <td>32.724617</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-75ed80f8-b0b0-4351-910e-96b66a3ad0f0')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-75ed80f8-b0b0-4351-910e-96b66a3ad0f0 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-75ed80f8-b0b0-4351-910e-96b66a3ad0f0');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":62}],"source":["p2 = people.assign(\n","    bmi = people[\"weight\"] / (people[\"height\"] / 100) ** 2,\n","    has_pets = people[\"pets\"] > 0\n",")\n","p2"]},{"cell_type":"markdown","source":["You can also rename the column name:"],"metadata":{"id":"li0DlR9hvHde"}},{"cell_type":"code","source":["p2.rename(columns={'bmi':'body_mass_index'})"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":143},"id":"WuV0yEpEuJD9","outputId":"2b6ab753-2db1-4f74-f11f-df0d16c67287","executionInfo":{"status":"ok","timestamp":1677381600160,"user_tz":-480,"elapsed":282,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight  height    hobby  age  over 30  pets  body_mass_index  \\\n","alice        68     172   Biking   38     True   NaN        22.985398   \n","bob          83     181  Dancing   39     True   0.0        25.335002   \n","charles     112     185      NaN   31     True   5.0        32.724617   \n","\n","         has_pets  \n","alice       False  \n","bob         False  \n","charles      True  "],"text/html":["\n","  <div id=\"df-131c72c3-7b1a-4b63-8b5b-708cf3d34181\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>height</th>\n","      <th>hobby</th>\n","      <th>age</th>\n","      <th>over 30</th>\n","      <th>pets</th>\n","      <th>body_mass_index</th>\n","      <th>has_pets</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>172</td>\n","      <td>Biking</td>\n","      <td>38</td>\n","      <td>True</td>\n","      <td>NaN</td>\n","      <td>22.985398</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>181</td>\n","      <td>Dancing</td>\n","      <td>39</td>\n","      <td>True</td>\n","      <td>0.0</td>\n","      <td>25.335002</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>185</td>\n","      <td>NaN</td>\n","      <td>31</td>\n","      <td>True</td>\n","      <td>5.0</td>\n","      <td>32.724617</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-131c72c3-7b1a-4b63-8b5b-708cf3d34181')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-131c72c3-7b1a-4b63-8b5b-708cf3d34181 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-131c72c3-7b1a-4b63-8b5b-708cf3d34181');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":63}]},{"cell_type":"markdown","source":["#### Querying a `DataFrame`"],"metadata":{"id":"a0XXvdMBvWFn"}},{"cell_type":"markdown","metadata":{"id":"8_UK9SF5bdkh"},"source":["The `query()` method lets you **filter a `DataFrame` based on a query expression**:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"hiVf_7cJbdkh","outputId":"4aac7b55-80ef-49b5-fe89-9528172c4512","colab":{"base_uri":"https://localhost:8080/","height":81},"executionInfo":{"status":"ok","timestamp":1677381694665,"user_tz":-480,"elapsed":296,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["     weight  height    hobby  age  over 30  pets\n","bob      83     181  Dancing   39     True   0.0"],"text/html":["\n","  <div id=\"df-596bd9bb-76e1-4313-bb7e-aecfd0d87198\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>height</th>\n","      <th>hobby</th>\n","      <th>age</th>\n","      <th>over 30</th>\n","      <th>pets</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>181</td>\n","      <td>Dancing</td>\n","      <td>39</td>\n","      <td>True</td>\n","      <td>0.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-596bd9bb-76e1-4313-bb7e-aecfd0d87198')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-596bd9bb-76e1-4313-bb7e-aecfd0d87198 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-596bd9bb-76e1-4313-bb7e-aecfd0d87198');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":65}],"source":["people.query(\"age > 30 and pets == 0\")"]},{"cell_type":"markdown","source":["#### Sorting a `DataFrame`"],"metadata":{"id":"iXDwWBGtveGm"}},{"cell_type":"markdown","metadata":{"id":"zQYCm9izbdkh"},"source":["You can sort a `DataFrame` by calling its `sort_index` method. By default it sorts the rows by their index label, in ascending order, but let's reverse the order:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"vTUlVF6ibdki","outputId":"29c99c7c-2f5b-419a-c2cc-22fd7a017733","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677381730615,"user_tz":-480,"elapsed":268,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         weight  height    hobby  age  over 30  pets\n","charles     112     185      NaN   31     True   5.0\n","bob          83     181  Dancing   39     True   0.0\n","alice        68     172   Biking   38     True   NaN"],"text/html":["\n","  <div id=\"df-74443914-b3cc-4257-aafc-2894a5a272bb\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>weight</th>\n","      <th>height</th>\n","      <th>hobby</th>\n","      <th>age</th>\n","      <th>over 30</th>\n","      <th>pets</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>charles</th>\n","      <td>112</td>\n","      <td>185</td>\n","      <td>NaN</td>\n","      <td>31</td>\n","      <td>True</td>\n","      <td>5.0</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>83</td>\n","      <td>181</td>\n","      <td>Dancing</td>\n","      <td>39</td>\n","      <td>True</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>alice</th>\n","      <td>68</td>\n","      <td>172</td>\n","      <td>Biking</td>\n","      <td>38</td>\n","      <td>True</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-74443914-b3cc-4257-aafc-2894a5a272bb')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-74443914-b3cc-4257-aafc-2894a5a272bb button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-74443914-b3cc-4257-aafc-2894a5a272bb');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":66}],"source":["people.sort_index(ascending=False)"]},{"cell_type":"markdown","metadata":{"id":"hyUelvGXbdki"},"source":["Note that `sort_index` returned a sorted *copy* of the `DataFrame`. To modify `people` directly, we can set the `inplace` argument to `True`. Also, we can sort the columns instead of the rows by setting `axis=1`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"GTIgiGLTbdki","outputId":"09e939e3-1822-49e4-c562-bdc97ffcdf2c","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677381784353,"user_tz":-480,"elapsed":276,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         age  height    hobby  over 30  pets  weight\n","alice     38     172   Biking     True   NaN      68\n","bob       39     181  Dancing     True   0.0      83\n","charles   31     185      NaN     True   5.0     112"],"text/html":["\n","  <div id=\"df-9c8c4afc-a85d-4f72-8173-c2e12e94ec47\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>age</th>\n","      <th>height</th>\n","      <th>hobby</th>\n","      <th>over 30</th>\n","      <th>pets</th>\n","      <th>weight</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>38</td>\n","      <td>172</td>\n","      <td>Biking</td>\n","      <td>True</td>\n","      <td>NaN</td>\n","      <td>68</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>39</td>\n","      <td>181</td>\n","      <td>Dancing</td>\n","      <td>True</td>\n","      <td>0.0</td>\n","      <td>83</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>31</td>\n","      <td>185</td>\n","      <td>NaN</td>\n","      <td>True</td>\n","      <td>5.0</td>\n","      <td>112</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9c8c4afc-a85d-4f72-8173-c2e12e94ec47')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-9c8c4afc-a85d-4f72-8173-c2e12e94ec47 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-9c8c4afc-a85d-4f72-8173-c2e12e94ec47');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":67}],"source":["people.sort_index(axis=1, inplace=True)\n","people"]},{"cell_type":"markdown","metadata":{"id":"2-4p9J5Dbdki"},"source":["To sort the `DataFrame` by the values instead of the labels, we can use `sort_values` and specify the column to sort by:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JHukwfIIbdki","outputId":"d8a9f582-1583-4615-afb8-64d649b8e830","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677381814703,"user_tz":-480,"elapsed":271,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         age  height    hobby  over 30  pets  weight\n","charles   31     185      NaN     True   5.0     112\n","alice     38     172   Biking     True   NaN      68\n","bob       39     181  Dancing     True   0.0      83"],"text/html":["\n","  <div id=\"df-224a50f1-0cdd-417f-852a-9038eafce25f\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>age</th>\n","      <th>height</th>\n","      <th>hobby</th>\n","      <th>over 30</th>\n","      <th>pets</th>\n","      <th>weight</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>charles</th>\n","      <td>31</td>\n","      <td>185</td>\n","      <td>NaN</td>\n","      <td>True</td>\n","      <td>5.0</td>\n","      <td>112</td>\n","    </tr>\n","    <tr>\n","      <th>alice</th>\n","      <td>38</td>\n","      <td>172</td>\n","      <td>Biking</td>\n","      <td>True</td>\n","      <td>NaN</td>\n","      <td>68</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>39</td>\n","      <td>181</td>\n","      <td>Dancing</td>\n","      <td>True</td>\n","      <td>0.0</td>\n","      <td>83</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-224a50f1-0cdd-417f-852a-9038eafce25f')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-224a50f1-0cdd-417f-852a-9038eafce25f button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-224a50f1-0cdd-417f-852a-9038eafce25f');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":68}],"source":["people.sort_values(by=\"age\", inplace=True)\n","people"]},{"cell_type":"markdown","source":["#### Plotting a `DataFrame`"],"metadata":{"id":"d5AyYBEvv2cV"}},{"cell_type":"markdown","metadata":{"id":"OBVCkgSGbdki"},"source":["Just like for `Series`, pandas makes it easy to draw nice graphs based on a `DataFrame`.\n","\n","For example, it is trivial to create a line plot from a `DataFrame`'s data by calling its `plot` method:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7gv0BV_ebdki","outputId":"94403ce1-0060-4661-e2fa-0db5b9499788","colab":{"base_uri":"https://localhost:8080/","height":280},"executionInfo":{"status":"ok","timestamp":1677381902591,"user_tz":-480,"elapsed":526,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 432x288 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXcAAAEHCAYAAABV4gY/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoBUlEQVR4nO3deXzV1Z3/8dcHshGysQSBLCQqiywBISKLuFarjhXtuHcsrkyVqaNOp2PtPKrT2haV2oczPqb+tCB1xuqo2OponWkdtSpUMCg7LqgsYd8SlpD9/P443+ReYiB7bvLN+/l45MG95/u9954vyjvnnu9ZzDmHiIiES69YV0BERNqfwl1EJIQU7iIiIaRwFxEJIYW7iEgIxcW6AgADBw50eXl5sa6GiEi3snz58j3OuczGjnWJcM/Ly6OoqCjW1RAR6VbMbNOxjqlbRkQkhBTuIiIhpHAXEQkhhbuISAgp3EVEQkjhLiISQgp3EZEQ6hLj3EVEwq68qob9ZZWUlFWxv6yS0rIq9pdVMXJwKpOG9Wv3z1O4i4i0QEV1TX0wl5RVsr+sitIjlcHz4PHhKkqO+CCvC/OK6tpG32/2mScq3EVE2ktVTW19GJdEhXVJmQ/m/WVVQYjXhXQlJUeqKKusOeZ7xvc2MpIT6JccT0afBHL7J1OQHU9GcgIZQVm/5HjSk+PpF5T1S07okOtTuItIt1ZT6yg90qAVfbiKkiORsN5fVknpkeigruJQRfUx37N3LyOjT7wP5OQEhmYkccqQNB/ayZGw7pecQHqfePr1TSCjTzzJCb0xs068+mNTuItIl1Bb6zhYXu0D+Eh0v/TRLWffoq6sb2kfKD92SPcySO8TCePMlERGDEo9quWckeyDue55enI8qYlxXSakW0vhLiLtyjnHwYrqo4K5ruV89M3EyqB17UO69EgVtcfZ0jktKa6+hZyRnEDewL71Lef6VnRdaAdhnZoUR69e3TukW0vhLiKNcs5RVlnjW8uHG3ZrRPqpIzcT6/qrq6g5TkqnJMYFLWYfwFkZfb7aiu4bT3rQP50RBHjvHhrSrdVkuJvZAuASYJdzbmxQNgF4HEgCqoHbnXPLzH+PeRS4GCgDbnDOfdhBdZfjqKqp5ZMdB1lVXMqq4hIm5GRwzeTcWFdLYuRYw/AiIzoa3EA84h9X1jQ+wgMgOaF3fSs6IzmeUYPTgpazv3FYF9bR/dTpfeKJ763pNZ2hOS33hcBjwNNRZQ8B/+Kce93MLg6enw1cBAwPfk4HfhX8KR2ottbxxZ7DrCouYVVxKSuLS1i37UD90Ku0pDgGpSbGuJbSHtp7GB5AYlyvqJZzPCdlptS3nDOCsI5uRdeN9kiM692JVy4t1WS4O+feMbO8hsVAWvA4HdgWPJ4JPO2cc8D7ZpZhZkOcc9vbq8I9nXOOrSVH6kN81ZZS1mwt5WBw579PfG/GZqXxN1OGUZCdzvjsDIYNSO72N4fCpiOH4dX1N+c0cxheUrxCOoxa2+d+J/C/ZjYPv4TBtKA8C9gSdV5xUPaVcDez2cBsgNxcdRccy55DFawqLmHlFt+9snprKXsOVQL+H/OowWlcOmEo47MzKMhJ5+TMFOL0tbfTaBiedFWtDffbgLucc4vM7CpgPvC1lryBc+4J4AmAwsLC49wj7zkOllexemtpfT/5yi2lbC05AoAZnJyZwlkjBjE+J52C7AxGDU5Vq6udNHcY3lFD8g63fRheep+jW9FhGYYnsdfacJ8F/H3w+AXg18HjrUBO1HnZQZk0UF5Vw7rtB1i1JdJP/sWew7jg11xO/z5MyM1g1rRhFGRnMDYrnZREDW5qSmPD8L7Sim7lMLxIf7Mfhhd9M1HD8KSraW1abAPOAt4GzgU+C8pfAf7OzJ7D30gtVX87VNfU8unOQ741HrTKP9lxkOogTQamJDI+O51Lx2fVt8r79+2YKcndhYbhibRNc4ZCPosfCTPQzIqB+4BbgUfNLA4oJ+g7B/6AHwa5AT8U8sYOqHOXVlvr2Lj3cOSGZ3Epa7eVUl4VGblSkJ3BrWeeyPhsH+RD0pNC/TW8JcPwom8mtmUY3tE3DSMjPzQMT3qK5oyWufYYhyY1cq4D5rS1Ut2Fc47tpeVHtchXFZdyMOiHTYrvxZih6Vw7OZfx2RmMz8lgWP/kbvtV/VjD8I5qRbdhGF56n3hOHJjS4KbhV4fhpfWJ170GkSaoE7cF9h2urB9+WBfoew5VABDXyxg5OJVLCobWt8hHnNA1R65U1dTWj/DQMDyRcFK4H8OhimpWR7XGVxaXULw/MnLlxIF9OXP4QAqy0ynIyWD0kLROD6pYDsOLzED0Ia1heCJdi8Id3ye8fvuBo/rJP999qH7kSlZGH8bnpNdPDBqXlU5qUny7fX5Tw/C+cjOxGcPwzKjvk07v44fhDR+UGmlF99UwPJEw63HhXl1Ty2e7vjpypaqmbuRKAgXZGVxSMMRPDMpOZ0BK86buRw/Dq2s1axieiMRCqMPdOcfGvWVHzfBcu+0AR6p833FqYhzjstO5+Yxg5EpOBkPTkwDqh+FtLy3n4x0HYzYMLy0prkv224tI1xaqcN9RWs6KLSX1/eSriku+0nUxOC2Jcdnp5PZPpl9yPKVHqvh89yE+3LS/1cPwRg5OPepmoobhiUisdetwr6iu4anFGynauJ9VxSXsOljR5Gt2HChnx4Fyln25j4S4XvSLWqNDw/BEJCy6dbh/vuswj/zxUxyOjOQEhg9KiWo5Nz4Mr+5mYkafBPokKKRFJJy6dbiPHprG2h9/nbhephEeIiJRunW4A+rHFhFphJJRRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQkhhbuISAgp3EVEQkjhLiISQk2Gu5ktMLNdZramQfl3zexjM1trZg9Flf/AzDaY2Sdm9vWOqLSIiBxfc9ZzXwg8BjxdV2Bm5wAzgfHOuQozGxSUjwauAcYAQ4E3zGyEc66mvSsuIiLH1mTL3Tn3DrCvQfFtwFznXEVwzq6gfCbwnHOuwjn3JbABmNyO9RURkWZobZ/7CGCGmS01sz+b2WlBeRawJeq84qBMREQ6UWu32YsD+gNTgNOA583sxJa8gZnNBmYD5ObmtrIaIiLSmNa23IuBl5y3DKgFBgJbgZyo87KDsq9wzj3hnCt0zhVmZma2shoiItKY1ob774FzAMxsBJAA7AFeAa4xs0QzyweGA8vaoZ4iItICTXbLmNmzwNnAQDMrBu4DFgALguGRlcAs55wD1prZ88A6oBqYo5EyIiKdz3wmx1ZhYaErKiqKdTVERLoVM1vunCts7JhmqIqIhJDCXUQkhBTuIiIhpHAXEQkhhbuISAgp3EVEQkjhLiISQgp3EZEQUriLiISQwl1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhFCT4W5mC8xsl5mtaeTYP5iZM7OBwXMzs381sw1mtsrMJnZEpUVE5Pia03JfCFzYsNDMcoALgM1RxRcBw4Of2cCv2l5FERFpqSbD3Tn3DrCvkUO/BL4PuKiymcDTznsfyDCzIe1SUxERabZW9bmb2Uxgq3NuZYNDWcCWqOfFQVlj7zHbzIrMrGj37t2tqYaIiBxDi8PdzJKBe4EfteWDnXNPOOcKnXOFmZmZbXkrERFpIK4VrzkJyAdWmhlANvChmU0GtgI5UedmB2UiItKJWtxyd86tds4Ncs7lOefy8F0vE51zO4BXgG8Ho2amAKXOue3tW2UREWlKc4ZCPgv8BRhpZsVmdvNxTv8D8AWwAXgSuL1daikiIi3SZLeMc+7aJo7nRT12wJy2V0tERNpCM1RFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQkhhbuISAgp3EVEQkjhLiISQgp3EZEQUriLiISQwl1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREKoyXA3swVmtsvM1kSVPWxmH5vZKjP7nZllRB37gZltMLNPzOzrHVRvERE5jua03BcCFzYo+xMw1jlXAHwK/ADAzEYD1wBjgtf8u5n1brfaiohIszQZ7s65d4B9Dcr+6JyrDp6+D2QHj2cCzznnKpxzXwIbgMntWF8REWmG9uhzvwl4PXicBWyJOlYclH2Fmc02syIzK9q9e3c7VENEROq0KdzN7IdANfBMS1/rnHvCOVfonCvMzMxsSzVERKSBuNa+0MxuAC4BznPOuaB4K5ATdVp2UCYiIp2oVS13M7sQ+D5wqXOuLOrQK8A1ZpZoZvnAcGBZ26spIiIt0WTL3cyeBc4GBppZMXAffnRMIvAnMwN43zn3HefcWjN7HliH766Z45yr6ajKi4hI4yzSoxI7hYWFrqioKNbVEBHpVsxsuXOusLFjmqEqIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQkhhbuISKxUHISyfU2f1wqtXn5ARERa6Mh+2Pw+bHwPNi2B7Sthxt1w7j+3+0cp3EVEOsqh3bB5CWxc7MN85xrAQe8EyD7NB/vIizrkoxXuIiLt5cA2H+J1LfM9n/jyuD6QMxnOuReGTYOsQohP6tCqKNxFRFrDOSjZFIT5Yti0GPZ/6Y8lpELuFJhwLQybDkMmQFxCp1ZP4S4i0hzOwd7PYdN7kUA/UOyP9ekHudNg8q2+ZT64AHrFdodRhbuISGNqa2H3x75FvinoMz+00x/rO8iHeN6d/s/MU6BX1xp8qHAXEQGorYEdqyNBvmkJHAmGKaZlQf5ZkDfdd7MMOBn8cuddlsJdRHqmmirYtiLSzbL5fag44I/1y4ORFwet8+mQMazLh3lDCncR6RmqymHr8qBV/h5sWQZVwUZyA0fA2L/2rfJh0yA9K7Z1bQcKdxEJp8rDPsA3LfFdLcVFUFMBGJwwBk693gf5sOmQkhnr2rY7hbuIhEN5KWxeGrkBuu0jqK0G6wVDxgcjWab7IYrJ/WNd2w6ncBeR7qlsX+TG56b3/M1QVwu94iFrIky7w4d5zmRISot1bTudwl1EuoeDO6NGsiyGXet8eVySn8p/5j/6MM8+DRKSY1vXLkDhLiJdU2nx0VP5937my+P7Qu7pMPabMOwM30qPS4xtXbsghbuIxJ5zfup+9FT+kk3+WGI6DJsKE6/3YT6kAHrHx7a+3YDCXUQ6n3Ow51Mf4nUrJh7c5o8lD/CjWKbc5rtZThgT86n83ZHCXUQ6Xm2t7yPftDjSzVK2xx9LOcGHeN3sz4Eju9xU/u5I4S4i7a+mGnasOnoqf3mJP5aeCyd/LRLm/U/sdrM/u4Mmw93MFgCXALucc2ODsv7AfwF5wEbgKufcfjMz4FHgYqAMuME592HHVF1EuozqSj+uvH4q/1KoPOiP9T8JTvkG5J3hu1sycmNb1x6iOS33hcBjwNNRZfcA/+ecm2tm9wTP/wm4CBge/JwO/Cr4U0TCpOqIn/FZN2FoywdQfcQfyzwFCq7yLfPcaZA2JLZ17aGaDHfn3DtmltegeCZwdvD4N8Db+HCfCTztnHPA+2aWYWZDnHPb263GItL5Kg7BlqWRbpaty6GmEjAYPA4m3RCE+VToOzDWtRVa3+d+QlRg7wBOCB5nAVuizisOyr4S7mY2G5gNkJurr2kiXcqREr9KYl03y7YV4GrAesPQCXD6d3w3S87p0CcjtnWVRrX5hqpzzpmZa8XrngCeACgsLGzx60WkHR3eG3Xz8z3YEbWRc9YkOOMu31+eczokpsS6ttIMrQ33nXXdLWY2BNgVlG8FcqLOyw7KRKQrObgjMiRx02K/4xAEGzmfBmf/wId5diHE94ltXaVVWhvurwCzgLnBny9Hlf+dmT2Hv5Faqv52kS6gZHNk5uemxbDvC1+ekOqn8hdc7YclDj210zdylo7RnKGQz+Jvng40s2LgPnyoP29mNwObgKuC0/+AHwa5AT8U8sYOqLOIHI9zPryjW+alwa2wpAzfIi+8ObKRc29Ndwmj5oyWufYYh85r5FwHzGlrpUSkBZzz3Sr1Yb4EDu3wx/pm+hCfdof/c9Bozf7sIfQrW6S7qa2BnWsiKyZu/guU7fXHUodC/oxgh6EzYOBwzf7soRTuIl1dTRVsXxlZZGvz+1BR6o9lDIMRF0a2i+uXpzAXQOEu0vVUVwQbOQdhvmUZVB32xwYMh7GXR23knB3bukqXpXAXibXKMij+IBLmxR8EGzkDg8bAhOsii2ylDIptXaXbULiLdLbyA741Xjf7c+uHUFvlN3IeXACn3RKZyt8DNnKWjqFwF+loZfuCqfzBWuY7VgUbOcfB0IkwdU4wlX8yJKXHurYSEgp3kfZ2aFdkfPmmJbBzLX4qf6LfvHnG93zLPPs0SOgb69pKSCncRdqqdGtkTZZNS/z2cQDxyX4tlnN+6MN86ESIT4ptXaXHULiLtIRzfuPm6Kn8+zf6Y4lpkDsFJnzLd7MMGa+NnCVmFO4ix+Mc7N1w9FT+A8FaeH36+REsk/82mMo/Ths5S5ehcBeJVlsLu9dHtcyXwOFg0dO+gyJDEodNh8xRmsovXZbCXXq2mmrYuToI8yWweQkc2e+PpWXDSedEwnzASZr9Kd2Gwl16lupK2L4i0s2y+f2ojZxPhFF/5ddkGTYN+g2LaVVF2kLhLuFWVQ5biyLdLMUfQFWZPzZwJBRcGZnKnzY0tnUVaUcKdwmXysPBRs5LfKBvLYps5HzCWJj4bR/kudMgJTPWtRXpMAp36d7KS6Nmfy72XS611X4j5yHj4fS/9S3z3Cl+dItID6Fwl+7l8F5/07NuLfOda4Kp/PF+I+fpfx+1kXNqrGsrEjMKd+naDu6MzPzcuNgPUwSIS/LT98/8fmQqvzZyFqmncJeupWTL0VP5927w5QkpvjU+7go/+3PoqRCXGNu6inRhCneJnbqNnOsX2VoMJZv9saR0f9Nz4izfMh88Xhs5i7SA/rVI53EOdn8SCfJNS+Dgdn8seaDvK58yx4f5oNGayt9DVVVVUVxcTHl5eayr0mUkJSWRnZ1NfHzz1ypSuPcEFYf82O7O3sWntjaykXNdN0v9Rs5DIuPL886AgSM0+1MAKC4uJjU1lby8PEz/T+CcY+/evRQXF5Ofn9/s1yncw6qqHDb8CVa/CJ/+L5z6LfirX3TsZ9ZURzZy3rQYNv/FD1UEyMiF4Rf4QM+bDv3yFebSqPLycgV7FDNjwIAB7N69u0WvU7iHSU0VfPFnWPMirH/VT6tPHgin/g2Mv6b9P6+6ArZ9FJnKv2UpVB7yxwacDKNnRqbyZ+S0/+dLaCnYj9aav482hbuZ3QXcAjhgNXAjMAR4DhgALAeud85VtuVz5Dhqa/2479UvwrqX4cg+SEyHMTNh7F9D3pntdyOysuyrU/mrg37RQaP9L5C6rpbUwe3zmSLSKq3+V29mWcAdwGjn3BEzex64BrgY+KVz7jkzexy4GfhVu9RWPOf8psprFsHal/xNyfhkGHkRjL0CTj6vfYYJVhz0rfG6FRO3Lo/ayHkcFN4UzP6cCn0HtP3zRLqIjRs3cskll7BmzZpmnf/444+TnJzMt7/97WOes3DhQoqKinjssce+cuxnP/sZ9957b6vr25i2NunigD5mVgUkA9uBc4HrguO/Ae5H4d52zsGudT7Q1yzyu//0ToCTz4dxfw0jLmz7fpxH9vup/HXdLNtXgqsJNnI+Fabe7rtZck/XRs4iUb7zne+06fVdKtydc1vNbB6wGTgC/BHfDVPinKsOTisGstpcy55s7+eRQN/9sV8z5cSz/MzMUX8FfTJa/96HdkdN5V/sR7bg/C+N7NNgxt2+ZZ59GiSmtNcViTTbv/z3WtZtO9Cu7zl6aBr3fWNMk+fV1NRw6623smTJErKysnj55ZfZtm0bc+bMYffu3SQnJ/Pkk08yatQo7r//flJSUvje977HBx98wM0330yvXr04//zzef311+u/AWzbto0LL7yQzz//nMsvv5yHHnqIe+65hyNHjjBhwgTGjBnDM8880y7X2ZZumX7ATCAfKAFeAC5swetnA7MBcnNzW1uNcCothjUv+UDfvsKX5U7zo11Omdn61QwPbI+MZNm4GPZ84svj+kDOZDjnXt9fnlWojZylx/vss8949tlnefLJJ7nqqqtYtGgRTz31FI8//jjDhw9n6dKl3H777bz55ptHve7GG2/kySefZOrUqdxzzz1HHVuxYgUfffQRiYmJjBw5ku9+97vMnTuXxx57jBUrVrRr/dvSLfM14Evn3G4AM3sJmA5kmFlc0HrPBrY29mLn3BPAEwCFhYWuDfUIh0O7Yd3vfaBv/osvG3oqXPBTGHM5pLfwC5BzfrZndJjv/9IfS0gNNnK+1rfMh0yAuIT2vBqRdtGcFnZHyc/PZ8KECQBMmjSJjRs3smTJEq688sr6cyoqKo56TUlJCQcPHmTq1KkAXHfddbz66qv1x8877zzS032X5ujRo9m0aRM5OR0zkqwt4b4ZmGJmyfhumfOAIuAt4Ar8iJlZwMttrWRoHSmBj1/1I12+/LNf3TBzFJzzzzD2m35bt+ZyznfhRC+ydaDYH+vTz7f8J9/qW+YnjNNUfpEmJCZGBiX07t2bnTt3kpGR0aYWdsP3rK6uPs7ZbdOWPvelZvYi8CFQDXyEb4m/BjxnZg8EZfPbo6KhUXkYPnndt9A3vOE3kuiXB2fc7YcunjC6ee9TW+v74KOn8h/a6Y/1zQwmC93pwzzzFG3kLNJGaWlp5Ofn88ILL3DllVfinGPVqlWMHz++/pyMjAxSU1NZunQpp59+Os8991yz3js+Pp6qqqoWLS/QlDY135xz9wH3NSj+ApjclvcNneoKH+SrX4RP/8cvBZA6FCbP9i30oRObnq1ZWwM7VkctsrXEj2kHSMuC/LMiU/kHnKzZnyId4JlnnuG2227jgQceoKqqimuuueaocAeYP38+t956K7169eKss86q74Y5ntmzZ1NQUMDEiRPb7YaqORf77u7CwkJXVFQU62q0r5pq39WyZpGfLVpRCn36w5jL/Fj03KnHb03XVMG2FVFT+d+HimDUQL+8yMzPvOmQMUxhLqGxfv16TjnllFhXo9UOHTpESoofXTZ37ly2b9/Oo48+2ub3bezvxcyWO+cKGztfHa/tqbYWtrwfmS1atgcS02DUJX4sev5Z0PsYX7uqyv0kobpFtrYsi9rIeYTvsqmb/dnSm6si0mlee+01fv7zn1NdXc2wYcNYuHBhTOqhcG8r5/z6KmsW+eGLB7f5oYUjLwxmi36t8WGFlYd9gNd1sxQXQU0FfiPnMXDq9T7Ih03XRs4i3cjVV1/N1VdfHetqKNxbbdf6yOSifV/4PTxP/hpc8BM/W7ThpJ/yA5GNnDct9r8Qaqv9VP4h44ORLMFGzsn9Y3NNIhIaCveW2PdFZHLRrnU+mPPPhDPuglO+4Ycc1inb58er1y2ytWNV1EbOE2Had32/ec5kSEqL3TWJSCgp3JtyYBus/Z3vR9/2oS/LmQIXPexvjtZtgHFolz+vbpGtXWt9ef1Gzv8YmcqfkByTSxGRnkPh3pjDe4LZoi/5oMb5rpPzfwxjvunXJi8thi/ejiyytfcz/9r4vn5hrbGX+5Z51kRt5CwinU7hXqe8FD5+zbfQv3jbr4Y4cASc/QM/UqVXb9+98tbPgo2cN/nXJabDsKkw8Xof5kMKjj0iRkRC7ZZbbuHuu+9m9OhjT0a84YYbuOSSS7jiiiuOKq9b3uC66647xitbpmeHe2WZn1S0ZhF89kc/WzQj1/eHDynwywNs/gv85ht+FAxA8oBgI+fbfDfLCWO0kbOIAPDrX/+61a/duHEjv/3tbxXurVZdAZ+/6Vvon7wOVYeh7yDIOd1v2lx9BD76T1i8x5+fckJk389h02HgSE3lF+ksr9/jZ2a3p8Hj4KK5xz3l4YcfJjExkTvuuIO77rqLlStX8uabb/Lmm28yf/58Zs2axX333UdFRQUnnXQSTz31FCkpKZx99tnMmzePwsJC5s+fz4MPPkhGRgbjx48nMTGxfqOOd955h0ceeYQdO3bw0EMPccUVV3DPPfewfv16JkyYwKxZs7jrrrvadJk9I9xrqmHju8Heov8d2bQZ/AqJ1RX+OEB6rh/SWBfm/U/U7E+RHmbGjBn84he/4I477qCoqIiKigqqqqp49913KSgo4IEHHuCNN96gb9++PPjggzzyyCP86Ec/qn/9tm3b+MlPfsKHH35Iamoq55577lHLFGzfvp333nuPjz/+mEsvvZQrrriCuXPnMm/evKNWkWyL8IZ7bS0ULwtmi/4eDh9j5/CUQZE1WYZN890yItI1NNHC7iiTJk1i+fLlHDhwgMTERCZOnEhRURHvvvsul156KevWrWP69OkAVFZW1i/xW2fZsmWcddZZ9O/v56xceeWVfPrpp/XHL7vsMnr16sXo0aPZuXNnh1xDuMLdOb813JoXYc3vIkveRsscFelmyZ0GaUM6v54i0qXFx8eTn5/PwoULmTZtGgUFBbz11lts2LCB/Px8zj//fJ599tlWv3/00r8dtb5XOMJ99ye+hb5mEez7POqA+f61+jCfCn0HxqyaItJ9zJgxg3nz5rFgwQLGjRvH3XffzaRJk5gyZQpz5sxhw4YNnHzyyRw+fJitW7cyYsSI+teedtpp3Hnnnezfv5/U1FQWLVrEuHHjjvt5qampHDx4sN3q373D/cB2+O1VfvYn+P1FsyYFYX6Gv0nalj1GRaTHmjFjBj/96U+ZOnUqffv2JSkpiRkzZpCZmcnChQu59tpr63dieuCBB44K96ysLO69914mT55M//79GTVqVJNL/xYUFNC7d2/Gjx/PDTfc0OYbqt17yd8D2+D1f/Lj0YdN82GujZxFurXuvuRvnbqlf6urq7n88su56aabuPzyy1v9fj1ryd+0oXD1f8S6FiIiX3H//ffzxhtvUF5ezgUXXMBll13WqZ/fvcNdRKSLmjdvXkw/X7NxRKTL6QrdxV1Ja/4+FO4i0qUkJSWxd+9eBXzAOcfevXtJSmpk05/jULeMiHQp2dnZFBcXs3v3MSYe9kBJSUlkZ2e36DUKdxHpUuomEEnbqFtGRCSEFO4iIiGkcBcRCaEuMUPVzHYDm2Jdj1YaCOyJdSVipKdee0+9bui5195Vr3uYcy6zsQNdIty7MzMrOtb037DrqdfeU68beu61d8frVreMiEgIKdxFREJI4d52T8S6AjHUU6+9p1439Nxr73bXrT53EZEQUstdRCSEFO4iIiGkcG8BM8sxs7fMbJ2ZrTWzv29w/B/MzJlZqDZqPd51m9l3zezjoPyhWNazIxzr2s1sgpm9b2YrzKzIzCbHuq7tycySzGyZma0MrvtfgvJ8M1tqZhvM7L/MLCHWdW1vx7n2Z8zsEzNbY2YLzCw+1nU9Luecfpr5AwwBJgaPU4FPgdHB8xzgf/GTsQbGuq6dcd3AOcAbQGJwbFCs69qJ1/5H4KKg/GLg7VjXtZ2v24CU4HE8sBSYAjwPXBOUPw7cFuu6duK1XxwcM+DZrn7tarm3gHNuu3Puw+DxQWA9kBUc/iXwfSB0d6iPc923AXOdcxXBsV2xq2XHOM61OyAtOC0d2BabGnYM5x0KnsYHPw44F3gxKP8NcFnn165jHevanXN/CI45YBnQsjV4O5nCvZXMLA84FVhqZjOBrc65lbGtVceLvm5gBDAj+Jr+ZzM7LaaV62ANrv1O4GEz2wLMA34Qu5p1DDPrbWYrgF3An4DPgRLnXHVwSjGRxk2oNLx259zSqGPxwPXA/8Soes2icG8FM0sBFuH/gVcD9wI/imWdOkP0dTvnDuD3A+iP/8r6j8DzZmYxrGKHaeTabwPucs7lAHcB82NZv47gnKtxzk3At1AnA6NiW6PO0/DazWxs1OF/B95xzr0bk8o1k8K9hYLf2ouAZ5xzLwEnAfnASjPbiP+f4UMzGxy7Wra/Rq4bfMvtpeCb6jKgFr/AUqgc49pnAXWPX8CHXyg550qAt4CpQIaZ1W3ykw1sjVW9OkPUtV8IYGb3AZnA3TGsVrMo3FsgaJXOB9Y75x4BcM6tds4Ncs7lOefy8IE30Tm3I4ZVbVeNXXfg9/ibqpjZCCCBrrlyXqsd59q3AWcFj88FPuvsunUkM8s0s4zgcR/gfPz9hreAK4LTZgEvx6SCHegY1/6xmd0CfB241jlXG8MqNotmqLaAmZ0BvAusxrdSAe51zv0h6pyNQKFzLjQhd6zrxo+UWQBMACqB7znn3oxFHTvKca79APAovmuqHLjdObc8JpXsAGZWgL9h2hvfCHzeOfdjMzsReA7fHfcR8Dd1N9TD4jjXXo0fDXcwOPUl59yPY1TNJincRURCSN0yIiIhpHAXEQkhhbuISAgp3EVEQkjhLiISQgp3EZEQUrhLpzOzPDNb08rXnm1mr7Z3nTqSmRWa2b+28DX3m9n3OqpOEn5xTZ8iIm3hnCsCimJdD+lZ1HKXWIkLNj9Yb2YvmlmymZ1nZh+Z2epgM4READO7MNgQ5EPgm0FZLzP7zMwyo55vqHvekJktNLNfBRtsfBF8A1gQfP7CqPN+FWy+Ub9JQ1A+N9iwY5WZzQvKrgw2blhpZu8c60Kjv20ELfIFZvZ2UI87os77oZl9ambvASOjyk8ys/8xs+Vm9q6ZjTKzODP7wMzODs75uZn9tMX/FSS8Yr2gvH563g+Qh18bfHrwfAHwz8AWYERQ9jR+1c2koHw4fpOE54FXg3Puw6/SCHABsOg4n7kQP23egJn45QPG4Rs4y4EJwXn9gz97A28DBcAA4BMiM7ozgj9XA1nRZcf47LOj6nw/sARIxC+ythe/Xvik4P2S8evEb8Av5wDwf8Dw4PHpwJvB4zH49V6+hl8KICHW/23103V+1HKXWNninFscPP5P4DzgS+fcp0HZb4Az8cvMfumc+8w554Jz6ywAvh08vgl4qonP/O/gPVYDO51f9K0WWIv/hQNwVfAN4SN8eI4GSvHrx8w3s28CZcG5i4GFZnYr/pdBc73mnKtwfv2hXcAJwAzgd865MueXFH4F6pcanga8EKwv/v/wu0PhnFsL/AfwKnCTc66yBXWQkFO4S6w0XNSopMVv4NwWYKeZnYtfcvf1Jl5St8BVbdTjuudxZpYPfA84zzlXALwGJDm/OcVk/A5ElxBs0uCc+w7+G0cOsNzMBjSz6tGfXcPx7331wm+QMSHq55So4+Pwf3eDmvnZ0kMo3CVWcs1savD4OvwNxzwzOzkoux74M/BxUH5SUH5tg/f5Nb41/4JzrqaNdUoDDgOlZnYCcBHUt57TnV/98y5gfFB+knNuqXPuR8BufMi31jvAZWbWx8xSgW8ABK34L83syuAzzczqPv+b+NUZzwT+rW6ZWhFQuEvsfALMMbP1QD/8HrQ34rsf6pbXfdw5Vw7MBl4Luksa7tP6CpBC010yTXJ+m8SP8L9QfovvdgG/MfarZrYKeI/IRg0PBzd/1+D70Vu9zaLz+7T+V/AerwMfRB3+FnCzma3EdyHNNLOBwFzglqAr6zH8EsQigJb8lW7OzAqBXzrnZsS6LiJdica5S7dlZvfg9zL9VqzrItLVqOUuoWJmPwSubFD8gnOuw8eAm9nXgQcbFH/pnLu8oz9bpCGFu4hICOmGqohICCncRURCSOEuIhJCCncRkRD6/6y/m6rt3y1FAAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}],"source":["people = people.assign(\n","    body_mass_index = people[\"weight\"] / (people[\"height\"] / 100) ** 2\n",")\n","people.plot(kind = \"line\", x = \"body_mass_index\", y = [\"height\", \"weight\"])\n","plt.show()"]},{"cell_type":"markdown","metadata":{"id":"l0DjpOZJbdkj"},"source":["Again, there are way too many options to list here: the best option is to scroll through the [Visualization](http://pandas.pydata.org/pandas-docs/stable/visualization.html) page in pandas' documentation, find the plot you are interested in and look at the example code."]},{"cell_type":"markdown","source":["#### Operations on `DataFrame`s"],"metadata":{"id":"2tR7pbKywTiP"}},{"cell_type":"markdown","metadata":{"id":"6m0s18-Hbdkj"},"source":["Although `DataFrame`s do not try to mimick NumPy arrays, there are a few similarities. Let's create a `DataFrame` to demonstrate this:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"-D1z8SV2bdkj","outputId":"d128e487-675f-428b-9219-0dd4bd9e6ada","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677381954030,"user_tz":-480,"elapsed":293,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         sep  oct  nov\n","alice      8    8    9\n","bob       10    9    9\n","charles    4    8    2\n","darwin     9   10   10"],"text/html":["\n","  <div id=\"df-da40268f-0757-44d9-a922-53c8529a909c\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sep</th>\n","      <th>oct</th>\n","      <th>nov</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>8</td>\n","      <td>8</td>\n","      <td>9</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>10</td>\n","      <td>9</td>\n","      <td>9</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>4</td>\n","      <td>8</td>\n","      <td>2</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>9</td>\n","      <td>10</td>\n","      <td>10</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-da40268f-0757-44d9-a922-53c8529a909c')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-da40268f-0757-44d9-a922-53c8529a909c button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-da40268f-0757-44d9-a922-53c8529a909c');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":71}],"source":["grades_array = np.array([[8,8,9],[10,9,9],[4, 8, 2], [9, 10, 10]])\n","grades = pd.DataFrame(grades_array, columns=[\"sep\", \"oct\", \"nov\"], index=[\"alice\",\"bob\",\"charles\",\"darwin\"])\n","grades"]},{"cell_type":"markdown","metadata":{"id":"YNYV5RC2bdkj"},"source":["You can apply NumPy mathematical functions on a `DataFrame`: the function is applied to all values:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"tVS8LYkqbdkj","outputId":"49080743-aa75-4381-c144-9f2b138375da","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677381959214,"user_tz":-480,"elapsed":344,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["              sep       oct       nov\n","alice    2.828427  2.828427  3.000000\n","bob      3.162278  3.000000  3.000000\n","charles  2.000000  2.828427  1.414214\n","darwin   3.000000  3.162278  3.162278"],"text/html":["\n","  <div id=\"df-012eb8b5-dfd0-4cba-96d7-76c4aa21dc7e\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sep</th>\n","      <th>oct</th>\n","      <th>nov</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>2.828427</td>\n","      <td>2.828427</td>\n","      <td>3.000000</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>3.162278</td>\n","      <td>3.000000</td>\n","      <td>3.000000</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>2.000000</td>\n","      <td>2.828427</td>\n","      <td>1.414214</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>3.000000</td>\n","      <td>3.162278</td>\n","      <td>3.162278</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-012eb8b5-dfd0-4cba-96d7-76c4aa21dc7e')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-012eb8b5-dfd0-4cba-96d7-76c4aa21dc7e button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-012eb8b5-dfd0-4cba-96d7-76c4aa21dc7e');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":72}],"source":["np.sqrt(grades)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"H9ZQAzESbdkj","outputId":"6d583c1d-c917-4f56-fa92-441d477d7875","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677381962665,"user_tz":-480,"elapsed":399,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         sep  oct  nov\n","alice      9    9   10\n","bob       11   10   10\n","charles    5    9    3\n","darwin    10   11   11"],"text/html":["\n","  <div id=\"df-da7ccc63-96da-405f-b220-2826678f6cbe\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sep</th>\n","      <th>oct</th>\n","      <th>nov</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>9</td>\n","      <td>9</td>\n","      <td>10</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>11</td>\n","      <td>10</td>\n","      <td>10</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>5</td>\n","      <td>9</td>\n","      <td>3</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>10</td>\n","      <td>11</td>\n","      <td>11</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-da7ccc63-96da-405f-b220-2826678f6cbe')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-da7ccc63-96da-405f-b220-2826678f6cbe button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-da7ccc63-96da-405f-b220-2826678f6cbe');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":73}],"source":["grades + 1"]},{"cell_type":"markdown","metadata":{"id":"tUIPt4C-bdkk"},"source":["Aggregation operations, such as computing the `max()`, the `sum()` or the `mean()` of a `DataFrame`, apply to each column, and you get back a `Series` object:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1C8lhBIXbdkk","outputId":"0966a0da-e1d4-40fc-eadd-56caf13d30bc","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677381984289,"user_tz":-480,"elapsed":306,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["sep    7.75\n","oct    8.75\n","nov    7.50\n","dtype: float64"]},"metadata":{},"execution_count":74}],"source":["grades.mean()"]},{"cell_type":"markdown","metadata":{"id":"qWRNPxVGbdkl"},"source":["Most of these functions take an optional `axis` parameter which lets you specify along which axis of the `DataFrame` you want the operation executed. The default is `axis=0`, meaning that the operation is executed vertically (on each column). You can set `axis=1` to execute the operation horizontally (on each row). For example, let's find out which students had all grades greater than `5`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"HoyAlWC6bdkl","outputId":"31f5ef5a-0e36-4706-9314-b6da0cd0ff9f","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677382028176,"user_tz":-480,"elapsed":3,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["alice       True\n","bob         True\n","charles    False\n","darwin      True\n","dtype: bool"]},"metadata":{},"execution_count":75}],"source":["(grades > 5).all(axis = 1)"]},{"cell_type":"markdown","metadata":{"id":"mdIwFwAgbdkl"},"source":["If you add a `Series` object to a `DataFrame` (or execute any other binary operation), Pandas attempts to broadcast the operation to all *rows* in the `DataFrame`. This only works if the `Series` has the same size as the `DataFrame`s rows. For example, let's subtract the `mean` of the `DataFrame` (a `Series` object) from the `DataFrame`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"sUzEkkg1bdkl","outputId":"e211169f-e50c-447a-a887-66c606d29b0e","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677382078296,"user_tz":-480,"elapsed":297,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["          sep   oct  nov\n","alice    0.25 -0.75  1.5\n","bob      2.25  0.25  1.5\n","charles -3.75 -0.75 -5.5\n","darwin   1.25  1.25  2.5"],"text/html":["\n","  <div id=\"df-a9ae8eb8-8a16-4174-b8af-2431db464326\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sep</th>\n","      <th>oct</th>\n","      <th>nov</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>0.25</td>\n","      <td>-0.75</td>\n","      <td>1.5</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>2.25</td>\n","      <td>0.25</td>\n","      <td>1.5</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>-3.75</td>\n","      <td>-0.75</td>\n","      <td>-5.5</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>1.25</td>\n","      <td>1.25</td>\n","      <td>2.5</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a9ae8eb8-8a16-4174-b8af-2431db464326')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-a9ae8eb8-8a16-4174-b8af-2431db464326 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-a9ae8eb8-8a16-4174-b8af-2431db464326');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":76}],"source":["grades - grades.mean()  # equivalent to: grades - [7.75, 8.75, 7.50]"]},{"cell_type":"markdown","metadata":{"id":"cc1Lz1SSbdkm"},"source":["If you want to subtract the global mean from every grade, here is one way to do it:"]},{"cell_type":"code","execution_count":null,"metadata":{"scrolled":true,"id":"nCJUsodMbdkm","outputId":"3625954b-5ee0-4f16-ee50-9b96b697b827","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677382334450,"user_tz":-480,"elapsed":480,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         sep  oct  nov\n","alice    0.0  0.0  1.0\n","bob      2.0  1.0  1.0\n","charles -4.0  0.0 -6.0\n","darwin   1.0  2.0  2.0"],"text/html":["\n","  <div id=\"df-18fb2034-257a-4bbf-b4ba-f9bdf3d7e9ef\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sep</th>\n","      <th>oct</th>\n","      <th>nov</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>1.0</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>2.0</td>\n","      <td>1.0</td>\n","      <td>1.0</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>-4.0</td>\n","      <td>0.0</td>\n","      <td>-6.0</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>1.0</td>\n","      <td>2.0</td>\n","      <td>2.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-18fb2034-257a-4bbf-b4ba-f9bdf3d7e9ef')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-18fb2034-257a-4bbf-b4ba-f9bdf3d7e9ef button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-18fb2034-257a-4bbf-b4ba-f9bdf3d7e9ef');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":78}],"source":["grades - grades.values.mean() # subtracts the global mean (8.00) from all grades"]},{"cell_type":"markdown","source":["The following shows the behavior of `nan`"],"metadata":{"id":"lJ63V0Qix7G8"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"Qeykr6KIbdkm","outputId":"8d49f6a9-0305-4f99-cadc-75b562cd3143","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677382387692,"user_tz":-480,"elapsed":278,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         oct  nov  dec\n","bob      0.0  NaN  2.0\n","colin    NaN  1.0  0.0\n","darwin   0.0  1.0  0.0\n","charles  3.0  3.0  0.0"],"text/html":["\n","  <div id=\"df-e39e8c3d-06d3-4925-898d-bcd128b3e3f6\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>oct</th>\n","      <th>nov</th>\n","      <th>dec</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>bob</th>\n","      <td>0.0</td>\n","      <td>NaN</td>\n","      <td>2.0</td>\n","    </tr>\n","    <tr>\n","      <th>colin</th>\n","      <td>NaN</td>\n","      <td>1.0</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>0.0</td>\n","      <td>1.0</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>3.0</td>\n","      <td>3.0</td>\n","      <td>0.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e39e8c3d-06d3-4925-898d-bcd128b3e3f6')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-e39e8c3d-06d3-4925-898d-bcd128b3e3f6 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-e39e8c3d-06d3-4925-898d-bcd128b3e3f6');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":79}],"source":["bonus_array = np.array([[0,np.nan,2],[np.nan,1,0],[0, 1, 0], [3, 3, 0]])\n","bonus_points = pd.DataFrame(bonus_array, columns=[\"oct\", \"nov\", \"dec\"], index=[\"bob\",\"colin\", \"darwin\", \"charles\"])\n","bonus_points"]},{"cell_type":"code","execution_count":null,"metadata":{"scrolled":true,"id":"PmBlxZ2bbdkm","outputId":"73574006-1cd8-4e35-ba41-39496a27bcb1","colab":{"base_uri":"https://localhost:8080/","height":206},"executionInfo":{"status":"ok","timestamp":1677382395137,"user_tz":-480,"elapsed":295,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         dec   nov   oct  sep\n","alice    NaN   NaN   NaN  NaN\n","bob      NaN   NaN   9.0  NaN\n","charles  NaN   5.0  11.0  NaN\n","colin    NaN   NaN   NaN  NaN\n","darwin   NaN  11.0  10.0  NaN"],"text/html":["\n","  <div id=\"df-8966be32-712d-4e5a-a33d-6856c8501dfa\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>dec</th>\n","      <th>nov</th>\n","      <th>oct</th>\n","      <th>sep</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>9.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>NaN</td>\n","      <td>5.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>colin</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>10.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8966be32-712d-4e5a-a33d-6856c8501dfa')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-8966be32-712d-4e5a-a33d-6856c8501dfa button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-8966be32-712d-4e5a-a33d-6856c8501dfa');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":80}],"source":["grades + bonus_points"]},{"cell_type":"markdown","source":["#### Handling missing data"],"metadata":{"id":"z_zoKdfCyGae"}},{"cell_type":"markdown","metadata":{"id":"ZEtUuQNHbdkm"},"source":["Dealing with missing data is a frequent task when working with real life data. Pandas offers a few tools to handle missing data.\n"," \n","Let's try to fix the problem above. For example, we can decide that missing data should result in a zero, instead of `NaN`. We can replace all `NaN` values by a any value using the `fillna()` method:"]},{"cell_type":"code","execution_count":null,"metadata":{"scrolled":true,"id":"89vMid5vbdkm","outputId":"11285425-cd80-440b-8fea-a18adb29c9c9","colab":{"base_uri":"https://localhost:8080/","height":206},"executionInfo":{"status":"ok","timestamp":1677382419485,"user_tz":-480,"elapsed":301,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         dec   nov   oct  sep\n","alice    0.0   0.0   0.0  0.0\n","bob      0.0   0.0   9.0  0.0\n","charles  0.0   5.0  11.0  0.0\n","colin    0.0   0.0   0.0  0.0\n","darwin   0.0  11.0  10.0  0.0"],"text/html":["\n","  <div id=\"df-d6bd5a43-e5d6-423e-a2ee-547f55a15311\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>dec</th>\n","      <th>nov</th>\n","      <th>oct</th>\n","      <th>sep</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>9.0</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>0.0</td>\n","      <td>5.0</td>\n","      <td>11.0</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>colin</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>0.0</td>\n","      <td>11.0</td>\n","      <td>10.0</td>\n","      <td>0.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d6bd5a43-e5d6-423e-a2ee-547f55a15311')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-d6bd5a43-e5d6-423e-a2ee-547f55a15311 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-d6bd5a43-e5d6-423e-a2ee-547f55a15311');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":81}],"source":["(grades + bonus_points).fillna(0)"]},{"cell_type":"code","execution_count":null,"metadata":{"scrolled":true,"id":"Nb9szHYrbdko","outputId":"da581bd7-db3e-4c6b-cbfe-cf5df23ba27e","colab":{"base_uri":"https://localhost:8080/","height":206},"executionInfo":{"status":"ok","timestamp":1677382430246,"user_tz":-480,"elapsed":296,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         dec   nov   oct  sep\n","alice    NaN   NaN   NaN  NaN\n","bob      NaN   NaN   9.0  NaN\n","charles  NaN   5.0  11.0  NaN\n","colin    NaN   NaN   NaN  NaN\n","darwin   NaN  11.0  10.0  NaN"],"text/html":["\n","  <div id=\"df-d8ba295d-d22c-438b-82a9-bbd4227ab97f\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>dec</th>\n","      <th>nov</th>\n","      <th>oct</th>\n","      <th>sep</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>9.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>NaN</td>\n","      <td>5.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>colin</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>10.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d8ba295d-d22c-438b-82a9-bbd4227ab97f')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-d8ba295d-d22c-438b-82a9-bbd4227ab97f button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-d8ba295d-d22c-438b-82a9-bbd4227ab97f');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":82}],"source":["final_grades = grades + bonus_points\n","final_grades"]},{"cell_type":"markdown","metadata":{"id":"84Ov3x5Ubdko"},"source":["We can call the `dropna()` method to get rid of rows that are full of `NaN`s:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ACmmOPyrbdko","outputId":"0f45c062-967c-4b4d-9561-4c320c6bc63b","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677382443500,"user_tz":-480,"elapsed":270,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         dec   nov   oct  sep\n","bob      NaN   NaN   9.0  NaN\n","charles  NaN   5.0  11.0  NaN\n","darwin   NaN  11.0  10.0  NaN"],"text/html":["\n","  <div id=\"df-fe4b09cc-af50-4a65-bbba-0f887c85dbc8\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>dec</th>\n","      <th>nov</th>\n","      <th>oct</th>\n","      <th>sep</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>bob</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>9.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>NaN</td>\n","      <td>5.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>10.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fe4b09cc-af50-4a65-bbba-0f887c85dbc8')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-fe4b09cc-af50-4a65-bbba-0f887c85dbc8 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-fe4b09cc-af50-4a65-bbba-0f887c85dbc8');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":83}],"source":["final_grades_clean = final_grades.dropna(how=\"all\")\n","final_grades_clean"]},{"cell_type":"markdown","metadata":{"id":"F_ft7zzcbdko"},"source":["Now let's remove columns that are full of `NaN`s by setting the `axis` argument to `1`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"DeqnrLEnbdko","outputId":"98bd6066-caad-4d2b-8d8d-584d851ccfd9","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677382454351,"user_tz":-480,"elapsed":370,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["          nov   oct\n","bob       NaN   9.0\n","charles   5.0  11.0\n","darwin   11.0  10.0"],"text/html":["\n","  <div id=\"df-bc17d255-74ed-4a9b-ace1-1f04302934e3\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>nov</th>\n","      <th>oct</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>bob</th>\n","      <td>NaN</td>\n","      <td>9.0</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>5.0</td>\n","      <td>11.0</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>11.0</td>\n","      <td>10.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bc17d255-74ed-4a9b-ace1-1f04302934e3')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-bc17d255-74ed-4a9b-ace1-1f04302934e3 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-bc17d255-74ed-4a9b-ace1-1f04302934e3');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":84}],"source":["final_grades_clean = final_grades_clean.dropna(axis=1, how=\"all\")\n","final_grades_clean"]},{"cell_type":"markdown","source":["#### Aggregating with `groupby`"],"metadata":{"id":"VAkGd0MCySaP"}},{"cell_type":"markdown","metadata":{"id":"NbyYrlrzbdko"},"source":["Similar to the SQL language, pandas allows grouping your data into groups to run calculations over each group.\n","\n","First, let's add some extra data about each person so we can group them, and let's go back to the `final_grades` `DataFrame` so we can see how `NaN` values are handled:"]},{"cell_type":"code","execution_count":null,"metadata":{"scrolled":true,"id":"II2IdemTbdkp","outputId":"e62d921c-6450-4bbc-8ae5-c79fa26620e7","colab":{"base_uri":"https://localhost:8080/","height":206},"executionInfo":{"status":"ok","timestamp":1677382475284,"user_tz":-480,"elapsed":324,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         dec   nov   oct  sep    hobby\n","alice    NaN   NaN   NaN  NaN   Biking\n","bob      NaN   NaN   9.0  NaN  Dancing\n","charles  NaN   5.0  11.0  NaN      NaN\n","colin    NaN   NaN   NaN  NaN  Dancing\n","darwin   NaN  11.0  10.0  NaN   Biking"],"text/html":["\n","  <div id=\"df-328c6c04-6e46-4156-afc7-731498176260\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>dec</th>\n","      <th>nov</th>\n","      <th>oct</th>\n","      <th>sep</th>\n","      <th>hobby</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>Biking</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>9.0</td>\n","      <td>NaN</td>\n","      <td>Dancing</td>\n","    </tr>\n","    <tr>\n","      <th>charles</th>\n","      <td>NaN</td>\n","      <td>5.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>colin</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>Dancing</td>\n","    </tr>\n","    <tr>\n","      <th>darwin</th>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>10.0</td>\n","      <td>NaN</td>\n","      <td>Biking</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-328c6c04-6e46-4156-afc7-731498176260')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-328c6c04-6e46-4156-afc7-731498176260 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-328c6c04-6e46-4156-afc7-731498176260');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":85}],"source":["final_grades[\"hobby\"] = [\"Biking\", \"Dancing\", np.nan, \"Dancing\", \"Biking\"]\n","final_grades"]},{"cell_type":"markdown","metadata":{"id":"2suQBYhGbdkp"},"source":["Now let's group data in this `DataFrame` by hobby:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"dOhALCgkbdkp"},"outputs":[],"source":["grouped_grades = final_grades.groupby(\"hobby\")"]},{"cell_type":"markdown","metadata":{"id":"DZAFvTlTbdkp"},"source":["We are ready to compute the average grade per hobby:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"o5Gs6lcLbdkp","outputId":"afdbea76-8c33-4696-8b76-a3430bb1c697","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677382516264,"user_tz":-480,"elapsed":281,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         dec   nov   oct  sep\n","hobby                        \n","Biking   NaN  11.0  10.0  NaN\n","Dancing  NaN   NaN   9.0  NaN"],"text/html":["\n","  <div id=\"df-53cbe11d-62cb-4427-9013-043a0863ff09\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>dec</th>\n","      <th>nov</th>\n","      <th>oct</th>\n","      <th>sep</th>\n","    </tr>\n","    <tr>\n","      <th>hobby</th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>Biking</th>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>10.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>Dancing</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>9.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-53cbe11d-62cb-4427-9013-043a0863ff09')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-53cbe11d-62cb-4427-9013-043a0863ff09 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-53cbe11d-62cb-4427-9013-043a0863ff09');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":87}],"source":["grouped_grades.mean()"]},{"cell_type":"markdown","metadata":{"id":"pgOZUDdRbdkp"},"source":["That was easy! Note that the `NaN` values have simply been skipped when computing the means."]},{"cell_type":"markdown","source":["#### Pivot tables (Optional)"],"metadata":{"id":"LV79_gGTyvhU"}},{"cell_type":"markdown","metadata":{"id":"PB0GOgYLbdkp"},"source":["Pandas supports spreadsheet-like [pivot tables](https://en.wikipedia.org/wiki/Pivot_table) that allow quick data summarization."]},{"cell_type":"markdown","source":["#### Overview functions"],"metadata":{"id":"8EG2nW8wzvTF"}},{"cell_type":"markdown","metadata":{"id":"rK5gDiX6bdkr"},"source":["When dealing with large `DataFrames`, it is useful to get a quick overview of its content. Pandas offers a few functions for this. First, let's create a large `DataFrame` with a mix of numeric values, missing values and text values. Notice how Jupyter displays only the corners of the `DataFrame`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"6u57h9MQbdkr","outputId":"16c58852-9247-4c3a-b529-e12fc3763807","colab":{"base_uri":"https://localhost:8080/","height":424},"executionInfo":{"status":"ok","timestamp":1677382861029,"user_tz":-480,"elapsed":332,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         A     B     C some_text      D      E      F     G      H      I  \\\n","0      NaN  11.0  44.0    Blabla   99.0    NaN   88.0  22.0  165.0  143.0   \n","1     11.0  22.0  55.0    Blabla  110.0    NaN   99.0  33.0    NaN  154.0   \n","2     22.0  33.0  66.0    Blabla  121.0   11.0  110.0  44.0    NaN  165.0   \n","3     33.0  44.0  77.0    Blabla  132.0   22.0  121.0  55.0   11.0    NaN   \n","4     44.0  55.0  88.0    Blabla  143.0   33.0  132.0  66.0   22.0    NaN   \n","...    ...   ...   ...       ...    ...    ...    ...   ...    ...    ...   \n","9995   NaN   NaN  33.0    Blabla   88.0  165.0   77.0  11.0  154.0  132.0   \n","9996   NaN  11.0  44.0    Blabla   99.0    NaN   88.0  22.0  165.0  143.0   \n","9997  11.0  22.0  55.0    Blabla  110.0    NaN   99.0  33.0    NaN  154.0   \n","9998  22.0  33.0  66.0    Blabla  121.0   11.0  110.0  44.0    NaN  165.0   \n","9999  33.0  44.0  77.0    Blabla  132.0   22.0  121.0  55.0   11.0    NaN   \n","\n","      ...     Q     R     S     T      U      V      W     X      Y      Z  \n","0     ...  11.0   NaN  11.0  44.0   99.0    NaN   88.0  22.0  165.0  143.0  \n","1     ...  22.0  11.0  22.0  55.0  110.0    NaN   99.0  33.0    NaN  154.0  \n","2     ...  33.0  22.0  33.0  66.0  121.0   11.0  110.0  44.0    NaN  165.0  \n","3     ...  44.0  33.0  44.0  77.0  132.0   22.0  121.0  55.0   11.0    NaN  \n","4     ...  55.0  44.0  55.0  88.0  143.0   33.0  132.0  66.0   22.0    NaN  \n","...   ...   ...   ...   ...   ...    ...    ...    ...   ...    ...    ...  \n","9995  ...   NaN   NaN   NaN  33.0   88.0  165.0   77.0  11.0  154.0  132.0  \n","9996  ...  11.0   NaN  11.0  44.0   99.0    NaN   88.0  22.0  165.0  143.0  \n","9997  ...  22.0  11.0  22.0  55.0  110.0    NaN   99.0  33.0    NaN  154.0  \n","9998  ...  33.0  22.0  33.0  66.0  121.0   11.0  110.0  44.0    NaN  165.0  \n","9999  ...  44.0  33.0  44.0  77.0  132.0   22.0  121.0  55.0   11.0    NaN  \n","\n","[10000 rows x 27 columns]"],"text/html":["\n","  <div id=\"df-67305636-1b16-4653-9603-8e2078264f5f\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>A</th>\n","      <th>B</th>\n","      <th>C</th>\n","      <th>some_text</th>\n","      <th>D</th>\n","      <th>E</th>\n","      <th>F</th>\n","      <th>G</th>\n","      <th>H</th>\n","      <th>I</th>\n","      <th>...</th>\n","      <th>Q</th>\n","      <th>R</th>\n","      <th>S</th>\n","      <th>T</th>\n","      <th>U</th>\n","      <th>V</th>\n","      <th>W</th>\n","      <th>X</th>\n","      <th>Y</th>\n","      <th>Z</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>44.0</td>\n","      <td>Blabla</td>\n","      <td>99.0</td>\n","      <td>NaN</td>\n","      <td>88.0</td>\n","      <td>22.0</td>\n","      <td>165.0</td>\n","      <td>143.0</td>\n","      <td>...</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>44.0</td>\n","      <td>99.0</td>\n","      <td>NaN</td>\n","      <td>88.0</td>\n","      <td>22.0</td>\n","      <td>165.0</td>\n","      <td>143.0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>11.0</td>\n","      <td>22.0</td>\n","      <td>55.0</td>\n","      <td>Blabla</td>\n","      <td>110.0</td>\n","      <td>NaN</td>\n","      <td>99.0</td>\n","      <td>33.0</td>\n","      <td>NaN</td>\n","      <td>154.0</td>\n","      <td>...</td>\n","      <td>22.0</td>\n","      <td>11.0</td>\n","      <td>22.0</td>\n","      <td>55.0</td>\n","      <td>110.0</td>\n","      <td>NaN</td>\n","      <td>99.0</td>\n","      <td>33.0</td>\n","      <td>NaN</td>\n","      <td>154.0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>22.0</td>\n","      <td>33.0</td>\n","      <td>66.0</td>\n","      <td>Blabla</td>\n","      <td>121.0</td>\n","      <td>11.0</td>\n","      <td>110.0</td>\n","      <td>44.0</td>\n","      <td>NaN</td>\n","      <td>165.0</td>\n","      <td>...</td>\n","      <td>33.0</td>\n","      <td>22.0</td>\n","      <td>33.0</td>\n","      <td>66.0</td>\n","      <td>121.0</td>\n","      <td>11.0</td>\n","      <td>110.0</td>\n","      <td>44.0</td>\n","      <td>NaN</td>\n","      <td>165.0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>33.0</td>\n","      <td>44.0</td>\n","      <td>77.0</td>\n","      <td>Blabla</td>\n","      <td>132.0</td>\n","      <td>22.0</td>\n","      <td>121.0</td>\n","      <td>55.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","      <td>...</td>\n","      <td>44.0</td>\n","      <td>33.0</td>\n","      <td>44.0</td>\n","      <td>77.0</td>\n","      <td>132.0</td>\n","      <td>22.0</td>\n","      <td>121.0</td>\n","      <td>55.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>44.0</td>\n","      <td>55.0</td>\n","      <td>88.0</td>\n","      <td>Blabla</td>\n","      <td>143.0</td>\n","      <td>33.0</td>\n","      <td>132.0</td>\n","      <td>66.0</td>\n","      <td>22.0</td>\n","      <td>NaN</td>\n","      <td>...</td>\n","      <td>55.0</td>\n","      <td>44.0</td>\n","      <td>55.0</td>\n","      <td>88.0</td>\n","      <td>143.0</td>\n","      <td>33.0</td>\n","      <td>132.0</td>\n","      <td>66.0</td>\n","      <td>22.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>9995</th>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>33.0</td>\n","      <td>Blabla</td>\n","      <td>88.0</td>\n","      <td>165.0</td>\n","      <td>77.0</td>\n","      <td>11.0</td>\n","      <td>154.0</td>\n","      <td>132.0</td>\n","      <td>...</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>33.0</td>\n","      <td>88.0</td>\n","      <td>165.0</td>\n","      <td>77.0</td>\n","      <td>11.0</td>\n","      <td>154.0</td>\n","      <td>132.0</td>\n","    </tr>\n","    <tr>\n","      <th>9996</th>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>44.0</td>\n","      <td>Blabla</td>\n","      <td>99.0</td>\n","      <td>NaN</td>\n","      <td>88.0</td>\n","      <td>22.0</td>\n","      <td>165.0</td>\n","      <td>143.0</td>\n","      <td>...</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>44.0</td>\n","      <td>99.0</td>\n","      <td>NaN</td>\n","      <td>88.0</td>\n","      <td>22.0</td>\n","      <td>165.0</td>\n","      <td>143.0</td>\n","    </tr>\n","    <tr>\n","      <th>9997</th>\n","      <td>11.0</td>\n","      <td>22.0</td>\n","      <td>55.0</td>\n","      <td>Blabla</td>\n","      <td>110.0</td>\n","      <td>NaN</td>\n","      <td>99.0</td>\n","      <td>33.0</td>\n","      <td>NaN</td>\n","      <td>154.0</td>\n","      <td>...</td>\n","      <td>22.0</td>\n","      <td>11.0</td>\n","      <td>22.0</td>\n","      <td>55.0</td>\n","      <td>110.0</td>\n","      <td>NaN</td>\n","      <td>99.0</td>\n","      <td>33.0</td>\n","      <td>NaN</td>\n","      <td>154.0</td>\n","    </tr>\n","    <tr>\n","      <th>9998</th>\n","      <td>22.0</td>\n","      <td>33.0</td>\n","      <td>66.0</td>\n","      <td>Blabla</td>\n","      <td>121.0</td>\n","      <td>11.0</td>\n","      <td>110.0</td>\n","      <td>44.0</td>\n","      <td>NaN</td>\n","      <td>165.0</td>\n","      <td>...</td>\n","      <td>33.0</td>\n","      <td>22.0</td>\n","      <td>33.0</td>\n","      <td>66.0</td>\n","      <td>121.0</td>\n","      <td>11.0</td>\n","      <td>110.0</td>\n","      <td>44.0</td>\n","      <td>NaN</td>\n","      <td>165.0</td>\n","    </tr>\n","    <tr>\n","      <th>9999</th>\n","      <td>33.0</td>\n","      <td>44.0</td>\n","      <td>77.0</td>\n","      <td>Blabla</td>\n","      <td>132.0</td>\n","      <td>22.0</td>\n","      <td>121.0</td>\n","      <td>55.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","      <td>...</td>\n","      <td>44.0</td>\n","      <td>33.0</td>\n","      <td>44.0</td>\n","      <td>77.0</td>\n","      <td>132.0</td>\n","      <td>22.0</td>\n","      <td>121.0</td>\n","      <td>55.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>10000 rows × 27 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-67305636-1b16-4653-9603-8e2078264f5f')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-67305636-1b16-4653-9603-8e2078264f5f button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-67305636-1b16-4653-9603-8e2078264f5f');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":90}],"source":["much_data = np.fromfunction(lambda x,y: (x+y*y)%17*11, (10000, 26))\n","large_df = pd.DataFrame(much_data, columns=list(\"ABCDEFGHIJKLMNOPQRSTUVWXYZ\"))\n","large_df[large_df % 16 == 0] = np.nan\n","large_df.insert(3,\"some_text\", \"Blabla\")\n","large_df"]},{"cell_type":"markdown","metadata":{"id":"qwsVdaFNbdkr"},"source":["The `head()` method returns the top 5 rows:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZD-qCazAbdkr","outputId":"48732cb7-dca8-47a0-cc33-26344b123d76","colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"status":"ok","timestamp":1677382874499,"user_tz":-480,"elapsed":286,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["      A     B     C some_text      D     E      F     G      H      I  ...  \\\n","0   NaN  11.0  44.0    Blabla   99.0   NaN   88.0  22.0  165.0  143.0  ...   \n","1  11.0  22.0  55.0    Blabla  110.0   NaN   99.0  33.0    NaN  154.0  ...   \n","2  22.0  33.0  66.0    Blabla  121.0  11.0  110.0  44.0    NaN  165.0  ...   \n","3  33.0  44.0  77.0    Blabla  132.0  22.0  121.0  55.0   11.0    NaN  ...   \n","4  44.0  55.0  88.0    Blabla  143.0  33.0  132.0  66.0   22.0    NaN  ...   \n","\n","      Q     R     S     T      U     V      W     X      Y      Z  \n","0  11.0   NaN  11.0  44.0   99.0   NaN   88.0  22.0  165.0  143.0  \n","1  22.0  11.0  22.0  55.0  110.0   NaN   99.0  33.0    NaN  154.0  \n","2  33.0  22.0  33.0  66.0  121.0  11.0  110.0  44.0    NaN  165.0  \n","3  44.0  33.0  44.0  77.0  132.0  22.0  121.0  55.0   11.0    NaN  \n","4  55.0  44.0  55.0  88.0  143.0  33.0  132.0  66.0   22.0    NaN  \n","\n","[5 rows x 27 columns]"],"text/html":["\n","  <div id=\"df-e00eef93-023f-4640-a1ab-9bd1febc90fa\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>A</th>\n","      <th>B</th>\n","      <th>C</th>\n","      <th>some_text</th>\n","      <th>D</th>\n","      <th>E</th>\n","      <th>F</th>\n","      <th>G</th>\n","      <th>H</th>\n","      <th>I</th>\n","      <th>...</th>\n","      <th>Q</th>\n","      <th>R</th>\n","      <th>S</th>\n","      <th>T</th>\n","      <th>U</th>\n","      <th>V</th>\n","      <th>W</th>\n","      <th>X</th>\n","      <th>Y</th>\n","      <th>Z</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>44.0</td>\n","      <td>Blabla</td>\n","      <td>99.0</td>\n","      <td>NaN</td>\n","      <td>88.0</td>\n","      <td>22.0</td>\n","      <td>165.0</td>\n","      <td>143.0</td>\n","      <td>...</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","      <td>11.0</td>\n","      <td>44.0</td>\n","      <td>99.0</td>\n","      <td>NaN</td>\n","      <td>88.0</td>\n","      <td>22.0</td>\n","      <td>165.0</td>\n","      <td>143.0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>11.0</td>\n","      <td>22.0</td>\n","      <td>55.0</td>\n","      <td>Blabla</td>\n","      <td>110.0</td>\n","      <td>NaN</td>\n","      <td>99.0</td>\n","      <td>33.0</td>\n","      <td>NaN</td>\n","      <td>154.0</td>\n","      <td>...</td>\n","      <td>22.0</td>\n","      <td>11.0</td>\n","      <td>22.0</td>\n","      <td>55.0</td>\n","      <td>110.0</td>\n","      <td>NaN</td>\n","      <td>99.0</td>\n","      <td>33.0</td>\n","      <td>NaN</td>\n","      <td>154.0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>22.0</td>\n","      <td>33.0</td>\n","      <td>66.0</td>\n","      <td>Blabla</td>\n","      <td>121.0</td>\n","      <td>11.0</td>\n","      <td>110.0</td>\n","      <td>44.0</td>\n","      <td>NaN</td>\n","      <td>165.0</td>\n","      <td>...</td>\n","      <td>33.0</td>\n","      <td>22.0</td>\n","      <td>33.0</td>\n","      <td>66.0</td>\n","      <td>121.0</td>\n","      <td>11.0</td>\n","      <td>110.0</td>\n","      <td>44.0</td>\n","      <td>NaN</td>\n","      <td>165.0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>33.0</td>\n","      <td>44.0</td>\n","      <td>77.0</td>\n","      <td>Blabla</td>\n","      <td>132.0</td>\n","      <td>22.0</td>\n","      <td>121.0</td>\n","      <td>55.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","      <td>...</td>\n","      <td>44.0</td>\n","      <td>33.0</td>\n","      <td>44.0</td>\n","      <td>77.0</td>\n","      <td>132.0</td>\n","      <td>22.0</td>\n","      <td>121.0</td>\n","      <td>55.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>44.0</td>\n","      <td>55.0</td>\n","      <td>88.0</td>\n","      <td>Blabla</td>\n","      <td>143.0</td>\n","      <td>33.0</td>\n","      <td>132.0</td>\n","      <td>66.0</td>\n","      <td>22.0</td>\n","      <td>NaN</td>\n","      <td>...</td>\n","      <td>55.0</td>\n","      <td>44.0</td>\n","      <td>55.0</td>\n","      <td>88.0</td>\n","      <td>143.0</td>\n","      <td>33.0</td>\n","      <td>132.0</td>\n","      <td>66.0</td>\n","      <td>22.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>5 rows × 27 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e00eef93-023f-4640-a1ab-9bd1febc90fa')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-e00eef93-023f-4640-a1ab-9bd1febc90fa button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-e00eef93-023f-4640-a1ab-9bd1febc90fa');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":91}],"source":["large_df.head()"]},{"cell_type":"markdown","metadata":{"id":"An6ZPfSvbdkr"},"source":["Of course there's also a `tail()` function to view the bottom 5 rows. You can pass the number of rows you want:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"a4I2ghvbbdkr","outputId":"088b2801-1ee4-489f-e001-646b93867f16","colab":{"base_uri":"https://localhost:8080/","height":141},"executionInfo":{"status":"ok","timestamp":1677382876960,"user_tz":-480,"elapsed":280,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         A     B     C some_text      D     E      F     G     H      I  ...  \\\n","9998  22.0  33.0  66.0    Blabla  121.0  11.0  110.0  44.0   NaN  165.0  ...   \n","9999  33.0  44.0  77.0    Blabla  132.0  22.0  121.0  55.0  11.0    NaN  ...   \n","\n","         Q     R     S     T      U     V      W     X     Y      Z  \n","9998  33.0  22.0  33.0  66.0  121.0  11.0  110.0  44.0   NaN  165.0  \n","9999  44.0  33.0  44.0  77.0  132.0  22.0  121.0  55.0  11.0    NaN  \n","\n","[2 rows x 27 columns]"],"text/html":["\n","  <div id=\"df-bbd6b26f-712d-43b4-9968-c51bf02bfd24\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>A</th>\n","      <th>B</th>\n","      <th>C</th>\n","      <th>some_text</th>\n","      <th>D</th>\n","      <th>E</th>\n","      <th>F</th>\n","      <th>G</th>\n","      <th>H</th>\n","      <th>I</th>\n","      <th>...</th>\n","      <th>Q</th>\n","      <th>R</th>\n","      <th>S</th>\n","      <th>T</th>\n","      <th>U</th>\n","      <th>V</th>\n","      <th>W</th>\n","      <th>X</th>\n","      <th>Y</th>\n","      <th>Z</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>9998</th>\n","      <td>22.0</td>\n","      <td>33.0</td>\n","      <td>66.0</td>\n","      <td>Blabla</td>\n","      <td>121.0</td>\n","      <td>11.0</td>\n","      <td>110.0</td>\n","      <td>44.0</td>\n","      <td>NaN</td>\n","      <td>165.0</td>\n","      <td>...</td>\n","      <td>33.0</td>\n","      <td>22.0</td>\n","      <td>33.0</td>\n","      <td>66.0</td>\n","      <td>121.0</td>\n","      <td>11.0</td>\n","      <td>110.0</td>\n","      <td>44.0</td>\n","      <td>NaN</td>\n","      <td>165.0</td>\n","    </tr>\n","    <tr>\n","      <th>9999</th>\n","      <td>33.0</td>\n","      <td>44.0</td>\n","      <td>77.0</td>\n","      <td>Blabla</td>\n","      <td>132.0</td>\n","      <td>22.0</td>\n","      <td>121.0</td>\n","      <td>55.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","      <td>...</td>\n","      <td>44.0</td>\n","      <td>33.0</td>\n","      <td>44.0</td>\n","      <td>77.0</td>\n","      <td>132.0</td>\n","      <td>22.0</td>\n","      <td>121.0</td>\n","      <td>55.0</td>\n","      <td>11.0</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>2 rows × 27 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bbd6b26f-712d-43b4-9968-c51bf02bfd24')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-bbd6b26f-712d-43b4-9968-c51bf02bfd24 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-bbd6b26f-712d-43b4-9968-c51bf02bfd24');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":92}],"source":["large_df.tail(n=2)"]},{"cell_type":"markdown","metadata":{"id":"WtzFculCbdkr"},"source":["The `info()` method prints out a summary of each columns contents:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"m0kK-Undbdkr","outputId":"5f5c35f6-ead9-4cbd-a561-35071e44bc5b","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677382879205,"user_tz":-480,"elapsed":291,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 10000 entries, 0 to 9999\n","Data columns (total 27 columns):\n"," #   Column     Non-Null Count  Dtype  \n","---  ------     --------------  -----  \n"," 0   A          8823 non-null   float64\n"," 1   B          8824 non-null   float64\n"," 2   C          8824 non-null   float64\n"," 3   some_text  10000 non-null  object \n"," 4   D          8824 non-null   float64\n"," 5   E          8822 non-null   float64\n"," 6   F          8824 non-null   float64\n"," 7   G          8824 non-null   float64\n"," 8   H          8822 non-null   float64\n"," 9   I          8823 non-null   float64\n"," 10  J          8823 non-null   float64\n"," 11  K          8822 non-null   float64\n"," 12  L          8824 non-null   float64\n"," 13  M          8824 non-null   float64\n"," 14  N          8822 non-null   float64\n"," 15  O          8824 non-null   float64\n"," 16  P          8824 non-null   float64\n"," 17  Q          8824 non-null   float64\n"," 18  R          8823 non-null   float64\n"," 19  S          8824 non-null   float64\n"," 20  T          8824 non-null   float64\n"," 21  U          8824 non-null   float64\n"," 22  V          8822 non-null   float64\n"," 23  W          8824 non-null   float64\n"," 24  X          8824 non-null   float64\n"," 25  Y          8822 non-null   float64\n"," 26  Z          8823 non-null   float64\n","dtypes: float64(26), object(1)\n","memory usage: 2.1+ MB\n"]}],"source":["large_df.info()"]},{"cell_type":"markdown","metadata":{"id":"-93LVb6xbdks"},"source":["Finally, the `describe()` method gives a nice overview of the main aggregated values over each column:\n","* `count`: number of non-null (not NaN) values\n","* `mean`: mean of non-null values\n","* `std`: [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of non-null values\n","* `min`: minimum of non-null values\n","* `25%`, `50%`, `75%`: 25th, 50th and 75th [percentile](https://en.wikipedia.org/wiki/Percentile) of non-null values\n","* `max`: maximum of non-null values"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"DqG1O-2Cbdks","outputId":"3ded5901-5761-40a2-969d-b0a2ea22e7f0","colab":{"base_uri":"https://localhost:8080/","height":394},"executionInfo":{"status":"ok","timestamp":1677382886145,"user_tz":-480,"elapsed":305,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["                 A            B            C            D            E  \\\n","count  8823.000000  8824.000000  8824.000000  8824.000000  8822.000000   \n","mean     87.977559    87.972575    87.987534    88.012466    87.983791   \n","std      47.535911    47.535523    47.521679    47.521679    47.535001   \n","min      11.000000    11.000000    11.000000    11.000000    11.000000   \n","25%      44.000000    44.000000    44.000000    44.000000    44.000000   \n","50%      88.000000    88.000000    88.000000    88.000000    88.000000   \n","75%     132.000000   132.000000   132.000000   132.000000   132.000000   \n","max     165.000000   165.000000   165.000000   165.000000   165.000000   \n","\n","                 F            G            H            I            J  ...  \\\n","count  8824.000000  8824.000000  8822.000000  8823.000000  8823.000000  ...   \n","mean     88.007480    87.977561    88.000000    88.022441    88.022441  ...   \n","std      47.519371    47.529755    47.536879    47.535911    47.535911  ...   \n","min      11.000000    11.000000    11.000000    11.000000    11.000000  ...   \n","25%      44.000000    44.000000    44.000000    44.000000    44.000000  ...   \n","50%      88.000000    88.000000    88.000000    88.000000    88.000000  ...   \n","75%     132.000000   132.000000   132.000000   132.000000   132.000000  ...   \n","max     165.000000   165.000000   165.000000   165.000000   165.000000  ...   \n","\n","                 Q            R            S            T            U  \\\n","count  8824.000000  8823.000000  8824.000000  8824.000000  8824.000000   \n","mean     87.972575    87.977559    87.972575    87.987534    88.012466   \n","std      47.535523    47.535911    47.535523    47.521679    47.521679   \n","min      11.000000    11.000000    11.000000    11.000000    11.000000   \n","25%      44.000000    44.000000    44.000000    44.000000    44.000000   \n","50%      88.000000    88.000000    88.000000    88.000000    88.000000   \n","75%     132.000000   132.000000   132.000000   132.000000   132.000000   \n","max     165.000000   165.000000   165.000000   165.000000   165.000000   \n","\n","                 V            W            X            Y            Z  \n","count  8822.000000  8824.000000  8824.000000  8822.000000  8823.000000  \n","mean     87.983791    88.007480    87.977561    88.000000    88.022441  \n","std      47.535001    47.519371    47.529755    47.536879    47.535911  \n","min      11.000000    11.000000    11.000000    11.000000    11.000000  \n","25%      44.000000    44.000000    44.000000    44.000000    44.000000  \n","50%      88.000000    88.000000    88.000000    88.000000    88.000000  \n","75%     132.000000   132.000000   132.000000   132.000000   132.000000  \n","max     165.000000   165.000000   165.000000   165.000000   165.000000  \n","\n","[8 rows x 26 columns]"],"text/html":["\n","  <div id=\"df-cf8c2dcf-f3c2-4227-8724-b671cd462c54\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>A</th>\n","      <th>B</th>\n","      <th>C</th>\n","      <th>D</th>\n","      <th>E</th>\n","      <th>F</th>\n","      <th>G</th>\n","      <th>H</th>\n","      <th>I</th>\n","      <th>J</th>\n","      <th>...</th>\n","      <th>Q</th>\n","      <th>R</th>\n","      <th>S</th>\n","      <th>T</th>\n","      <th>U</th>\n","      <th>V</th>\n","      <th>W</th>\n","      <th>X</th>\n","      <th>Y</th>\n","      <th>Z</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>count</th>\n","      <td>8823.000000</td>\n","      <td>8824.000000</td>\n","      <td>8824.000000</td>\n","      <td>8824.000000</td>\n","      <td>8822.000000</td>\n","      <td>8824.000000</td>\n","      <td>8824.000000</td>\n","      <td>8822.000000</td>\n","      <td>8823.000000</td>\n","      <td>8823.000000</td>\n","      <td>...</td>\n","      <td>8824.000000</td>\n","      <td>8823.000000</td>\n","      <td>8824.000000</td>\n","      <td>8824.000000</td>\n","      <td>8824.000000</td>\n","      <td>8822.000000</td>\n","      <td>8824.000000</td>\n","      <td>8824.000000</td>\n","      <td>8822.000000</td>\n","      <td>8823.000000</td>\n","    </tr>\n","    <tr>\n","      <th>mean</th>\n","      <td>87.977559</td>\n","      <td>87.972575</td>\n","      <td>87.987534</td>\n","      <td>88.012466</td>\n","      <td>87.983791</td>\n","      <td>88.007480</td>\n","      <td>87.977561</td>\n","      <td>88.000000</td>\n","      <td>88.022441</td>\n","      <td>88.022441</td>\n","      <td>...</td>\n","      <td>87.972575</td>\n","      <td>87.977559</td>\n","      <td>87.972575</td>\n","      <td>87.987534</td>\n","      <td>88.012466</td>\n","      <td>87.983791</td>\n","      <td>88.007480</td>\n","      <td>87.977561</td>\n","      <td>88.000000</td>\n","      <td>88.022441</td>\n","    </tr>\n","    <tr>\n","      <th>std</th>\n","      <td>47.535911</td>\n","      <td>47.535523</td>\n","      <td>47.521679</td>\n","      <td>47.521679</td>\n","      <td>47.535001</td>\n","      <td>47.519371</td>\n","      <td>47.529755</td>\n","      <td>47.536879</td>\n","      <td>47.535911</td>\n","      <td>47.535911</td>\n","      <td>...</td>\n","      <td>47.535523</td>\n","      <td>47.535911</td>\n","      <td>47.535523</td>\n","      <td>47.521679</td>\n","      <td>47.521679</td>\n","      <td>47.535001</td>\n","      <td>47.519371</td>\n","      <td>47.529755</td>\n","      <td>47.536879</td>\n","      <td>47.535911</td>\n","    </tr>\n","    <tr>\n","      <th>min</th>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>...</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","      <td>11.000000</td>\n","    </tr>\n","    <tr>\n","      <th>25%</th>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>...</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","      <td>44.000000</td>\n","    </tr>\n","    <tr>\n","      <th>50%</th>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>...</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","      <td>88.000000</td>\n","    </tr>\n","    <tr>\n","      <th>75%</th>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>...</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","      <td>132.000000</td>\n","    </tr>\n","    <tr>\n","      <th>max</th>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>...</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","      <td>165.000000</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>8 rows × 26 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cf8c2dcf-f3c2-4227-8724-b671cd462c54')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-cf8c2dcf-f3c2-4227-8724-b671cd462c54 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-cf8c2dcf-f3c2-4227-8724-b671cd462c54');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":94}],"source":["large_df.describe()"]},{"cell_type":"markdown","source":["#### Saving & loading"],"metadata":{"id":"H5Og8gGHz8Vn"}},{"cell_type":"markdown","metadata":{"id":"-LghiXSTbdks"},"source":["Pandas can save `DataFrame`s to various backends, including file formats such as CSV, Excel, JSON, HTML and HDF5, or to a SQL database. Let's create a `DataFrame` to demonstrate this:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"F6dGE_DDbdks","outputId":"3fce1520-f215-40ec-be1c-978ec3ad0766","colab":{"base_uri":"https://localhost:8080/","height":112},"executionInfo":{"status":"ok","timestamp":1677382934763,"user_tz":-480,"elapsed":3,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         hobby  weight  birthyear  children\n","alice   Biking    68.5       1985       NaN\n","bob    Dancing    83.1       1984       3.0"],"text/html":["\n","  <div id=\"df-eb2ce3af-4bb5-4c5f-9194-1dc5b6e5b09c\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>hobby</th>\n","      <th>weight</th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>Biking</td>\n","      <td>68.5</td>\n","      <td>1985</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>Dancing</td>\n","      <td>83.1</td>\n","      <td>1984</td>\n","      <td>3.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-eb2ce3af-4bb5-4c5f-9194-1dc5b6e5b09c')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-eb2ce3af-4bb5-4c5f-9194-1dc5b6e5b09c button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-eb2ce3af-4bb5-4c5f-9194-1dc5b6e5b09c');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":95}],"source":["my_df = pd.DataFrame(\n","    [[\"Biking\", 68.5, 1985, np.nan], [\"Dancing\", 83.1, 1984, 3]], \n","    columns=[\"hobby\",\"weight\",\"birthyear\",\"children\"],\n","    index=[\"alice\", \"bob\"]\n",")\n","my_df"]},{"cell_type":"markdown","metadata":{"id":"cjas67GYbdks"},"source":["Let's save it to CSV, HTML and JSON:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JUW20lWIbdku"},"outputs":[],"source":["my_df.to_csv(\"my_df.csv\")\n","my_df.to_html(\"my_df.html\")\n","my_df.to_json(\"my_df.json\")"]},{"cell_type":"markdown","metadata":{"id":"wkwqS47Obdku"},"source":["Done! Let's take a peek at what was saved:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"XiXTvwh6bdku","outputId":"86d80515-7d04-4b61-d938-2453b1814a9f","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677382939395,"user_tz":-480,"elapsed":398,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["# my_df.csv\n",",hobby,weight,birthyear,children\n","alice,Biking,68.5,1985,\n","bob,Dancing,83.1,1984,3.0\n","\n","\n","# my_df.html\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>hobby</th>\n","      <th>weight</th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>Biking</td>\n","      <td>68.5</td>\n","      <td>1985</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>Dancing</td>\n","      <td>83.1</td>\n","      <td>1984</td>\n","      <td>3.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","\n","# my_df.json\n","{\"hobby\":{\"alice\":\"Biking\",\"bob\":\"Dancing\"},\"weight\":{\"alice\":68.5,\"bob\":83.1},\"birthyear\":{\"alice\":1985,\"bob\":1984},\"children\":{\"alice\":null,\"bob\":3.0}}\n","\n"]}],"source":["for filename in (\"my_df.csv\", \"my_df.html\", \"my_df.json\"):\n","    print(\"#\", filename)\n","    with open(filename, \"rt\") as f:\n","        print(f.read())\n","        print()\n"]},{"cell_type":"markdown","metadata":{"id":"zX9DuLcJbdkv"},"source":["Note that the index is saved as the first column (with no name) in a CSV file, as `<th>` tags in HTML and as keys in JSON.\n","\n","Saving to other formats works very similarly, but some formats require extra libraries to be installed. For example, saving to Excel requires the `openpyxl` library:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"DUfZubs3bdkv"},"outputs":[],"source":["try:\n","    my_df.to_excel(\"my_df.xlsx\", sheet_name='People')\n","except ImportError as e:\n","    print(e)"]},{"cell_type":"markdown","metadata":{"id":"A92UjaI8bdkv"},"source":["Now let's load our CSV file back into a `DataFrame`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"6HKE0X1Ybdkv","outputId":"45d399ad-124e-4199-9b11-e933b6476a34","colab":{"base_uri":"https://localhost:8080/","height":112},"executionInfo":{"status":"ok","timestamp":1677382970385,"user_tz":-480,"elapsed":302,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["         hobby  weight  birthyear  children\n","alice   Biking    68.5       1985       NaN\n","bob    Dancing    83.1       1984       3.0"],"text/html":["\n","  <div id=\"df-98c45a30-312a-47f0-8ae7-faa275539320\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>hobby</th>\n","      <th>weight</th>\n","      <th>birthyear</th>\n","      <th>children</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>alice</th>\n","      <td>Biking</td>\n","      <td>68.5</td>\n","      <td>1985</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>bob</th>\n","      <td>Dancing</td>\n","      <td>83.1</td>\n","      <td>1984</td>\n","      <td>3.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-98c45a30-312a-47f0-8ae7-faa275539320')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-98c45a30-312a-47f0-8ae7-faa275539320 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-98c45a30-312a-47f0-8ae7-faa275539320');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":99}],"source":["my_df_loaded = pd.read_csv(\"my_df.csv\", index_col=0)\n","my_df_loaded"]},{"cell_type":"markdown","metadata":{"id":"-cs24XxSbdkv"},"source":["As you might guess, there are similar `read_json`, `read_html`, `read_excel` functions as well.  We can also read data straight from the Internet. For example, let's load the top 1,000 U.S. cities from github:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"J6WRgAnObdkv","outputId":"3848616a-c16a-4dd2-c693-51e26258076c","colab":{"base_uri":"https://localhost:8080/","height":238},"executionInfo":{"status":"ok","timestamp":1677382979218,"user_tz":-480,"elapsed":755,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["                     State  Population        lat         lon\n","City                                                         \n","Marysville      Washington       63269  48.051764 -122.177082\n","Perris          California       72326  33.782519 -117.228648\n","Cleveland             Ohio      390113  41.499320  -81.694361\n","Worcester    Massachusetts      182544  42.262593  -71.802293\n","Columbia    South Carolina      133358  34.000710  -81.034814"],"text/html":["\n","  <div id=\"df-7598272c-d041-493d-85c4-bd4cb1f0ee35\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>State</th>\n","      <th>Population</th>\n","      <th>lat</th>\n","      <th>lon</th>\n","    </tr>\n","    <tr>\n","      <th>City</th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>Marysville</th>\n","      <td>Washington</td>\n","      <td>63269</td>\n","      <td>48.051764</td>\n","      <td>-122.177082</td>\n","    </tr>\n","    <tr>\n","      <th>Perris</th>\n","      <td>California</td>\n","      <td>72326</td>\n","      <td>33.782519</td>\n","      <td>-117.228648</td>\n","    </tr>\n","    <tr>\n","      <th>Cleveland</th>\n","      <td>Ohio</td>\n","      <td>390113</td>\n","      <td>41.499320</td>\n","      <td>-81.694361</td>\n","    </tr>\n","    <tr>\n","      <th>Worcester</th>\n","      <td>Massachusetts</td>\n","      <td>182544</td>\n","      <td>42.262593</td>\n","      <td>-71.802293</td>\n","    </tr>\n","    <tr>\n","      <th>Columbia</th>\n","      <td>South Carolina</td>\n","      <td>133358</td>\n","      <td>34.000710</td>\n","      <td>-81.034814</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7598272c-d041-493d-85c4-bd4cb1f0ee35')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-7598272c-d041-493d-85c4-bd4cb1f0ee35 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-7598272c-d041-493d-85c4-bd4cb1f0ee35');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":100}],"source":["us_cities = None\n","try:\n","    csv_url = \"https://raw.githubusercontent.com/plotly/datasets/master/us-cities-top-1k.csv\"\n","    us_cities = pd.read_csv(csv_url, index_col=0)\n","    us_cities = us_cities.head()\n","except IOError as e:\n","    print(e)\n","us_cities"]},{"cell_type":"markdown","metadata":{"id":"XpWNQD_Jbdkv"},"source":["There are more options available, in particular regarding datetime format. Check out the [documentation](http://pandas.pydata.org/pandas-docs/stable/io.html) for more details."]},{"cell_type":"markdown","source":["#### Combining `DataFrame`s"],"metadata":{"id":"7Mb03gd70Vrn"}},{"cell_type":"markdown","metadata":{"id":"yHIObss4bdkv"},"source":["One powerful feature of Pandas is it's ability to perform SQL-like joins on `DataFrame`s. Various types of joins are supported: inner joins, left/right outer joins and full joins. To illustrate this, let's start by creating a couple simple `DataFrame`s:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"TgNPwsexbdkw","outputId":"3df2fe64-e3bd-4dd9-b2f0-03933be13d0a","colab":{"base_uri":"https://localhost:8080/","height":206},"executionInfo":{"status":"ok","timestamp":1677383013230,"user_tz":-480,"elapsed":302,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["  state            city        lat         lng\n","0    CA   San Francisco  37.781334 -122.416728\n","1    NY        New York  40.705649  -74.008344\n","2    FL           Miami  25.791100  -80.320733\n","3    OH       Cleveland  41.473508  -81.739791\n","4    UT  Salt Lake City  40.755851 -111.896657"],"text/html":["\n","  <div id=\"df-7355be02-5dbe-4743-9bc8-563acb6b65d6\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>state</th>\n","      <th>city</th>\n","      <th>lat</th>\n","      <th>lng</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>CA</td>\n","      <td>San Francisco</td>\n","      <td>37.781334</td>\n","      <td>-122.416728</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>NY</td>\n","      <td>New York</td>\n","      <td>40.705649</td>\n","      <td>-74.008344</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>FL</td>\n","      <td>Miami</td>\n","      <td>25.791100</td>\n","      <td>-80.320733</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>OH</td>\n","      <td>Cleveland</td>\n","      <td>41.473508</td>\n","      <td>-81.739791</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>UT</td>\n","      <td>Salt Lake City</td>\n","      <td>40.755851</td>\n","      <td>-111.896657</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7355be02-5dbe-4743-9bc8-563acb6b65d6')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-7355be02-5dbe-4743-9bc8-563acb6b65d6 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-7355be02-5dbe-4743-9bc8-563acb6b65d6');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":101}],"source":["city_loc = pd.DataFrame(\n","    [\n","        [\"CA\", \"San Francisco\", 37.781334, -122.416728],\n","        [\"NY\", \"New York\", 40.705649, -74.008344],\n","        [\"FL\", \"Miami\", 25.791100, -80.320733],\n","        [\"OH\", \"Cleveland\", 41.473508, -81.739791],\n","        [\"UT\", \"Salt Lake City\", 40.755851, -111.896657]\n","    ], columns=[\"state\", \"city\", \"lat\", \"lng\"])\n","city_loc"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"-F2yDn3cbdkw","outputId":"ff9b53cd-0dfc-43cd-cfbd-a7de3cdef383","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677383016740,"user_tz":-480,"elapsed":315,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   population           city       state\n","3      808976  San Francisco  California\n","4     8363710       New York    New-York\n","5      413201          Miami     Florida\n","6     2242193        Houston       Texas"],"text/html":["\n","  <div id=\"df-35d16286-038b-41e3-b132-db09593cd448\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>population</th>\n","      <th>city</th>\n","      <th>state</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>3</th>\n","      <td>808976</td>\n","      <td>San Francisco</td>\n","      <td>California</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>8363710</td>\n","      <td>New York</td>\n","      <td>New-York</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>413201</td>\n","      <td>Miami</td>\n","      <td>Florida</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>2242193</td>\n","      <td>Houston</td>\n","      <td>Texas</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-35d16286-038b-41e3-b132-db09593cd448')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-35d16286-038b-41e3-b132-db09593cd448 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-35d16286-038b-41e3-b132-db09593cd448');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":102}],"source":["city_pop = pd.DataFrame(\n","    [\n","        [808976, \"San Francisco\", \"California\"],\n","        [8363710, \"New York\", \"New-York\"],\n","        [413201, \"Miami\", \"Florida\"],\n","        [2242193, \"Houston\", \"Texas\"]\n","    ], index=[3,4,5,6], columns=[\"population\", \"city\", \"state\"])\n","city_pop"]},{"cell_type":"markdown","metadata":{"id":"e767etKZbdkw"},"source":["Now let's join these `DataFrame`s using the `merge()` function:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Mdztg6KPbdkw","outputId":"c5572a49-71fd-478e-a202-977abd830be3","colab":{"base_uri":"https://localhost:8080/","height":143},"executionInfo":{"status":"ok","timestamp":1677383023871,"user_tz":-480,"elapsed":265,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["  state_x           city        lat         lng  population     state_y\n","0      CA  San Francisco  37.781334 -122.416728      808976  California\n","1      NY       New York  40.705649  -74.008344     8363710    New-York\n","2      FL          Miami  25.791100  -80.320733      413201     Florida"],"text/html":["\n","  <div id=\"df-8c3c1144-c096-4f5b-8299-81bd2764ae8c\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>state_x</th>\n","      <th>city</th>\n","      <th>lat</th>\n","      <th>lng</th>\n","      <th>population</th>\n","      <th>state_y</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>CA</td>\n","      <td>San Francisco</td>\n","      <td>37.781334</td>\n","      <td>-122.416728</td>\n","      <td>808976</td>\n","      <td>California</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>NY</td>\n","      <td>New York</td>\n","      <td>40.705649</td>\n","      <td>-74.008344</td>\n","      <td>8363710</td>\n","      <td>New-York</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>FL</td>\n","      <td>Miami</td>\n","      <td>25.791100</td>\n","      <td>-80.320733</td>\n","      <td>413201</td>\n","      <td>Florida</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8c3c1144-c096-4f5b-8299-81bd2764ae8c')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-8c3c1144-c096-4f5b-8299-81bd2764ae8c button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-8c3c1144-c096-4f5b-8299-81bd2764ae8c');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":103}],"source":["pd.merge(left=city_loc, right=city_pop, on=\"city\")"]},{"cell_type":"markdown","metadata":{"id":"0aIeJRFNbdkw"},"source":["Note that both `DataFrame`s have a column named `state`, so in the result they got renamed to `state_x` and `state_y`.\n","\n","Also, note that Cleveland, Salt Lake City and Houston were dropped because they don't exist in *both* `DataFrame`s. This is the equivalent of a SQL `INNER JOIN`. If you want a `FULL OUTER JOIN`, where no city gets dropped and `NaN` values are added, you must specify `how=\"outer\"`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5p98Bdybbdkw","outputId":"9e1e705c-d8c2-4229-8b37-00ab1de368c3","colab":{"base_uri":"https://localhost:8080/","height":238},"executionInfo":{"status":"ok","timestamp":1677383043836,"user_tz":-480,"elapsed":303,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["  state_x            city        lat         lng  population     state_y\n","0      CA   San Francisco  37.781334 -122.416728    808976.0  California\n","1      NY        New York  40.705649  -74.008344   8363710.0    New-York\n","2      FL           Miami  25.791100  -80.320733    413201.0     Florida\n","3      OH       Cleveland  41.473508  -81.739791         NaN         NaN\n","4      UT  Salt Lake City  40.755851 -111.896657         NaN         NaN\n","5     NaN         Houston        NaN         NaN   2242193.0       Texas"],"text/html":["\n","  <div id=\"df-3bcbae49-47a2-4e86-91b0-a7ca9a9ddb4c\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>state_x</th>\n","      <th>city</th>\n","      <th>lat</th>\n","      <th>lng</th>\n","      <th>population</th>\n","      <th>state_y</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>CA</td>\n","      <td>San Francisco</td>\n","      <td>37.781334</td>\n","      <td>-122.416728</td>\n","      <td>808976.0</td>\n","      <td>California</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>NY</td>\n","      <td>New York</td>\n","      <td>40.705649</td>\n","      <td>-74.008344</td>\n","      <td>8363710.0</td>\n","      <td>New-York</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>FL</td>\n","      <td>Miami</td>\n","      <td>25.791100</td>\n","      <td>-80.320733</td>\n","      <td>413201.0</td>\n","      <td>Florida</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>OH</td>\n","      <td>Cleveland</td>\n","      <td>41.473508</td>\n","      <td>-81.739791</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>UT</td>\n","      <td>Salt Lake City</td>\n","      <td>40.755851</td>\n","      <td>-111.896657</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>NaN</td>\n","      <td>Houston</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>2242193.0</td>\n","      <td>Texas</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3bcbae49-47a2-4e86-91b0-a7ca9a9ddb4c')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-3bcbae49-47a2-4e86-91b0-a7ca9a9ddb4c button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-3bcbae49-47a2-4e86-91b0-a7ca9a9ddb4c');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":104}],"source":["all_cities = pd.merge(left=city_loc, right=city_pop, on=\"city\", how=\"outer\")\n","all_cities"]},{"cell_type":"markdown","metadata":{"id":"9InNQCuSbdkw"},"source":["Of course `LEFT OUTER JOIN` is also available by setting `how=\"left\"`: only the cities present in the left `DataFrame` end up in the result. Similarly, with `how=\"right\"` only cities in the right `DataFrame` appear in the result. For example:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"eJC2h_mAbdkw","outputId":"c08f5172-e69b-44f6-fde0-57859a10a3b8","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677383048336,"user_tz":-480,"elapsed":274,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["  state_x           city        lat         lng  population     state_y\n","0      CA  San Francisco  37.781334 -122.416728      808976  California\n","1      NY       New York  40.705649  -74.008344     8363710    New-York\n","2      FL          Miami  25.791100  -80.320733      413201     Florida\n","3     NaN        Houston        NaN         NaN     2242193       Texas"],"text/html":["\n","  <div id=\"df-635076ac-41c2-4b66-8a66-b6b354fab8fe\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>state_x</th>\n","      <th>city</th>\n","      <th>lat</th>\n","      <th>lng</th>\n","      <th>population</th>\n","      <th>state_y</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>CA</td>\n","      <td>San Francisco</td>\n","      <td>37.781334</td>\n","      <td>-122.416728</td>\n","      <td>808976</td>\n","      <td>California</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>NY</td>\n","      <td>New York</td>\n","      <td>40.705649</td>\n","      <td>-74.008344</td>\n","      <td>8363710</td>\n","      <td>New-York</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>FL</td>\n","      <td>Miami</td>\n","      <td>25.791100</td>\n","      <td>-80.320733</td>\n","      <td>413201</td>\n","      <td>Florida</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>NaN</td>\n","      <td>Houston</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>2242193</td>\n","      <td>Texas</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-635076ac-41c2-4b66-8a66-b6b354fab8fe')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-635076ac-41c2-4b66-8a66-b6b354fab8fe button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-635076ac-41c2-4b66-8a66-b6b354fab8fe');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":105}],"source":["pd.merge(left=city_loc, right=city_pop, on=\"city\", how=\"right\")"]},{"cell_type":"markdown","source":["#### Concatenation"],"metadata":{"id":"QTkpviRb05Ll"}},{"cell_type":"markdown","metadata":{"id":"GdkK0fRRbdkx"},"source":["Rather than joining `DataFrame`s, we may just want to concatenate them. That's what `concat()` is for:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"OB8vX0C7bdkx","outputId":"f85874d3-f6a1-4cbe-87b6-48ef41a3877e","colab":{"base_uri":"https://localhost:8080/","height":332},"executionInfo":{"status":"ok","timestamp":1677383154129,"user_tz":-480,"elapsed":265,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["        state            city        lat         lng  population\n","0          CA   San Francisco  37.781334 -122.416728         NaN\n","1          NY        New York  40.705649  -74.008344         NaN\n","2          FL           Miami  25.791100  -80.320733         NaN\n","3          OH       Cleveland  41.473508  -81.739791         NaN\n","4          UT  Salt Lake City  40.755851 -111.896657         NaN\n","3  California   San Francisco        NaN         NaN    808976.0\n","4    New-York        New York        NaN         NaN   8363710.0\n","5     Florida           Miami        NaN         NaN    413201.0\n","6       Texas         Houston        NaN         NaN   2242193.0"],"text/html":["\n","  <div id=\"df-370ececf-23eb-48b9-8aae-02cebb9fbce7\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>state</th>\n","      <th>city</th>\n","      <th>lat</th>\n","      <th>lng</th>\n","      <th>population</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>CA</td>\n","      <td>San Francisco</td>\n","      <td>37.781334</td>\n","      <td>-122.416728</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>NY</td>\n","      <td>New York</td>\n","      <td>40.705649</td>\n","      <td>-74.008344</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>FL</td>\n","      <td>Miami</td>\n","      <td>25.791100</td>\n","      <td>-80.320733</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>OH</td>\n","      <td>Cleveland</td>\n","      <td>41.473508</td>\n","      <td>-81.739791</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>UT</td>\n","      <td>Salt Lake City</td>\n","      <td>40.755851</td>\n","      <td>-111.896657</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>California</td>\n","      <td>San Francisco</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>808976.0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>New-York</td>\n","      <td>New York</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>8363710.0</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>Florida</td>\n","      <td>Miami</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>413201.0</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>Texas</td>\n","      <td>Houston</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>2242193.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-370ececf-23eb-48b9-8aae-02cebb9fbce7')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-370ececf-23eb-48b9-8aae-02cebb9fbce7 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-370ececf-23eb-48b9-8aae-02cebb9fbce7');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":107}],"source":["result_concat = pd.concat([city_loc, city_pop])\n","result_concat"]},{"cell_type":"markdown","metadata":{"id":"mNaralK8bdkx"},"source":["Note that this operation aligned the data horizontally (by columns) but not vertically (by rows). In this example, we end up with multiple rows having the same index (eg. 3). Pandas handles this rather gracefully:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VDFrmNMGbdkx","outputId":"eaec25d9-8cba-4f9d-bfda-6517986ccb52","colab":{"base_uri":"https://localhost:8080/","height":112},"executionInfo":{"status":"ok","timestamp":1677383173016,"user_tz":-480,"elapsed":271,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["        state           city        lat        lng  population\n","3          OH      Cleveland  41.473508 -81.739791         NaN\n","3  California  San Francisco        NaN        NaN    808976.0"],"text/html":["\n","  <div id=\"df-403a5690-7d3e-4abe-9206-36f6bd74e89d\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>state</th>\n","      <th>city</th>\n","      <th>lat</th>\n","      <th>lng</th>\n","      <th>population</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>3</th>\n","      <td>OH</td>\n","      <td>Cleveland</td>\n","      <td>41.473508</td>\n","      <td>-81.739791</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>California</td>\n","      <td>San Francisco</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>808976.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-403a5690-7d3e-4abe-9206-36f6bd74e89d')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-403a5690-7d3e-4abe-9206-36f6bd74e89d button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-403a5690-7d3e-4abe-9206-36f6bd74e89d');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":108}],"source":["result_concat.loc[3]"]},{"cell_type":"markdown","metadata":{"id":"oSp4G88tbdkx"},"source":["Or you can tell Pandas to just ignore the index:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"BrVK8LtDbdkx","outputId":"ac861bbd-ba09-4166-ab57-5da983a1cdce","colab":{"base_uri":"https://localhost:8080/","height":332},"executionInfo":{"status":"ok","timestamp":1677383179318,"user_tz":-480,"elapsed":297,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["        state            city        lat         lng  population\n","0          CA   San Francisco  37.781334 -122.416728         NaN\n","1          NY        New York  40.705649  -74.008344         NaN\n","2          FL           Miami  25.791100  -80.320733         NaN\n","3          OH       Cleveland  41.473508  -81.739791         NaN\n","4          UT  Salt Lake City  40.755851 -111.896657         NaN\n","5  California   San Francisco        NaN         NaN    808976.0\n","6    New-York        New York        NaN         NaN   8363710.0\n","7     Florida           Miami        NaN         NaN    413201.0\n","8       Texas         Houston        NaN         NaN   2242193.0"],"text/html":["\n","  <div id=\"df-05e0d9eb-9ba4-446b-a9f5-d91ea7c00b8c\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>state</th>\n","      <th>city</th>\n","      <th>lat</th>\n","      <th>lng</th>\n","      <th>population</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>CA</td>\n","      <td>San Francisco</td>\n","      <td>37.781334</td>\n","      <td>-122.416728</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>NY</td>\n","      <td>New York</td>\n","      <td>40.705649</td>\n","      <td>-74.008344</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>FL</td>\n","      <td>Miami</td>\n","      <td>25.791100</td>\n","      <td>-80.320733</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>OH</td>\n","      <td>Cleveland</td>\n","      <td>41.473508</td>\n","      <td>-81.739791</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>UT</td>\n","      <td>Salt Lake City</td>\n","      <td>40.755851</td>\n","      <td>-111.896657</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>California</td>\n","      <td>San Francisco</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>808976.0</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>New-York</td>\n","      <td>New York</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>8363710.0</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>Florida</td>\n","      <td>Miami</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>413201.0</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>Texas</td>\n","      <td>Houston</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>2242193.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-05e0d9eb-9ba4-446b-a9f5-d91ea7c00b8c')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-05e0d9eb-9ba4-446b-a9f5-d91ea7c00b8c button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-05e0d9eb-9ba4-446b-a9f5-d91ea7c00b8c');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":109}],"source":["pd.concat([city_loc, city_pop], ignore_index=True)"]},{"cell_type":"markdown","metadata":{"id":"rHrwIwkobdkx"},"source":["Notice that when a column does not exist in a `DataFrame`, it acts as if it was filled with `NaN` values. If we set `join=\"inner\"`, then only columns that exist in *both* `DataFrame`s are returned:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"FlkwidKubdky","outputId":"50224329-b873-490d-dd92-0e86e9021e35","colab":{"base_uri":"https://localhost:8080/","height":332},"executionInfo":{"status":"ok","timestamp":1677383204915,"user_tz":-480,"elapsed":618,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["        state            city\n","0          CA   San Francisco\n","1          NY        New York\n","2          FL           Miami\n","3          OH       Cleveland\n","4          UT  Salt Lake City\n","3  California   San Francisco\n","4    New-York        New York\n","5     Florida           Miami\n","6       Texas         Houston"],"text/html":["\n","  <div id=\"df-cdf6cc26-c99b-4343-8491-4369c5d22c1c\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>state</th>\n","      <th>city</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>CA</td>\n","      <td>San Francisco</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>NY</td>\n","      <td>New York</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>FL</td>\n","      <td>Miami</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>OH</td>\n","      <td>Cleveland</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>UT</td>\n","      <td>Salt Lake City</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>California</td>\n","      <td>San Francisco</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>New-York</td>\n","      <td>New York</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>Florida</td>\n","      <td>Miami</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>Texas</td>\n","      <td>Houston</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cdf6cc26-c99b-4343-8491-4369c5d22c1c')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-cdf6cc26-c99b-4343-8491-4369c5d22c1c button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-cdf6cc26-c99b-4343-8491-4369c5d22c1c');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":110}],"source":["pd.concat([city_loc, city_pop], join=\"inner\")"]},{"cell_type":"markdown","source":["#### Categories"],"metadata":{"id":"cYaNu6Jn1LEN"}},{"cell_type":"markdown","metadata":{"id":"fMXy67Thbdkz"},"source":["It is quite frequent to have values that represent categories, for example `1` for female and `2` for male, or `\"A\"` for Good, `\"B\"` for Average, `\"C\"` for Bad. These categorical values can be hard to read and cumbersome to handle, but fortunately pandas makes it easy. To illustrate this, let's take the `city_pop` `DataFrame` we created earlier, and add a column that represents a category:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ca9MbHkJbdkz","outputId":"fd370439-3bf6-4ea2-84c9-e24a17a0de97","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677383244068,"user_tz":-480,"elapsed":293,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   population           city       state  eco_code\n","3      808976  San Francisco  California        17\n","4     8363710       New York    New-York        17\n","5      413201          Miami     Florida        34\n","6     2242193        Houston       Texas        20"],"text/html":["\n","  <div id=\"df-04f602fe-bfaa-4d77-95ab-de19fd7fa3a3\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>population</th>\n","      <th>city</th>\n","      <th>state</th>\n","      <th>eco_code</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>3</th>\n","      <td>808976</td>\n","      <td>San Francisco</td>\n","      <td>California</td>\n","      <td>17</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>8363710</td>\n","      <td>New York</td>\n","      <td>New-York</td>\n","      <td>17</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>413201</td>\n","      <td>Miami</td>\n","      <td>Florida</td>\n","      <td>34</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>2242193</td>\n","      <td>Houston</td>\n","      <td>Texas</td>\n","      <td>20</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-04f602fe-bfaa-4d77-95ab-de19fd7fa3a3')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-04f602fe-bfaa-4d77-95ab-de19fd7fa3a3 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-04f602fe-bfaa-4d77-95ab-de19fd7fa3a3');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":111}],"source":["city_eco = city_pop.copy()\n","city_eco[\"eco_code\"] = [17, 17, 34, 20]\n","city_eco"]},{"cell_type":"markdown","metadata":{"id":"Z8kMBnYCbdkz"},"source":["Right now the `eco_code` column is full of apparently meaningless codes. Let's fix that. First, we will create a new categorical column based on the `eco_code`s:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"3Vvut-u9bdkz","outputId":"5e4367cf-6c02-4737-9fea-cad3d1fcbb64","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1677383256322,"user_tz":-480,"elapsed":306,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["Int64Index([17, 20, 34], dtype='int64')"]},"metadata":{},"execution_count":112}],"source":["city_eco[\"economy\"] = city_eco[\"eco_code\"].astype('category')\n","city_eco[\"economy\"].cat.categories"]},{"cell_type":"markdown","metadata":{"id":"8NwGaKhebdkz"},"source":["Now we can give each category a meaningful name:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NEJS59vnbdkz","outputId":"56a34d42-d658-4af2-9737-0295479159eb","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677383263597,"user_tz":-480,"elapsed":287,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   population           city       state  eco_code  economy\n","3      808976  San Francisco  California        17  Finance\n","4     8363710       New York    New-York        17  Finance\n","5      413201          Miami     Florida        34  Tourism\n","6     2242193        Houston       Texas        20   Energy"],"text/html":["\n","  <div id=\"df-61c22a35-4817-462b-afff-5fa29ab1cd49\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>population</th>\n","      <th>city</th>\n","      <th>state</th>\n","      <th>eco_code</th>\n","      <th>economy</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>3</th>\n","      <td>808976</td>\n","      <td>San Francisco</td>\n","      <td>California</td>\n","      <td>17</td>\n","      <td>Finance</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>8363710</td>\n","      <td>New York</td>\n","      <td>New-York</td>\n","      <td>17</td>\n","      <td>Finance</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>413201</td>\n","      <td>Miami</td>\n","      <td>Florida</td>\n","      <td>34</td>\n","      <td>Tourism</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>2242193</td>\n","      <td>Houston</td>\n","      <td>Texas</td>\n","      <td>20</td>\n","      <td>Energy</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-61c22a35-4817-462b-afff-5fa29ab1cd49')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-61c22a35-4817-462b-afff-5fa29ab1cd49 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-61c22a35-4817-462b-afff-5fa29ab1cd49');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":113}],"source":["city_eco[\"economy\"].cat.categories = [\"Finance\", \"Energy\", \"Tourism\"]\n","city_eco"]},{"cell_type":"markdown","metadata":{"id":"b5l_n08-bdkz"},"source":["Note that categorical values are sorted according to their categorical order, *not* their alphabetical order:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"r5e0jca6bdkz","outputId":"022c94d7-1b8a-44a5-a59d-9b78f291aa46","colab":{"base_uri":"https://localhost:8080/","height":175},"executionInfo":{"status":"ok","timestamp":1677383270015,"user_tz":-480,"elapsed":304,"user":{"displayName":"phonchi chung","userId":"13517391734500420886"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   population           city       state  eco_code  economy\n","5      413201          Miami     Florida        34  Tourism\n","6     2242193        Houston       Texas        20   Energy\n","3      808976  San Francisco  California        17  Finance\n","4     8363710       New York    New-York        17  Finance"],"text/html":["\n","  <div id=\"df-b6eec08b-ae09-45d0-b122-c8cea0fa648b\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>population</th>\n","      <th>city</th>\n","      <th>state</th>\n","      <th>eco_code</th>\n","      <th>economy</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>5</th>\n","      <td>413201</td>\n","      <td>Miami</td>\n","      <td>Florida</td>\n","      <td>34</td>\n","      <td>Tourism</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>2242193</td>\n","      <td>Houston</td>\n","      <td>Texas</td>\n","      <td>20</td>\n","      <td>Energy</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>808976</td>\n","      <td>San Francisco</td>\n","      <td>California</td>\n","      <td>17</td>\n","      <td>Finance</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>8363710</td>\n","      <td>New York</td>\n","      <td>New-York</td>\n","      <td>17</td>\n","      <td>Finance</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b6eec08b-ae09-45d0-b122-c8cea0fa648b')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-b6eec08b-ae09-45d0-b122-c8cea0fa648b button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-b6eec08b-ae09-45d0-b122-c8cea0fa648b');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":114}],"source":["city_eco.sort_values(by=\"economy\", ascending=False)"]},{"cell_type":"markdown","source":["## What next?"],"metadata":{"id":"7dIzNQ8-ognZ"}},{"cell_type":"markdown","metadata":{"id":"-c0f9fnEbdk0"},"source":["As you probably noticed by now, pandas is quite a large library with *many* features. Although we went through the most important features, there is still a lot to discover. Probably the best way to learn more is to get your hands dirty with some real-life data. It is also a good idea to go through pandas' excellent [documentation](http://pandas.pydata.org/pandas-docs/stable/index.html), in particular the [Cookbook](http://pandas.pydata.org/pandas-docs/stable/cookbook.html).\n","\n","You can also work with BigQuery in Pandas. Check out https://pandas.pydata.org/docs/reference/api/pandas.read_gbq.html and https://pandas-gbq.readthedocs.io/en/latest/ for more details."]},{"cell_type":"markdown","source":["## References"],"metadata":{"id":"-mFVEYOxZ4BE"}},{"cell_type":"markdown","source":["1. https://www.kaggle.com/learn/\n","2. https://github.com/ageron/handson-ml3\n","3. https://github.com/wesm/pydata-book."],"metadata":{"id":"lvi_RRPeZ6mX"}}]}