Selecting_&_Replacing_Values_In_Pandas_DataFrame_Effectively.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyP9vDdajsOpwGvHUf1bJ2O5",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/youssefHosni/Efficient-Python-for-Data-Scientists/blob/main/Selecting_%26_Replacing_Values_In_Pandas_DataFrame_Effectively.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Selecting & Replacing Values In Pandas DataFrame  Effectively\n"
      ],
      "metadata": {
        "id": "6YSgf9BIR-1i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "As a data scientist, it is important to use the right tools and techniques to get the most out of the data. The Pandas library is a great tool for data manipulation, analysis, and visualization, and it is an essential part of any data scientist's toolkit. However, it can be challenging to use Pandas efficiently, and this can lead to wasted time and effort. \n",
        "\n",
        "Fortunately, there are a few best practices that can help data scientists get the most out of their Pandas experience. From using vectorized operations to taking advantage of built-in functions, these best practices will help data scientists quickly and accurately analyze and visualize data using Pandas. Understanding and applying these best practices will help data scientists increase their productivity and accuracy, allowing them to make better decisions faster.\n",
        "\n",
        "In this article, we will focus on two of the most common tasks that you will commonly do on your DataFrame, especially in the data manipulation phase of a data science project. These two tasks are selecting specific and random rows and columns efficiently and the usage of the replace() function for replacing one or multiple values using lists and dictionaries."
      ],
      "metadata": {
        "id": "v6d_VLUMSArG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Table of Contents:\n",
        "1. Why do We need Efficient Coding?\n",
        "2. Selecting Rows & Columns Efficiently using .iloc[] & .loc[]\n",
        "3. Replacing Values in a DataFrame Effectively\n",
        "4. **Summary** of best practices for selecting and replacing values"
      ],
      "metadata": {
        "id": "SZiOdwbhS2ah"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Throughout this article, we will use three datasets:\n",
        "* [Poker card game dataset](https://github.com/youssefHosni/Efficient-Python-for-Data-Scientists/blob/main/Datasets/poker_hand.csv)\n",
        "* [Popular baby names](https://github.com/youssefHosni/Efficient-Python-for-Data-Scientists/blob/main/Datasets/Popular_Baby_Names.csv)\n",
        "\n",
        "The first dataset is the [Poker card game dataset](https://github.com/youssefHosni/Efficient-Python-for-Data-Scientists/blob/main/Datasets/poker_hand.csv) which is shown below."
      ],
      "metadata": {
        "id": "PS1mZXziTAnz"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "Gv6saGNHR9A8",
        "outputId": "915faed1-1c89-4abc-9ede-37ae7b53fd6b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   S1  R1  S2  R2  S3  R3  S4  R4  S5  R5  Class\n",
              "0   1  10   1  11   1  13   1  12   1   1      9\n",
              "1   2  11   2  13   2  10   2  12   2   1      9\n",
              "2   3  12   3  11   3  13   3  10   3   1      9\n",
              "3   4  10   4  11   4   1   4  13   4  12      9\n",
              "4   4   1   4  13   4  12   4  11   4  10      9"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5c6a7629-45ca-440e-843e-43e57b192674\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>S1</th>\n",
              "      <th>R1</th>\n",
              "      <th>S2</th>\n",
              "      <th>R2</th>\n",
              "      <th>S3</th>\n",
              "      <th>R3</th>\n",
              "      <th>S4</th>\n",
              "      <th>R4</th>\n",
              "      <th>S5</th>\n",
              "      <th>R5</th>\n",
              "      <th>Class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>10</td>\n",
              "      <td>1</td>\n",
              "      <td>11</td>\n",
              "      <td>1</td>\n",
              "      <td>13</td>\n",
              "      <td>1</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>11</td>\n",
              "      <td>2</td>\n",
              "      <td>13</td>\n",
              "      <td>2</td>\n",
              "      <td>10</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>12</td>\n",
              "      <td>3</td>\n",
              "      <td>11</td>\n",
              "      <td>3</td>\n",
              "      <td>13</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>10</td>\n",
              "      <td>4</td>\n",
              "      <td>11</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>13</td>\n",
              "      <td>4</td>\n",
              "      <td>12</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>13</td>\n",
              "      <td>4</td>\n",
              "      <td>12</td>\n",
              "      <td>4</td>\n",
              "      <td>11</td>\n",
              "      <td>4</td>\n",
              "      <td>10</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5c6a7629-45ca-440e-843e-43e57b192674')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5c6a7629-45ca-440e-843e-43e57b192674 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5c6a7629-45ca-440e-843e-43e57b192674');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import time \n",
        "poker_data = pd.read_csv('poker_hand.csv')\n",
        "poker_data.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In each poker round, each player has five cards in hand, each one characterized by its symbol, which can be either hearts, diamonds, clubs, or spades, and its rank, which ranges from 1 to 13. The dataset consists of every possible combination of five cards one person can possess.\n",
        "\n",
        "* Sn: symbol of the n-th card where: 1 (Hearts), 2 (Diamonds), 3 (Clubs), 4 (Spades)\n",
        "* Rn: rank of the n-th card where: 1 (Ace), 2–10, 11 (Jack), 12 (Queen), 13 (King)\n",
        "\n",
        "The second dataset we will work with is the [Popular baby names](https://github.com/youssefHosni/Efficient-Python-for-Data-Scientists/blob/main/Datasets/Popular_Baby_Names.csv) dataset which includes the most popular names that were given to newborns between 2011 and 2016. The dataset is loaded and shown below:"
      ],
      "metadata": {
        "id": "AQlb7Cncg3rl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "names = pd.read_csv('Popular_Baby_Names.csv')\n",
        "names.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "ziJbyTBhhGyW",
        "outputId": "7a3dafed-f44f-4079-9758-ea694a3c7359"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Year of Birth  Gender                   Ethnicity Child's First Name  \\\n",
              "0           2011  FEMALE  ASIAN AND PACIFIC ISLANDER             SOPHIA   \n",
              "1           2011  FEMALE  ASIAN AND PACIFIC ISLANDER              CHLOE   \n",
              "2           2011  FEMALE  ASIAN AND PACIFIC ISLANDER              EMILY   \n",
              "3           2011  FEMALE  ASIAN AND PACIFIC ISLANDER             OLIVIA   \n",
              "4           2011  FEMALE  ASIAN AND PACIFIC ISLANDER               EMMA   \n",
              "\n",
              "   Count  Rank  \n",
              "0    119     1  \n",
              "1    106     2  \n",
              "2     93     3  \n",
              "3     89     4  \n",
              "4     75     5  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3c315758-140b-45b8-a8b7-2de0ab704dea\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Year of Birth</th>\n",
              "      <th>Gender</th>\n",
              "      <th>Ethnicity</th>\n",
              "      <th>Child's First Name</th>\n",
              "      <th>Count</th>\n",
              "      <th>Rank</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>SOPHIA</td>\n",
              "      <td>119</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>CHLOE</td>\n",
              "      <td>106</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>EMILY</td>\n",
              "      <td>93</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>OLIVIA</td>\n",
              "      <td>89</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>EMMA</td>\n",
              "      <td>75</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3c315758-140b-45b8-a8b7-2de0ab704dea')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3c315758-140b-45b8-a8b7-2de0ab704dea button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3c315758-140b-45b8-a8b7-2de0ab704dea');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The dataset includes, among other information, the most popular names in the US by year, gender, and ethnicity. For example, the name Chloe was ranked second in popularity among all female newborns of Asian and Pacific Islander ethnicity in 2011."
      ],
      "metadata": {
        "id": "VYFqkoTOhJQE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. Why do We need Efficient Coding?\n",
        "Efficient code is code that executes faster and with lower computational meomery. In this article, we will use the time() function to measure the computational time. This function measures the current time so we will assign it to a variable before the code execution and after and then calculate the difference to know the computational time of the code. A simple example is shown in the code below:"
      ],
      "metadata": {
        "id": "Rxy_rQx-hM9Q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import time\n",
        "# record time before execution\n",
        "start_time = time.time()\n",
        "# execute operation\n",
        "result = 5 + 2\n",
        "# record time after execution\n",
        "end_time = time.time()\n",
        "print(\"Result calculated in {} sec\".format(end_time - start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YUriB2qZhRT3",
        "outputId": "7ea99993-0650-451c-ce45-6c3bc7c2b692"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Result calculated in 6.890296936035156e-05 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see some examples of how applying efficient code methods will improve the code runtime and decrease the computational time complexity: We will calculate the square of each number from zero, up to a million. At first, we will use a list comprehension to execute this operation, and then repeat the same procedure using a for-loop.\n",
        "\n",
        "First using list comprehension:"
      ],
      "metadata": {
        "id": "aYfDwt-PhTBb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#using List comprehension \n",
        "\n",
        "list_comp_start_time = time.time()\n",
        "result = [i*i for i in range(0,1000000)]\n",
        "list_comp_end_time = time.time()\n",
        "print(\"Time using the list_comprehension: {} sec\".format(list_comp_end_time -\n",
        "list_comp_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "02WsRp2EhUc1",
        "outputId": "f9de2c56-7550-4992-e350-ea460f1aa7c7"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using the list_comprehension: 0.3217921257019043 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we will use for loop to execute the same operation:"
      ],
      "metadata": {
        "id": "1s9XLwNehXir"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Using For loop\n",
        "\n",
        "for_loop_start_time= time.time()\n",
        "result=[]\n",
        "for i in range(0,1000000):\n",
        "  result.append(i*i)\n",
        "for_loop_end_time= time.time()\n",
        "print(\"Time using the for loop: {} sec\".format(for_loop_end_time - for_loop_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AQkisSjKhZGt",
        "outputId": "191cd1bc-c4aa-45eb-f509-a6ae86f81032"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using the for loop: 0.5098655223846436 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that there is a big difference between them, we can calculate the difference between them in percentage:"
      ],
      "metadata": {
        "id": "q3LRzuTwhamC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "list_comp_time = list_comp_end_time - list_comp_start_time\n",
        "for_loop_time = for_loop_end_time - for_loop_start_time\n",
        "print(\"Difference in time: {} %\".format((for_loop_time - list_comp_time)/\n",
        "list_comp_time*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eKTxv3vehcEm",
        "outputId": "c209788c-efb2-4fe9-9500-41103bf7568f"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Difference in time: 58.44561804379364 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here is another example to show the effect of writing efficient code. We would like to calculate the sum of all consecutive numbers from 1 to 1 million. There are two ways the first is to use brute force in which we will add one by one to a million."
      ],
      "metadata": {
        "id": "DI1-RRXiheZK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def sum_brute_force(N):\n",
        "  res = 0\n",
        "  for i in range(1,N+1):\n",
        "    res+=i\n",
        "  return res\n",
        "\n",
        "# Using brute force\n",
        "bf_start_time = time.time()\n",
        "bf_result = sum_brute_force(1000000)\n",
        "bf_end_time = time.time()\n",
        "\n",
        "print(\"Time using brute force: {} sec\".format(bf_end_time - bf_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r9OnGNAXhf2M",
        "outputId": "77aadb1b-903c-41d9-c246-7e392ed575a6"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using brute force: 0.20503568649291992 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Another more efficient method is to use a formula to calculate it. When we want to calculate the sum of all the integer numbers from 1 up to a number, let's say N, we can multiply N by N+1, and then divide by 2, and this will give us the result we want. This problem was actually given to some students back in Germany in the 19th century, and a bright student called Carl-Friedrich Gauss devised this formula to solve the problem in seconds."
      ],
      "metadata": {
        "id": "0KP0uVAdhhe7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def sum_formula(N):\n",
        "  return N*(N+1)/2\n",
        "  \n",
        "# Using the formula\n",
        "formula_start_time = time.time()\n",
        "formula_result = sum_formula(1000000)\n",
        "formula_end_time = time.time()\n",
        "\n",
        "print(\"Time using the formula: {} sec\".format(formula_end_time - formula_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P91qjhrdhiw9",
        "outputId": "400f1e7b-067e-41c5-82c6-9e283a7b7cdf"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using the formula: 4.172325134277344e-05 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "After running both methods, we achieve a massive improvement with a magnitude of over 160,000%, which clearly demonstrates why we need efficient and optimized code, even for simple tasks."
      ],
      "metadata": {
        "id": "kKPZRGJ-hkjC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Selecting Rows & Columns Efficiently using .iloc[] & .loc[]\n",
        "\n",
        "In this section, we will introduce how to locate and select rows efficiently from dataframes using **.iloc[]** & **.loc[]** pandas functions. We will use iloc[] for the index number locator and loc[] for the index name locator.\n",
        "In the example below we will select the first 500 rows of the poker dataset. Firstly by using the **.loc[]** function, and then by using the **.iloc[]** function."
      ],
      "metadata": {
        "id": "hlo9Cp3-hnOi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Specify the range of rows to select\n",
        "\n",
        "rows = range(0, 500)\n",
        "# Time selecting rows using .loc[]\n",
        "loc_start_time = time.time()\n",
        "poker_data.loc[rows]\n",
        "loc_end_time = time.time()\n",
        "print(\"Time using .loc[] : {} sec\".format(loc_end_time - loc_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SNArT10ThmV0",
        "outputId": "5f65f85a-992d-454c-b7b3-b17028e89745"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .loc[] : 0.005123138427734375 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Specify the range of rows to select\n",
        "rows = range(0, 500)\n",
        "# Time selecting rows using .iloc[]\n",
        "iloc_start_time = time.time()\n",
        "poker_data.iloc[rows]\n",
        "iloc_end_time = time.time()\n",
        "print(\"Time using .iloc[]: {} sec\".format(iloc_end_time - iloc_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lSTmiTcIh1es",
        "outputId": "d96a77b4-e6a3-4194-c86d-b974e0ebdadd"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .iloc[]: 0.0004892349243164062 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loc_comp_time = loc_end_time - loc_start_time\n",
        "iloc_comp_time = iloc_end_time - iloc_start_time\n",
        "print(\"Difference in time: {} %\".format((loc_comp_time - iloc_comp_time)/\n",
        "iloc_comp_time*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pBfen3ZCh31n",
        "outputId": "b8f92f91-d8ee-453e-bfbe-05d262a7e4ba"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Difference in time: 947.1734892787524 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "While these two methods have the same syntax, **.iloc[]** performs almost 70% faster than **.loc[]**. The **.iloc[]** function takes advantage of the order of the indices, which are already sorted, and is therefore faster.\n",
        "We can also use them to select columns not only rows. In the next example, we will select the first three columns using both methods."
      ],
      "metadata": {
        "id": "KldbY75Oh5pL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "iloc_start_time = time.time()\n",
        "poker_data.iloc[:,:3]\n",
        "iloc_end_time = time.time()\n",
        "print(\"Time using .iloc[]: {} sec\".format(iloc_end_time - iloc_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fSnA03c6iBx0",
        "outputId": "9bdaf26c-3727-4381-f7a5-22da424a522e"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .iloc[]: 0.0002753734588623047 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "names_start_time = time.time()\n",
        "poker_data[['S1', 'R1', 'S2']]\n",
        "names_end_time = time.time()\n",
        "print(\"Time using selection by name: {} sec\".format(names_end_time - names_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NiHPbIDNiEL_",
        "outputId": "f9374a09-3015-461d-faf6-8d7af8d64954"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using selection by name: 0.0016252994537353516 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loc_comp_time = names_end_time - names_start_time\n",
        "iloc_comp_time = iloc_end_time - iloc_start_time\n",
        "print(\"Difference in time: {} %\".format((loc_comp_time - iloc_comp_time)/\n",
        "loc_comp_time*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TMgqQR1hiFc9",
        "outputId": "6c477140-7141-44d1-b711-ce2350988af1"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Difference in time: 83.05706322429222 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see also that using the column indexing using **.iloc[]** is still 80% faster. So it is better to use **.iloc[]** as it is faster unless it is easier to use the loc[] to select certain columns by name."
      ],
      "metadata": {
        "id": "iMsWFbFBiIxu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3. Replacing Values in a DataFrame Effectively\n",
        "Replacing values in a DataFrame is a very important task, especially in the data cleaning phase. Since you will have to keep the whole values that represent the same object the same.\n",
        "\n",
        "Let's take a look at the popular baby names dataset we loaded before:"
      ],
      "metadata": {
        "id": "rLe8rutIiQNT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "names = pd.read_csv('Popular_Baby_Names.csv')\n",
        "names.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "4eX1lFChiPNU",
        "outputId": "68df50ee-b644-49f9-f6e7-fcdded755419"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Year of Birth  Gender                   Ethnicity Child's First Name  \\\n",
              "0           2011  FEMALE  ASIAN AND PACIFIC ISLANDER             SOPHIA   \n",
              "1           2011  FEMALE  ASIAN AND PACIFIC ISLANDER              CHLOE   \n",
              "2           2011  FEMALE  ASIAN AND PACIFIC ISLANDER              EMILY   \n",
              "3           2011  FEMALE  ASIAN AND PACIFIC ISLANDER             OLIVIA   \n",
              "4           2011  FEMALE  ASIAN AND PACIFIC ISLANDER               EMMA   \n",
              "\n",
              "   Count  Rank  \n",
              "0    119     1  \n",
              "1    106     2  \n",
              "2     93     3  \n",
              "3     89     4  \n",
              "4     75     5  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-76f63b74-d583-471f-b34a-9ef7682abb44\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Year of Birth</th>\n",
              "      <th>Gender</th>\n",
              "      <th>Ethnicity</th>\n",
              "      <th>Child's First Name</th>\n",
              "      <th>Count</th>\n",
              "      <th>Rank</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>SOPHIA</td>\n",
              "      <td>119</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>CHLOE</td>\n",
              "      <td>106</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>EMILY</td>\n",
              "      <td>93</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>OLIVIA</td>\n",
              "      <td>89</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2011</td>\n",
              "      <td>FEMALE</td>\n",
              "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
              "      <td>EMMA</td>\n",
              "      <td>75</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-76f63b74-d583-471f-b34a-9ef7682abb44')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-76f63b74-d583-471f-b34a-9ef7682abb44 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-76f63b74-d583-471f-b34a-9ef7682abb44');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's have a closer look at the Gender feature and see the unique values they have:"
      ],
      "metadata": {
        "id": "tBUAPG6WiZ1C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "names['Gender'].unique()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oe7CkJOUibCP",
        "outputId": "72f69a47-5b5b-4e74-e72c-f3056497f094"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['FEMALE', 'MALE'], dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that the female gender is represented with two values both uppercase and lowercase. This is very common in real data and an easy way to do so is to replace one of the values with the other to keep it consistent throughout the whole dataset. There are two ways to do it the first one is simply defining which values we want to replace, and then what we want to replace them with. This is shown in the code below:"
      ],
      "metadata": {
        "id": "o0k57v93ic1M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time = time.time()\n",
        "names['Gender'].loc[names.Gender=='female'] = 'FEMALE'\n",
        "end_time = time.time()\n",
        "\n",
        "pandas_time = end_time - start_time\n",
        "print(\"Replace values using .loc[]: {} sec\".format(pandas_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1bUWgVH6ie0E",
        "outputId": "a0701891-30dd-4a00-9d28-42f27852e02e"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Replace values using .loc[]: 0.002582073211669922 sec\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  self._setitem_single_block(indexer, value, name)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The second method is to use the panda's built-in function **.replace()** as shown in the code below:"
      ],
      "metadata": {
        "id": "KUboQDN-ihvS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time = time.time()\n",
        "names['Gender'].replace('female', 'FEMALE', inplace=True)\n",
        "end_time = time.time()\n",
        "replace_time = end_time - start_time\n",
        "\n",
        "print(\"Time using replace(): {} sec\".format(replace_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BcMvUfLQijtW",
        "outputId": "936b3dfb-74be-44f3-ba47-6b585096fef6"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using replace(): 0.0024094581604003906 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that there is a difference in time complexity with the built-in function 157% faster than using the **.loc()** method to find the rows and columns index of the values and replace it."
      ],
      "metadata": {
        "id": "4k9Taeb-ilhE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('The differnce: {} %'.format((pandas_time- replace_time )/replace_time*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lUFjiPI8ioKN",
        "outputId": "825ad842-2759-42d9-a462-5757e7cf7b64"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The differnce: 7.164060953888779 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also replace multiple values using lists. Our objective is to change all ethnicities classified as **WHITE NON-HISPANIC** or **WHITE NON-HISP** to **WNH**. Using the **.loc[]** function, we will locate babies of the ethnicities we are looking for, using the 'or' statement (which in Python is symbolized by the pipe). We will then assign the new value. As always, we also measure the CPU time needed for this operation."
      ],
      "metadata": {
        "id": "xTRhtBLyisK7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time = time.time()\n",
        "\n",
        "names['Ethnicity'].loc[(names[\"Ethnicity\"] == 'WHITE NON HISPANIC') |\n",
        "(names[\"Ethnicity\"] == 'WHITE NON HISP')] = 'WNH'\n",
        "\n",
        "end_time = time.time()\n",
        "pandas_time= end_time - start_time\n",
        "print(\"Results from the above operation calculated in %s seconds\" %(pandas_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hxs9HDJ2iyod",
        "outputId": "eb4b5e0e-434b-4e2d-f181-fb4f3057e9b0"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Results from the above operation calculated in 0.010097742080688477 seconds\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also do the same operation using the .replace() pandas built-in function as the following:"
      ],
      "metadata": {
        "id": "ylJECHuGi0ZH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time = time.time()\n",
        "names['Ethnicity'].replace(['WHITE NON HISPANIC','WHITE NON HISP'],\n",
        "'WNH', inplace=True)\n",
        "\n",
        "end_time = time.time()\n",
        "replace_time = end_time - start_time\n",
        "\n",
        "print(\"Time using .replace(): {} sec\".format(replace_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LNkJb1wZi1te",
        "outputId": "4b4c4599-f6a0-465c-e360-f831a4bdd86e"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .replace(): 0.003354787826538086 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that again using the **.replace()** method is much faster than using the **.loc[]** method. To have better intuition of how much faster it is let's run the code below:"
      ],
      "metadata": {
        "id": "t8xzJuc2i3Pi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('The differnce: {} %'.format((pandas_time- replace_time )/replace_time*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lyJM7JUji7Ql",
        "outputId": "e425e696-ffd4-4d86-dc8f-57bb522698f2"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The differnce: 200.99495416104043 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The **.replace()** method is 87% faster than using the .loc[] method. If your data is huge and need a lot of cleaning this tip will decrease the computational time of your data cleaning and makes your pandas code much faster and hence more efficient.\n",
        "\n",
        "Finally, we can also use dictionaries to replace both single and multiple values in your DataFrame. This will be very helpful if you would like to multiple replacing functions in one command.\n",
        "We're going to use dictionaries to replace every male's gender with BOY and every female's gender with GIRL."
      ],
      "metadata": {
        "id": "eHgQl0RDi9Mr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "names = pd.read_csv('Popular_Baby_Names.csv')\n",
        "\n",
        "start_time = time.time()\n",
        "names['Gender'].replace({'MALE':'BOY', 'FEMALE':'GIRL', 'female': 'girl'}, inplace=True)\n",
        "end_time = time.time()\n",
        "dict_time = end_time - start_time\n",
        "print(\"Time using .replace() with dictionary: {} sec\".format(dict_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Wio019_tjF5h",
        "outputId": "fd98f006-70f1-49e7-b2a0-8c93dfac7131"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .replace() with dictionary: 0.0034475326538085938 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "names = pd.read_csv('Popular_Baby_Names.csv')\n",
        "\n",
        "start_time = time.time()\n",
        "\n",
        "names['Gender'].replace('MALE', 'BOY', inplace=True)\n",
        "names['Gender'].replace('FEMALE', 'GIRL', inplace=True)\n",
        "names['Gender'].replace('female', 'girl', inplace=True)\n",
        "\n",
        "end_time = time.time()\n",
        "\n",
        "list_time = end_time - start_time\n",
        "print(\"Time using multiple .replace(): {} sec\".format(list_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VgKyc86wjI9i",
        "outputId": "69669fe1-fa84-40ea-87c8-0d91f685d1bb"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using multiple .replace(): 0.002288818359375 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('The differnce: {} %'.format((list_time- dict_time )/dict_time*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c67Rj8HGjMfw",
        "outputId": "bb3f07c8-df72-4f3e-f680-9a9ed00d99dd"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The differnce: -33.60995850622407 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We could do the same thing with lists, but it's more verbose. If we compare both methods, we can see that dictionaries run approximately 22% faster. In general, working with dictionaries in Python is very efficient compared to lists: looking through a list requires a pass in every element of the list while looking at a dictionary directs instantly to the key that matches the entry. The comparison is a little unfair though since both structures serve different purposes.\n",
        "\n",
        "Using dictionaries allows you to replace the same values on several different columns. In all the previous examples, we specified the column from which the values to replace came. We're now going to replace several values from the same column with one common value. We want to classify all ethnicities into three big categories: Black, Asian and White. The syntax again is very simple. We use nested dictionaries here: the outer key is the column in which we want to replace values. The value of this outer key is another dictionary, where the keys are the ethnicities to replace, and the values for the new ethnicity (Black, Asian or White)."
      ],
      "metadata": {
        "id": "aExPSYnqjQRL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time = time.time()\n",
        "names.replace({'Ethnicity': {'ASIAN AND PACI': 'ASIAN', 'ASIAN AND PACIFIC ISLANDER': 'ASIAN',\n",
        "'BLACK NON HISPANIC': 'BLACK', 'BLACK NON HISP': 'BLACK',\n",
        "'WHITE NON HISPANIC': 'WHITE', 'WHITE NON HISP': 'WHITE'}})\n",
        "print(\"Time using .replace() with dictionary: {} sec\".format (time.time() - start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LWED50r5jR6t",
        "outputId": "56072037-44ee-49a4-abff-d294f4713e0e"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .replace() with dictionary: 0.011412620544433594 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4. Summary of best practices for selecting and replacing values\n",
        "* Selecting rows and columns is faster using the .iloc[] function. So it is better to use unless it is easier or more convenient to use .loc[] and the speed is not a priority or you are just doing it once.\n",
        "* Using the built-in replace() function is much faster than just using conventional methods.\n",
        "* Replacing multiple values using python dictionaries is faster than using lists."
      ],
      "metadata": {
        "id": "qal-IBIbjZcU"
      }
    }
  ]
}