Stop_Looping_Through_Pandas_DataFrames_&_Do_This Instead.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPG/0UgH4QPGUoZduEMRpWs",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/youssefHosni/Efficient-Python-for-Data-Scientists/blob/main/Stop_Looping_Through_Pandas_DataFrames_%26_Do_This%C2%A0Instead.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Stop Looping Through Pandas DataFrames & Do This Instead\n"
      ],
      "metadata": {
        "id": "Tki3VuqnrCTo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Working with Pandas DataFrames can be a tedious and time-consuming task, especially when it comes to looping through them. If you're like most Python developers, you've probably spent a significant amount of time trying to figure out the most efficient way to loop through DataFrames. But what if there was an easier way? In this article, we'll show you better ways to work with Pandas DataFrames that don't require looping. We'll discuss the advantages of these approaches, as well as provide some practical examples to help you get started. So if you're ready to stop looping through Pandas DataFrames and do something better, read on!"
      ],
      "metadata": {
        "id": "fZn7EvAMrel6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Table of Contents:\n",
        "1. Why Do We Need Efficient Coding?\n",
        "2. Looping effectively Using the .iterrows()\n",
        "3. Looping Effectively Using .apply()\n",
        "4. Looping Effectively Using vectorization\n",
        "5. Summary of Best Practices"
      ],
      "metadata": {
        "id": "eAAS5oG7ri3e"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Throughout this article, we will use the Poker card game dataset. First let's load and explore the data:"
      ],
      "metadata": {
        "id": "jqQzluwOrqs1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import time\n",
        "poker_data = pd.read_csv('poker_hand.csv')\n",
        "poker_data.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "JkSjVUtIrruF",
        "outputId": "a839ba09-6b8d-4039-b030-db72c1d7ab4c"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   S1  R1  S2  R2  S3  R3  S4  R4  S5  R5  Class\n",
              "0   1  10   1  11   1  13   1  12   1   1      9\n",
              "1   2  11   2  13   2  10   2  12   2   1      9\n",
              "2   3  12   3  11   3  13   3  10   3   1      9\n",
              "3   4  10   4  11   4   1   4  13   4  12      9\n",
              "4   4   1   4  13   4  12   4  11   4  10      9"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5b9e2534-2ca2-48fc-82e7-53629aecbb13\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>S1</th>\n",
              "      <th>R1</th>\n",
              "      <th>S2</th>\n",
              "      <th>R2</th>\n",
              "      <th>S3</th>\n",
              "      <th>R3</th>\n",
              "      <th>S4</th>\n",
              "      <th>R4</th>\n",
              "      <th>S5</th>\n",
              "      <th>R5</th>\n",
              "      <th>Class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>10</td>\n",
              "      <td>1</td>\n",
              "      <td>11</td>\n",
              "      <td>1</td>\n",
              "      <td>13</td>\n",
              "      <td>1</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>11</td>\n",
              "      <td>2</td>\n",
              "      <td>13</td>\n",
              "      <td>2</td>\n",
              "      <td>10</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>12</td>\n",
              "      <td>3</td>\n",
              "      <td>11</td>\n",
              "      <td>3</td>\n",
              "      <td>13</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>10</td>\n",
              "      <td>4</td>\n",
              "      <td>11</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>13</td>\n",
              "      <td>4</td>\n",
              "      <td>12</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>13</td>\n",
              "      <td>4</td>\n",
              "      <td>12</td>\n",
              "      <td>4</td>\n",
              "      <td>11</td>\n",
              "      <td>4</td>\n",
              "      <td>10</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5b9e2534-2ca2-48fc-82e7-53629aecbb13')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5b9e2534-2ca2-48fc-82e7-53629aecbb13 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5b9e2534-2ca2-48fc-82e7-53629aecbb13');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In each poker round, each player has five cards in hand, each one characterized by its symbol, which can be either hearts, diamonds, clubs, or spades, and its rank, which ranges from 1 to 13. The dataset consists of every possible combination of five cards one person can possess.\n",
        "* Sn: symbol of the n-th card where: 1 (Hearts), 2 (Diamonds), 3 (Clubs), 4 (Spades)\n",
        "* Rn: rank of the n-th card where: 1 (Ace), 2–10, 11 (Jack), 12 (Queen), 13 (King)"
      ],
      "metadata": {
        "id": "vFyaahTCrz4n"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. Why do We need Efficient Coding?\n"
      ],
      "metadata": {
        "id": "YLPUOB6xyxdv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Efficient code is code that executes faster and with lower computational meomery. In this article, we will use the time() function to measure the computational time. \n",
        "\n",
        "This function measures the current time so we will assign it to a variable before the code execution and after and then calculate the difference to know the computational time of the code. A simple example is shown in the code below:\n",
        "\n"
      ],
      "metadata": {
        "id": "PBLJceGKy1FR"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0ronfYdopjMo",
        "outputId": "7d2f0a38-b742-40ab-826f-6e8c9276cd41"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Result calculated in 0.0001010894775390625 sec\n"
          ]
        }
      ],
      "source": [
        "# record time before execution\n",
        "start_time = time.time()\n",
        "# execute operation\n",
        "result = 5 + 2\n",
        "# record time after execution\n",
        "end_time = time.time()\n",
        "print(\"Result calculated in {} sec\".format(end_time - start_time))"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see some examples of how applying efficient code methods will improve the code runtime and decrease the computational time complexity: We will calculate the square of each number from zero, up to a million. At first, we will use a list comprehension to execute this operation, and then repeat the same procedure using a for-loop.\n",
        "\n",
        "First using list comprehension:\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "PMjqHZUYy6Im"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#using List comprehension \n",
        "list_comp_start_time = time.time()\n",
        "result = [i*i for i in range(0,1000000)]\n",
        "list_comp_end_time = time.time()\n",
        "print(\"Time using the list_comprehension: {} sec\".format(list_comp_end_time -\n",
        "list_comp_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OeXxZOMRy4Y-",
        "outputId": "4f25b0ce-dff4-440c-f2d4-f7d703cc7229"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using the list_comprehension: 0.12260246276855469 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we will use for loop to execute the same operation:\n",
        "\n"
      ],
      "metadata": {
        "id": "pqoqyCQbzBTt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Using For loop\n",
        "for_loop_start_time= time.time()\n",
        "result=[]\n",
        "for i in range(0,1000000):\n",
        "  result.append(i*i)\n",
        "for_loop_end_time= time.time()\n",
        "print(\"Time using the for loop: {} sec\".format(for_loop_end_time - for_loop_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pVJcr30ZzH9z",
        "outputId": "9a409a1b-3e26-454c-af52-e50dff54c165"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using the for loop: 0.37175941467285156 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that there is a big difference between them, we can calculate the difference between them in percentage:\n",
        "\n"
      ],
      "metadata": {
        "id": "gGKbSvTiznVq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "list_comp_time = list_comp_end_time - list_comp_start_time\n",
        "for_loop_time = for_loop_end_time - for_loop_start_time\n",
        "print(\"Difference in time: {} %\".format((for_loop_time - list_comp_time)/\n",
        "list_comp_time*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ka7TiiU7zO4j",
        "outputId": "e36554c8-0761-4261-d906-5debaa21be6b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Difference in time: 203.22344778232394 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here is another example to show the effect of writing efficient code. We would like to calculate the sum of all consecutive numbers from 1 to 1 million. There are two ways the first is to use brute force in which we will add one by one to a million.\n",
        "\n"
      ],
      "metadata": {
        "id": "8yWk4HrKzrjF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def sum_formula(N):\n",
        "  return N*(N+1)/2\n",
        "  \n",
        "# Using the formula\n",
        "formula_start_time = time.time()\n",
        "formula_result = sum_formula(1000000)\n",
        "formula_end_time = time.time()\n",
        "\n",
        "print(\"Time using the formula: {} sec\".format(formula_end_time - formula_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zglHUf7t0OPK",
        "outputId": "fc6007e5-5a75-4791-b9f6-0b9c6bc179b4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using the formula: 5.8650970458984375e-05 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Another more efficient method is to use a formula to calculate it. When we want to calculate the sum of all the integer numbers from 1 up to a number, let’s say N, we can multiply N by N+1, and then divide by 2, and this will give us the result we want. This problem was actually given to some students back in Germany in the 19th century, and a bright student called Carl-Friedrich Gauss devised this formula to solve the problem in seconds."
      ],
      "metadata": {
        "id": "Iek6ZUDuzvJe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def sum_brute_force(N):\n",
        "  res = 0\n",
        "  for i in range(1,N+1):\n",
        "    res+=i\n",
        "  return res\n",
        "\n",
        "# Using brute force\n",
        "bf_start_time = time.time()\n",
        "bf_result = sum_brute_force(1000000)\n",
        "bf_end_time = time.time()\n",
        "\n",
        "print(\"Time using brute force: {} sec\".format(bf_end_time - bf_start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kLHOBkEG012K",
        "outputId": "8c61e2ef-2335-49bb-eb3e-e599af4927ad"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using brute force: 0.06304192543029785 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "After running both methods, we achieve a massive improvement with a magnitude of over 160,000%, which clearly demonstrates why we need efficient and optimized code, even for simple tasks.\n",
        "\n"
      ],
      "metadata": {
        "id": "pD2YNf5Lzxg_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "One of the most inefficient methods to write python code is to have many loops in your code, especially if you have large data. Since as a data scientist, you will need to iterate through your dataframe extensively, especially in the data preparation and exploration phase, it is important to be able to do this efficiently, as it will save you much time and give space for more important work. We will walk through three methods to make your loops much faster and more efficient:\n",
        "\n",
        "* Looping using the .iterrows() function\n",
        "* Looping using the .apply() function\n",
        "* Vectorization\n"
      ],
      "metadata": {
        "id": "T2oxGNcW2SNq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Looping effectively using .iterrows()\n",
        "Before we talk about how to use the .iterrows() function to improve the looping process, let’s refresh the notion of a generator function.\n",
        "\n",
        "Generators are a simple tool to create iterators. Inside the body of a generator, instead of return statements, you will find only yield() statements. There can be just one, or several yield() statements. Here, we can see a generator, city_name_generator(), that produces four city names. We assign the generator to the variable city_names for simplicity.\n",
        "\n"
      ],
      "metadata": {
        "id": "SQjPBhCL2aDU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def city_name_generator():\n",
        "  yield('New York')\n",
        "  yield('London')\n",
        "  yield('Tokyo')\n",
        "  yield('Sao Paolo')\n",
        "\n",
        "city_names = city_name_generator()\n"
      ],
      "metadata": {
        "id": "Gs6UTJjqqLT-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "To access the elements that the generator yields we can use Python’s next() function. Each time the next() command is used, the generator will produce the next value to yield, until there are no more values to yield. We here have 4 cities. Let’s run the next command four times and see what it returns:\n",
        "\n"
      ],
      "metadata": {
        "id": "x7-XgpHX2fH7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "next(city_names)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "BdS9iCA2ESQu",
        "outputId": "8a25c76a-6fa2-47db-ddf9-977d7b57932f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'New York'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "next(city_names)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "vRT4Eaz1GgUI",
        "outputId": "14fae336-6331-48e8-fb83-317d168014eb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'London'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "next(city_names)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "hA6SCwAUGh79",
        "outputId": "0eaf8af0-f56f-4534-ffad-ce3c10590f4e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Tokyo'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "next(city_names)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "rKjG6CzhGjob",
        "outputId": "741af17b-186c-4ab0-d475-fe18e49cd5c9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Sao Paolo'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we can see that every time we run the next() function it will print a new city name.\n",
        "\n"
      ],
      "metadata": {
        "id": "5FFZfu_K2oP8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's go back to the .iterrows() function. The .iterrows() function is a property of every pandas DataFrame. When called, it produces a list with two elements. We will use this generator to iterate through each line of our poker DataFrame. The first element is the index of the row, while the second element contains a pandas Series of each feature of the row: the Symbol and the Rank for each of the five cards. It is very similar to the notion of the enumerate() function, which when applied to a list, returns each element along with its index.\n",
        "\n",
        "The most intuitive way to iterate through a Pandas DataFrame is to use the range() function, which is often called crude looping. This is shown with the code below:\n",
        "\n"
      ],
      "metadata": {
        "id": "Va18cE2T3Rvw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time = time.time()\n",
        "for index in range(poker_data.shape[0]):\n",
        "  next\n",
        "print(\"Time using range(): {} sec\".format(time.time() - start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r55m_fxePy5N",
        "outputId": "2432444e-a802-4e92-e7d1-fd0146b578c3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using range(): 0.0036385059356689453 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "One smarter way to iterate through a pandas DataFrame is to use the **.iterrows()** function, which is optimized for this task. We simply define the **‘for’** loop with two iterators, one for the number of each row and the other for all the values.\n",
        "\n",
        "Inside the loop, the **next()** command indicates that the loop moves to the next value of the iterator, without actually doing something."
      ],
      "metadata": {
        "id": "gFaF1Z-13Way"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data_generator = poker_data.iterrows()\n",
        "start_time = time.time()\n",
        "for index, values in data_generator:\n",
        "  next\n",
        "print(\"Time using .iterrows(): {} sec\".format(time.time() - start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "saqXt53lNPDQ",
        "outputId": "2ac4354c-25d1-4032-a9eb-df7d81e8a7d2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .iterrows(): 1.2583379745483398 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Comparing the two computational times we can also notice that the use of .iterrows() does not improve the speed of iterating through pandas DataFrame. It is very useful though when we need a cleaner way to use the values of each row while iterating through the dataset.\n"
      ],
      "metadata": {
        "id": "mD6O-NjL3hah"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3.Looping Effectively Using .apply()"
      ],
      "metadata": {
        "id": "0fqYrl6F3q-_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we will use the **.apply()** function to be able to perform a specific task while iterating through a pandas DataFrame. The **.apply()** function does exactly what it says; it applies another function to the whole DataFrame.\n",
        "\n",
        "The syntax of the **.apply()** function is simple: we create a mapping, using a lambda function in this case, and then declare the function we want to apply to every cell. Here, we’re applying the square root function to every cell of the DataFrame. In terms of speed, it matches the speed of just using the NumPy sqrt() function over the whole DataFrame.\n"
      ],
      "metadata": {
        "id": "x6Y4ACXY3pDX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data_sqrt = poker_data.apply(lambda x: np.sqrt(x), axis =0 )\n",
        "data_sqrt.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "GVixY9NAP7JI",
        "outputId": "447e5a58-91e1-42d1-cfa4-21582df6b580"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "         S1        R1        S2        R2        S3        R3        S4  \\\n",
              "0  1.000000  3.162278  1.000000  3.316625  1.000000  3.605551  1.000000   \n",
              "1  1.414214  3.316625  1.414214  3.605551  1.414214  3.162278  1.414214   \n",
              "2  1.732051  3.464102  1.732051  3.316625  1.732051  3.605551  1.732051   \n",
              "3  2.000000  3.162278  2.000000  3.316625  2.000000  1.000000  2.000000   \n",
              "4  2.000000  1.000000  2.000000  3.605551  2.000000  3.464102  2.000000   \n",
              "\n",
              "         R4        S5        R5  Class  \n",
              "0  3.464102  1.000000  1.000000    3.0  \n",
              "1  3.464102  1.414214  1.000000    3.0  \n",
              "2  3.162278  1.732051  1.000000    3.0  \n",
              "3  3.605551  2.000000  3.464102    3.0  \n",
              "4  3.316625  2.000000  3.162278    3.0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-68d3ecd1-faa7-407d-825a-d187213248a6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>S1</th>\n",
              "      <th>R1</th>\n",
              "      <th>S2</th>\n",
              "      <th>R2</th>\n",
              "      <th>S3</th>\n",
              "      <th>R3</th>\n",
              "      <th>S4</th>\n",
              "      <th>R4</th>\n",
              "      <th>S5</th>\n",
              "      <th>R5</th>\n",
              "      <th>Class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.162278</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.316625</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.605551</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.464102</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.414214</td>\n",
              "      <td>3.316625</td>\n",
              "      <td>1.414214</td>\n",
              "      <td>3.605551</td>\n",
              "      <td>1.414214</td>\n",
              "      <td>3.162278</td>\n",
              "      <td>1.414214</td>\n",
              "      <td>3.464102</td>\n",
              "      <td>1.414214</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.732051</td>\n",
              "      <td>3.464102</td>\n",
              "      <td>1.732051</td>\n",
              "      <td>3.316625</td>\n",
              "      <td>1.732051</td>\n",
              "      <td>3.605551</td>\n",
              "      <td>1.732051</td>\n",
              "      <td>3.162278</td>\n",
              "      <td>1.732051</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.162278</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.316625</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.605551</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.464102</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.605551</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.464102</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.316625</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.162278</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-68d3ecd1-faa7-407d-825a-d187213248a6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-68d3ecd1-faa7-407d-825a-d187213248a6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-68d3ecd1-faa7-407d-825a-d187213248a6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is a simple example since we would like to apply this function to the dataframe.\n",
        "\n",
        "But what happens when the function of interest is taking more than one cell as an input? For example, what if we want to calculate the sum of the rank of all the cards in each hand? In this case, we will use the .apply() function the same way as we did before, but we need to add ‘axis=1’ at the end of the line to specify we’re applying the function to each row.\n",
        "\n"
      ],
      "metadata": {
        "id": "dMg4d5SV4IL-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "apply_start_time = time.time()\n",
        "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].apply(lambda x: sum(x), axis=1)\n",
        "apply_end_time = time.time()\n",
        "apply_time = apply_end_time - apply_start_time\n",
        "print(\"Time using .apply(): {} sec\".format(apply_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6dzHsYQdjtfa",
        "outputId": "4e4351c3-174a-4961-83da-1627f5db1e89"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .apply(): 0.2000577449798584 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then, we will use the .iterrows() function we saw previously, and compare their efficiency.\n",
        "\n"
      ],
      "metadata": {
        "id": "6Jg-EIJV4McK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for_loop_start_time = time.time()\n",
        "for ind, value in poker_data.iterrows():\n",
        "  sum([value[1], value[3], value[5], value[7], value[9]])\n",
        "for_loop_end_time = time.time()\n",
        "\n",
        "for_loop_time = for_loop_end_time - for_loop_start_time\n",
        "print(\"Time using .iterrows(): {} sec\".format(for_loop_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sOkXe8gxrXOX",
        "outputId": "2b901612-ffb1-438d-952c-04b9ce221c62"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .iterrows(): 1.1545953750610352 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using the .apply() function is significantly faster than the .iterrows() function, with a magnitude of around 400 percent, which is a massive improvement!\n",
        "\n"
      ],
      "metadata": {
        "id": "V3FUIUts4PBG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('The differnce: {} %'.format((for_loop_time - apply_time) / apply_time * 100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JGFa3H3ysxmE",
        "outputId": "566e5293-2da4-4f63-9b0a-8e065689326f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The differnce: 477.1310554246618 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we did with rows, we can do exactly the same thing for the columns; apply one function to each column. By replacing the axis=1 with axis=0, we can apply the sum function on every column.\n",
        "\n"
      ],
      "metadata": {
        "id": "7FE-SES04VZc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "apply_start_time = time.time()\n",
        "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].apply(lambda x: sum(x), axis=0)\n",
        "apply_end_time = time.time()\n",
        "apply_time = apply_end_time - apply_start_time\n",
        "print(\"Time using .apply(): {} sec\".format(apply_time))"
      ],
      "metadata": {
        "id": "TqSssrebte0e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9aa644fc-cb77-4e73-823b-fd749617722e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .apply(): 0.021090030670166016 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "By comparing the **.apply()** function with the native panda's function for summing over rows, we can see that pandas’ native .sum() functions perform the same operation faster.\n",
        "\n"
      ],
      "metadata": {
        "id": "KybwcTHL4YNl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pandas_start_time = time.time()\n",
        "poker_data[['R1', 'R1', 'R3', 'R4', 'R5']].sum(axis=0)\n",
        "pandas_end_time = time.time()\n",
        "pandas_time = pandas_end_time - pandas_start_time\n",
        "print(\"Time using pandas: {} sec\".format(pandas_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c9N3WsQDP_kF",
        "outputId": "7a395b28-fb3f-4e58-8189-d24e9714fc62"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using pandas: 0.0039751529693603516 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('The differnce: {} %'.format((apply_time - pandas_time) / pandas_time * 100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "89czJRWaQC83",
        "outputId": "1e30f4ea-2d55-4035-9def-e2d587b4296d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The differnce: 430.54639237089907 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In conclusion, we observe that the .apply() function performs faster when we want to iterate through all the rows of a pandas DataFrame, but is slower when we perform the same operation through a column.\n",
        "\n"
      ],
      "metadata": {
        "id": "ul9vEJjD40cf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4.Looping effectively using vectorization"
      ],
      "metadata": {
        "id": "5RjLUtkmUuhy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To understand how we can reduce the amount of iteration performed by the function, recall that the fundamental units of Pandas, DataFrames, and Series, are both based on arrays. Pandas perform more efficiently when an operation is performed to a whole array than to each value separately or sequentially. This can be achieved through vectorization. Vectorization is the process of executing operations on entire arrays.\n",
        "\n",
        "In the code below we want to calculate the sum of the ranks of all the cards in each hand. In order to do that, we slice the poker dataset keeping only the columns that contain the ranks of each card. Then, we call the built-in .sum() property of the DataFrame, using the parameter axis = 1 to denote that we want the sum for each row. In the end, we print the sum of the first five rows of the data.\n",
        "\n"
      ],
      "metadata": {
        "id": "k1sJKs2d49tX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time_vectorization = time.time()\n",
        "\n",
        "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].sum(axis=1)\n",
        "end_time_vectorization = time.time()\n",
        "\n",
        "vectorization_time = end_time_vectorization  - start_time_vectorization\n",
        "print(\"Time using pandas vectorization: {} sec\".format(vectorization_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QsTG45mKQ-Mi",
        "outputId": "1acedf98-4a2a-4f9b-d04c-f4dc062baf52"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using pandas vectorization: 0.009327411651611328 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We saw previously various methods that perform functions applied to a DataFrame faster than simply iterating through all the rows of the DataFrame. Our goal is to find the most efficient method to perform this task.\n",
        "\n"
      ],
      "metadata": {
        "id": "ljv_-ySq7qRL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using .iterrows() to loop through the DataFrame:\n"
      ],
      "metadata": {
        "id": "3H338N5h7wgS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data_generator = poker_data.iterrows()\n",
        "\n",
        "start_time_iterrows = time.time()\n",
        "\n",
        "for index, value in data_generator:\n",
        "  sum([value[1], value[3], value[5], value[7]])\n",
        "\n",
        "end_time_iterrows = time.time()\n",
        "iterrows_time = end_time_iterrows - start_time_iterrows\n",
        "print(\"Time using .iterrows() {} seconds \" .format(iterrows_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kyivEyu_V1OO",
        "outputId": "816b1050-4228-42e2-a81c-164b75cab419"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using .iterrows() 1.1502439975738525 seconds \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using the .apply() mehtod\n"
      ],
      "metadata": {
        "id": "0FURW1ta71Mj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time_apply = time.time()\n",
        "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].apply(lambda x: sum(x),axis=1)\n",
        "end_time_apply = time.time()\n",
        "\n",
        "apply_time = end_time_apply - start_time_apply\n",
        "\n",
        "print(\"Time using apply() {} seconds\"  .format(apply_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M4rz9TNIWI1m",
        "outputId": "8434927d-9217-4c07-c08d-4ec2f97ba4db"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using apply() 0.3497791290283203 seconds\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Comparing the time it takes to sum the ranks of all the cards in each hand using vectorization, the .iterrows() function, and the .apply() function, we can see that the vectorization method performs much better.\n",
        "\n",
        "We can also use another vectorization method to effectively iterate through the DataFrame which is using Numpy arrays to vectorize the DataFrame.\n",
        "\n",
        "The NumPy library, which defines itself as a “fundamental package for scientific computing in Python”, performs operations under the hood in optimized, pre-compiled C code. Similar to pandas working with arrays, NumPy operates on arrays called ndarrays. A major difference between Series and ndarrays is that ndarrays leave out many operations such as indexing, data type checking, etc. As a result, operations on NumPy arrays can be significantly faster than operations on pandas Series. NumPy arrays can be used in place of the pandas Series when the additional functionality offered by the pandas Series isn’t critical.\n",
        "\n",
        "For the problems we explore in this article, we could use NumPy ndarrays instead of the pandas series. The question at stake is whether this would be more efficient or not.\n",
        "\n",
        "Again, we will calculate the sum of the ranks of all the cards in each hand. We convert our rank arrays from pandas Series to NumPy arrays simply by using the .values method of pandas Series, which returns a pandas Series as a NumPy ndarray. As with vectorization on the series, passing the NumPy array directly into the function will lead pandas to apply the function to the entire vector.\n",
        "\n"
      ],
      "metadata": {
        "id": "WO9COAX876tv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "start_time = time.time()\n",
        "\n",
        "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].values.sum(axis=1)\n",
        "\n",
        "print(\"Time using NumPy vectorization: {} sec\" .format(time.time() - start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wJaJBbiI5Q5M",
        "outputId": "27ca2ac9-9a87-4a3d-8b0d-2f1ec772a7d4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using NumPy vectorization: 0.001745462417602539 sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "start_time = time.time()\n",
        "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].sum(axis=1)\n",
        "print(\"Time using the pandas vectorization %s seconds\" % (time.time() - start_time))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OVGS_FGb5eeV",
        "outputId": "e8635e7b-ef4a-42a5-b4b2-ddca0778cd56"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time using the  0.003729104995727539 seconds\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 5. Summary of best practices for looping through DataFrame\n",
        "* Using **.iterrows()** does not improve the speed of iterating through the DataFrame but it is more efficient.\n",
        "* The **.apply()** function performs faster when we want to iterate through all the rows of a pandas DataFrame, but is slower when we perform the same operation through a column.\n",
        "* Vectorizing over the pandas series achieves the overwhelming majority of optimization needs for everyday calculations. However, if speed is of the highest priority, we can call in reinforcements in the form of the NumPy Python library."
      ],
      "metadata": {
        "id": "lXeC6MGF8MfY"
      }
    }
  ]
}