Pandas/Best-Pandas-Apply.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c7812816",
   "metadata": {},
   "source": [
    "# Daily-Dose-of-Data-Science\n",
    "\n",
    "[Daily Dose of Data Science](https://avichawla.substack.com) is a publication on Substack that brings together intriguing frameworks, libraries, technologies, and tips that make the life cycle of a Data Science project effortless. \n",
    "\n",
    "Author: Avi Chawla\n",
    "\n",
    "[Medium](https://medium.com/@avi_chawla) | [LinkedIn](https://www.linkedin.com/in/avi-chawla/)\n",
    "\n",
    "# The Best Way to Use Apply() in Pandas\n",
    "\n",
    "Post Link: [Substack](https://avichawla.substack.com/p/the-best-way-to-use-apply-in-pandas)\n",
    "\n",
    "LinkedIn Post: [LinkedIn](https://www.linkedin.com/posts/avi-chawla_python-datascience-pandas-activity-7007298172301520896-5xHq?utm_source=share&utm_medium=member_desktop)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d05fc64a",
   "metadata": {},
   "source": [
    "## Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd6cb195",
   "metadata": {},
   "source": [
    "Swifter: [https://github.com/jmcarpenter2/swifter](https://github.com/jmcarpenter2/swifter)\n",
    "\n",
    "Pandarallel: [https://github.com/nalepae/pandarallel](https://github.com/nalepae/pandarallel)\n",
    "\n",
    "Parallel Pandas: [https://pypi.org/project/parallel-pandas/](https://pypi.org/project/parallel-pandas/)\n",
    "\n",
    "Mapply: [https://pypi.org/project/mapply/](https://pypi.org/project/mapply/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4108012e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:10:41.284078Z",
     "start_time": "2022-12-10T11:10:23.460778Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandarallel in /Users/avi/anaconda3/lib/python3.9/site-packages (1.6.3)\n",
      "Requirement already satisfied: psutil in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandarallel) (5.8.0)\n",
      "Requirement already satisfied: dill>=0.3.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandarallel) (0.3.6)\n",
      "Requirement already satisfied: pandas>=1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandarallel) (1.4.4)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1->pandarallel) (2.8.2)\n",
      "Requirement already satisfied: numpy>=1.18.5 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1->pandarallel) (1.21.5)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1->pandarallel) (2021.3)\n",
      "Requirement already satisfied: six>=1.5 in /Users/avi/anaconda3/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas>=1->pandarallel) (1.16.0)\n",
      "Requirement already satisfied: mapply in /Users/avi/anaconda3/lib/python3.9/site-packages (0.1.17)\n",
      "Requirement already satisfied: pathos>=0.2.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from mapply) (0.3.0)\n",
      "Requirement already satisfied: tqdm>=4.27 in /Users/avi/anaconda3/lib/python3.9/site-packages (from mapply) (4.64.0)\n",
      "Requirement already satisfied: psutil in /Users/avi/anaconda3/lib/python3.9/site-packages (from mapply) (5.8.0)\n",
      "Requirement already satisfied: pox>=0.3.2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pathos>=0.2.0->mapply) (0.3.2)\n",
      "Requirement already satisfied: ppft>=1.7.6.6 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pathos>=0.2.0->mapply) (1.7.6.6)\n",
      "Requirement already satisfied: multiprocess>=0.70.14 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pathos>=0.2.0->mapply) (0.70.14)\n",
      "Requirement already satisfied: dill>=0.3.6 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pathos>=0.2.0->mapply) (0.3.6)\n",
      "Collecting pandas-parallel\n",
      "  Downloading pandas-parallel-0.1.7.tar.gz (3.6 kB)\n",
      "Requirement already satisfied: pandas in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas-parallel) (1.4.4)\n",
      "Requirement already satisfied: numpy in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas-parallel) (1.21.5)\n",
      "Requirement already satisfied: tqdm in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas-parallel) (4.64.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas->pandas-parallel) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas->pandas-parallel) (2021.3)\n",
      "Requirement already satisfied: six>=1.5 in /Users/avi/anaconda3/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas->pandas-parallel) (1.16.0)\n",
      "Building wheels for collected packages: pandas-parallel\n",
      "  Building wheel for pandas-parallel (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for pandas-parallel: filename=pandas_parallel-0.1.7-py3-none-any.whl size=5210 sha256=f345d9b536a942199cee8b4c0fba0690eebe07532c3b75ebacff190ceaf1fb53\n",
      "  Stored in directory: /Users/avi/Library/Caches/pip/wheels/52/5a/6f/1e9df17fb398cb15349aadc61ae90e587fd93ff8c7214938b6\n",
      "Successfully built pandas-parallel\n",
      "Installing collected packages: pandas-parallel\n",
      "Successfully installed pandas-parallel-0.1.7\n",
      "Requirement already satisfied: swifter in /Users/avi/anaconda3/lib/python3.9/site-packages (1.3.4)\n",
      "Requirement already satisfied: pandas>=1.0.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from swifter) (1.4.4)\n",
      "Requirement already satisfied: ipywidgets>=7.0.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from swifter) (7.6.5)\n",
      "Requirement already satisfied: dask[dataframe]>=2.10.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from swifter) (2022.2.1)\n",
      "Requirement already satisfied: bleach>=3.1.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from swifter) (4.1.0)\n",
      "Requirement already satisfied: psutil>=5.6.6 in /Users/avi/anaconda3/lib/python3.9/site-packages (from swifter) (5.8.0)\n",
      "Requirement already satisfied: cloudpickle>=0.2.2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from swifter) (2.0.0)\n",
      "Requirement already satisfied: tqdm>=4.33.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from swifter) (4.64.0)\n",
      "Requirement already satisfied: parso>0.4.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from swifter) (0.8.3)\n",
      "Requirement already satisfied: webencodings in /Users/avi/anaconda3/lib/python3.9/site-packages (from bleach>=3.1.1->swifter) (0.5.1)\n",
      "Requirement already satisfied: six>=1.9.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from bleach>=3.1.1->swifter) (1.16.0)\n",
      "Requirement already satisfied: packaging in /Users/avi/anaconda3/lib/python3.9/site-packages (from bleach>=3.1.1->swifter) (21.3)\n",
      "Requirement already satisfied: toolz>=0.8.2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from dask[dataframe]>=2.10.0->swifter) (0.11.2)\n",
      "Requirement already satisfied: pyyaml>=5.3.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from dask[dataframe]>=2.10.0->swifter) (5.4.1)\n",
      "Requirement already satisfied: fsspec>=0.6.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from dask[dataframe]>=2.10.0->swifter) (2022.2.0)\n",
      "Requirement already satisfied: partd>=0.3.10 in /Users/avi/anaconda3/lib/python3.9/site-packages (from dask[dataframe]>=2.10.0->swifter) (1.2.0)\n",
      "Requirement already satisfied: numpy>=1.18 in /Users/avi/anaconda3/lib/python3.9/site-packages (from dask[dataframe]>=2.10.0->swifter) (1.21.5)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipywidgets>=7.0.0->swifter) (0.2.0)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipywidgets>=7.0.0->swifter) (5.3.0)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipywidgets>=7.0.0->swifter) (3.5.2)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipywidgets>=7.0.0->swifter) (6.9.1)\n",
      "Requirement already satisfied: ipython>=4.0.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipywidgets>=7.0.0->swifter) (8.2.0)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipywidgets>=7.0.0->swifter) (1.0.0)\n",
      "Requirement already satisfied: traitlets>=4.3.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipywidgets>=7.0.0->swifter) (5.1.1)\n",
      "Requirement already satisfied: appnope in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (0.1.2)\n",
      "Requirement already satisfied: nest-asyncio in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (1.5.5)\n",
      "Requirement already satisfied: debugpy<2.0,>=1.0.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (1.5.1)\n",
      "Requirement already satisfied: tornado<7.0,>=4.2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (6.1)\n",
      "Requirement already satisfied: jupyter-client<8.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (6.1.12)\n",
      "Requirement already satisfied: matplotlib-inline<0.2.0,>=0.1.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (0.1.2)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (3.0.20)\n",
      "Requirement already satisfied: jedi>=0.16 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (0.18.1)\n",
      "Requirement already satisfied: pygments>=2.4.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (2.11.2)\n",
      "Requirement already satisfied: stack-data in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (0.2.0)\n",
      "Requirement already satisfied: pickleshare in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (0.7.5)\n",
      "Requirement already satisfied: backcall in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (0.2.0)\n",
      "Requirement already satisfied: decorator in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (5.1.1)\n",
      "Requirement already satisfied: setuptools>=18.5 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (61.2.0)\n",
      "Requirement already satisfied: pexpect>4.3 in /Users/avi/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (4.8.0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: python-dateutil>=2.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (2.8.2)\n",
      "Requirement already satisfied: pyzmq>=13 in /Users/avi/anaconda3/lib/python3.9/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (22.3.0)\n",
      "Requirement already satisfied: jupyter-core>=4.6.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.0.0->swifter) (4.9.2)\n",
      "Requirement already satisfied: jsonschema>=2.6 in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbformat>=4.2.0->ipywidgets>=7.0.0->swifter) (3.2.0)\n",
      "Requirement already satisfied: fastjsonschema in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbformat>=4.2.0->ipywidgets>=7.0.0->swifter) (2.15.1)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->ipywidgets>=7.0.0->swifter) (0.18.0)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->ipywidgets>=7.0.0->swifter) (21.4.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from packaging->bleach>=3.1.1->swifter) (3.0.4)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1.0.0->swifter) (2021.3)\n",
      "Requirement already satisfied: locket in /Users/avi/anaconda3/lib/python3.9/site-packages (from partd>=0.3.10->dask[dataframe]>=2.10.0->swifter) (0.2.1)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pexpect>4.3->ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /Users/avi/anaconda3/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (0.2.5)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (6.4.8)\n",
      "Requirement already satisfied: Send2Trash>=1.8.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (1.8.0)\n",
      "Requirement already satisfied: prometheus-client in /Users/avi/anaconda3/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (0.13.1)\n",
      "Requirement already satisfied: argon2-cffi in /Users/avi/anaconda3/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (21.3.0)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /Users/avi/anaconda3/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (0.13.1)\n",
      "Requirement already satisfied: nbconvert in /Users/avi/anaconda3/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (7.2.3)\n",
      "Requirement already satisfied: jinja2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (3.0.0)\n",
      "Requirement already satisfied: argon2-cffi-bindings in /Users/avi/anaconda3/lib/python3.9/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (21.2.0)\n",
      "Requirement already satisfied: cffi>=1.0.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (1.15.0)\n",
      "Requirement already satisfied: pycparser in /Users/avi/anaconda3/lib/python3.9/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (2.21)\n",
      "Requirement already satisfied: MarkupSafe>=2.0.0rc2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (2.1.1)\n",
      "Requirement already satisfied: jupyterlab-pygments in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (0.1.2)\n",
      "Requirement already satisfied: defusedxml in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (0.7.1)\n",
      "Requirement already satisfied: importlib-metadata>=3.6 in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (4.11.3)\n",
      "Requirement already satisfied: beautifulsoup4 in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (4.11.1)\n",
      "Requirement already satisfied: nbclient>=0.5.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (0.5.13)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (1.5.0)\n",
      "Requirement already satisfied: mistune<3,>=2.0.3 in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (2.0.4)\n",
      "Requirement already satisfied: tinycss2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (1.2.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /Users/avi/anaconda3/lib/python3.9/site-packages (from importlib-metadata>=3.6->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (3.7.0)\n",
      "Requirement already satisfied: soupsieve>1.2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from beautifulsoup4->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->swifter) (2.3.1)\n",
      "Requirement already satisfied: executing in /Users/avi/anaconda3/lib/python3.9/site-packages (from stack-data->ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (0.8.3)\n",
      "Requirement already satisfied: asttokens in /Users/avi/anaconda3/lib/python3.9/site-packages (from stack-data->ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (2.0.5)\n",
      "Requirement already satisfied: pure-eval in /Users/avi/anaconda3/lib/python3.9/site-packages (from stack-data->ipython>=4.0.0->ipywidgets>=7.0.0->swifter) (0.2.2)\n"
     ]
    }
   ],
   "source": [
    "!pip install pandarallel\n",
    "!pip install mapply\n",
    "!pip install pandas-parallel\n",
    "!pip install swifter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0369e1ae",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:10:56.137676Z",
     "start_time": "2022-12-10T11:10:54.116571Z"
    }
   },
   "outputs": [],
   "source": [
    "import mapply\n",
    "import pandas as pd\n",
    "import swifter\n",
    "import numpy as np\n",
    "from time import perf_counter\n",
    "from pandarallel import pandarallel\n",
    "from parallel_pandas import ParallelPandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ba6484ea",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:11:03.359599Z",
     "start_time": "2022-12-10T11:11:03.021363Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>694195</td>\n",
       "      <td>867235</td>\n",
       "      <td>625585</td>\n",
       "      <td>499243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>829266</td>\n",
       "      <td>925135</td>\n",
       "      <td>347425</td>\n",
       "      <td>698796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>296075</td>\n",
       "      <td>722698</td>\n",
       "      <td>465227</td>\n",
       "      <td>220427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>871312</td>\n",
       "      <td>193671</td>\n",
       "      <td>464802</td>\n",
       "      <td>270806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>752223</td>\n",
       "      <td>333601</td>\n",
       "      <td>904237</td>\n",
       "      <td>650913</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        A       B       C       D\n",
       "0  694195  867235  625585  499243\n",
       "1  829266  925135  347425  698796\n",
       "2  296075  722698  465227  220427\n",
       "3  871312  193671  464802  270806\n",
       "4  752223  333601  904237  650913"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(np.random.randint(1, 10**6, size = (10**7, 4)), columns = list(\"ABCD\"))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cc75bb03",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:11:09.495259Z",
     "start_time": "2022-12-10T11:11:09.488851Z"
    }
   },
   "outputs": [],
   "source": [
    "def sum_row(row):\n",
    "    return sum(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "947cbc87",
   "metadata": {},
   "source": [
    "## Pandas Apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "22d776ff",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:12:58.234543Z",
     "start_time": "2022-12-10T11:12:17.043272Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "41.18502562499998\n"
     ]
    }
   ],
   "source": [
    "start = perf_counter()\n",
    "a = df.apply(sum_row, axis = 1)\n",
    "print(perf_counter()-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66e9dc7f",
   "metadata": {},
   "source": [
    "## Pandarallel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ed85efed",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:12:58.240075Z",
     "start_time": "2022-12-10T11:12:58.236753Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Pandarallel will run on 8 workers.\n",
      "INFO: Pandarallel will use standard multiprocessing data transfer (pipe) to transfer data between the main process and workers.\n"
     ]
    }
   ],
   "source": [
    "pandarallel.initialize(progress_bar=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8986ef3d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:13:09.042873Z",
     "start_time": "2022-12-10T11:12:58.241148Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49ec4cb0df094025b8040f52944f8922",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=1250000), Label(value='0 / 1250000…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.799626000000018\n"
     ]
    }
   ],
   "source": [
    "start = perf_counter()\n",
    "a = df.parallel_apply(sum_row, axis = 1)\n",
    "print(perf_counter()-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8d4ea1f",
   "metadata": {},
   "source": [
    "## Swifter Apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7ade35bc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:13:40.330531Z",
     "start_time": "2022-12-10T11:13:19.253161Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2867bdbbe94946768ab7dd52025a340e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dask Apply:   0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21.07202774999999\n"
     ]
    }
   ],
   "source": [
    "start = perf_counter()\n",
    "a = df.swifter.apply(sum_row, axis = 1)\n",
    "print(perf_counter()-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7478c16b",
   "metadata": {},
   "source": [
    "## Parallel-Pandas Apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2c703288",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:13:40.335391Z",
     "start_time": "2022-12-10T11:13:40.332410Z"
    }
   },
   "outputs": [],
   "source": [
    "ParallelPandas.initialize(n_cpu=16, split_factor=4, disable_pr_bar=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "acb417b5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:13:58.342863Z",
     "start_time": "2022-12-10T11:13:45.135099Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13.201775666000003\n"
     ]
    }
   ],
   "source": [
    "start = perf_counter()\n",
    "a = df.p_apply(sum_row, axis = 1)\n",
    "print(perf_counter()-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6050bb78",
   "metadata": {},
   "source": [
    "## Mapply Apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a0bfd811",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:14:03.442005Z",
     "start_time": "2022-12-10T11:14:03.433460Z"
    }
   },
   "outputs": [],
   "source": [
    "mapply.init(\n",
    "    n_workers=-1,\n",
    "    chunk_size=100,\n",
    "    max_chunks_per_worker=8,\n",
    "    progressbar=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "5be15496",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-10T11:14:17.584976Z",
     "start_time": "2022-12-10T11:14:07.868096Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c5440a36f0d443390c7ee87f8ceac3b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|                                                    | 0/64 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.712804667\n"
     ]
    }
   ],
   "source": [
    "start = perf_counter()\n",
    "a = df.mapply(sum_row, axis = 1)\n",
    "print(perf_counter()-start)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}