Pandas/Pandarallel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6bcdc9e3",
   "metadata": {},
   "source": [
    "# Daily-Dose-of-Data-Science\n",
    "\n",
    "[Daily Dose of Data Science](https://avichawla.substack.com) is a publication on Substack that brings together intriguing frameworks, libraries, technologies, and tips that make the life cycle of a Data Science project effortless. \n",
    "\n",
    "Author: Avi Chawla\n",
    "\n",
    "[Medium](https://medium.com/@avi_chawla) | [LinkedIn](https://www.linkedin.com/in/avi-chawla/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4df7f42d",
   "metadata": {},
   "source": [
    "# Parallelize Pandas with Pandarallel\n",
    "\n",
    "Post Link: [Substack](https://avichawla.substack.com/p/parallelize-pandas-with-pandarallel)\n",
    "\n",
    "LinkedIn Post: [LinkedIn](https://www.linkedin.com/posts/avi-chawla_datascience-python-pandas-activity-6985871823158562816-IeKY?utm_source=share&utm_medium=member_desktop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "86c60d17",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-01T19:21:44.917915Z",
     "start_time": "2022-12-01T19:21:41.601912Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandarallel in /Users/avi/anaconda3/lib/python3.9/site-packages (1.6.3)\n",
      "Requirement already satisfied: dill>=0.3.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandarallel) (0.3.6)\n",
      "Requirement already satisfied: pandas>=1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandarallel) (1.5.0)\n",
      "Requirement already satisfied: psutil in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandarallel) (5.8.0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1->pandarallel) (2021.3)\n",
      "Requirement already satisfied: numpy>=1.20.3 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1->pandarallel) (1.21.5)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1->pandarallel) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in /Users/avi/anaconda3/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas>=1->pandarallel) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install pandarallel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "58fa75f3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-01T19:21:46.085398Z",
     "start_time": "2022-12-01T19:21:46.074274Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Pandarallel will run on 8 workers.\n",
      "INFO: Pandarallel will use standard multiprocessing data transfer (pipe) to transfer data between the main process and workers.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from pandarallel import pandarallel\n",
    "pandarallel.initialize(progress_bar=True)\n",
    "\n",
    "from time import time\n",
    "from tqdm.notebook import tqdm\n",
    "tqdm.pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7140c609",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-01T19:21:47.866022Z",
     "start_time": "2022-12-01T19:21:47.563737Z"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(np.random.randint(1, 10**6, (10**7, 4)), columns = [\"col1\", \"col2\", \"col3\", \"col4\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9b8a8d92",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-01T19:21:51.712787Z",
     "start_time": "2022-12-01T19:21:51.707838Z"
    }
   },
   "outputs": [],
   "source": [
    "def add_row(row):\n",
    "    return sum(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "406417e5",
   "metadata": {},
   "source": [
    "## Pandas Apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "47812b91",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-01T19:28:02.357735Z",
     "start_time": "2022-12-01T19:24:51.623163Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "38.1 s ± 5.98 s per loop (mean ± std. dev. of 5 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit -n1 -r5 sum_apply = df.apply(add_row, axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29cd5db6",
   "metadata": {},
   "source": [
    "## Pandarallel Apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d8057063",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-01T19:29:17.484122Z",
     "start_time": "2022-12-01T19:28:10.191493Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "40a236b683d84b9e807d0e6f407fee71",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=1250000), Label(value='0 / 1250000…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1224e2b80418444cbf4078828cea0b15",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=1250000), Label(value='0 / 1250000…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff844d421479434ca19f0f120c0d3b3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=1250000), Label(value='0 / 1250000…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8892f4b28e7f4a18ae71cb06e11245ad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=1250000), Label(value='0 / 1250000…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6bf933670d424c56aeee953642e3c032",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=1250000), Label(value='0 / 1250000…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13.4 s ± 888 ms per loop (mean ± std. dev. of 5 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit -n1 -r5 sum_parallel_apply = df.parallel_apply(add_row, axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95bd712e",
   "metadata": {},
   "source": [
    "### Further Reading: [Introducing Pandarallel: Never Use The Apply Method In Pandas Again](https://towardsdatascience.com/introducing-pandarallel-never-use-the-apply-method-in-pandas-again-7f3ff74fbe8e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19aa0c92",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}