Pandas/Polars-vs-Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "47a9f777",
   "metadata": {},
   "source": [
    "# Daily-Dose-of-Data-Science\n",
    "\n",
    "[Daily Dose of Data Science](https://avichawla.substack.com) is a publication on Substack that brings together intriguing frameworks, libraries, technologies, and tips that make the life cycle of a Data Science project effortless. \n",
    "\n",
    "Author: Avi Chawla\n",
    "\n",
    "[Medium](https://medium.com/@avi_chawla) | [LinkedIn](https://www.linkedin.com/in/avi-chawla/)\n",
    "\n",
    "# Pandas vs Polars — Run-time and Memory Comparison\n",
    "\n",
    "Post Link: [Substack](https://www.blog.dailydoseofds.com/p/pandas-vs-polars-run-time-and-memory)\n",
    "\n",
    "LinkedIn Post: [LinkedIn](https://www.linkedin.com/feed/update/urn:li:share:7073950186053484544/)\n",
    "\n",
    "Twitter Post: [Twitter](https://twitter.com/_avichawla/status/1668184509500489731)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4f7b14f7",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: polars in /Users/avi/anaconda3/lib/python3.9/site-packages (0.13.38)\r\n",
      "Requirement already satisfied: typing_extensions>=4.0.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from polars) (4.5.0)\r\n",
      "Requirement already satisfied: numpy>=1.16.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from polars) (1.23.2)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install polars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ecd28581",
   "metadata": {},
   "outputs": [],
   "source": [
    "import polars as pl\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3ae33ec",
   "metadata": {},
   "source": [
    "Download the dataset from here: [Dataset](https://drive.google.com/file/d/1sSugZwVWCSsep0-4Xi7peXCd5dvglm4a/view?usp=sharing)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eebea438",
   "metadata": {},
   "source": [
    "## Read CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f793636a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.52 s ± 171 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit pd.read_csv(\"dataset.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "db5ed30c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "723 ms ± 182 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit pl.read_csv(\"dataset.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3d1cca2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_pd = pd.read_csv(\"dataset.csv\")\n",
    "df_pl = pl.read_csv(\"dataset.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f261d75",
   "metadata": {},
   "source": [
    "## To CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1e82017d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15.3 s ± 515 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pd.to_csv(\"dataset_dummy_pandas.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2ba8d221",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.26 s ± 96.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pl.to_csv(\"dataset_dummy_polars.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5ce6efb",
   "metadata": {},
   "source": [
    "## Memory Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b629b32a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "633987120"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pl.estimated_size() # in Bytes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "73fa7150",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4096000 entries, 0 to 4095999\n",
      "Data columns (total 9 columns):\n",
      " #   Column              Dtype  \n",
      "---  ------              -----  \n",
      " 0   Name                object \n",
      " 1   Company_Name        object \n",
      " 2   Employee_Job_Title  object \n",
      " 3   Employee_City       object \n",
      " 4   Employee_Country    object \n",
      " 5   Employee_Salary     int64  \n",
      " 6   Employment_Status   object \n",
      " 7   Employee_Rating     float64\n",
      " 8   Credits             int64  \n",
      "dtypes: float64(1), int64(2), object(6)\n",
      "memory usage: 1.7 GB\n"
     ]
    }
   ],
   "source": [
    "df_pd.info(memory_usage=\"deep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91569c60",
   "metadata": {},
   "source": [
    "### Selecting Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e66e1ef4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29.2 ms ± 54.7 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pd[[\"Name\", \"Employee_Rating\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "06fb78be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.22 µs ± 35.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pl[[\"Name\", \"Employee_Rating\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d5c3293",
   "metadata": {},
   "source": [
    "### Filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ea1e6bf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "130 ms ± 16.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pd[df_pd.Credits>2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "edef1319",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "93.5 ms ± 7.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pl.filter(pl.col('Credits') > 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f89eda9b",
   "metadata": {},
   "source": [
    "### Grouping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6cf2727a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "219 ms ± 8.75 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pd.groupby(\"Company_Name\").Employee_Salary.mean().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6b389237",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47.5 ms ± 1.72 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pl.groupby(\"Company_Name\").agg([(\"Employee_Salary\", \"mean\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e82f7c22",
   "metadata": {},
   "source": [
    "### Sorting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "87fddf36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "884 ms ± 55.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pd.sort_values(\"Employee_Salary\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f9c58719",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "932 ms ± 287 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_pl.sort(\"Employee_Salary\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}