Pandas/CSV-Feather-Parquet-Pickle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c3934ee4",
   "metadata": {},
   "source": [
    "# Daily-Dose-of-Data-Science\n",
    "\n",
    "[Daily Dose of Data Science](https://avichawla.substack.com) is a publication on Substack that brings together intriguing frameworks, libraries, technologies, and tips that make the life cycle of a Data Science project effortless. \n",
    "\n",
    "Author: Avi Chawla\n",
    "\n",
    "[Medium](https://medium.com/@avi_chawla) | [LinkedIn](https://www.linkedin.com/in/avi-chawla/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "413fd92c",
   "metadata": {},
   "source": [
    "# The Best File Format To Store A Pandas DataFrame\n",
    "\n",
    "Post Link: [Substack](https://avichawla.substack.com/p/the-best-file-format-to-store-a-pandas)\n",
    "\n",
    "LinkedIn Post: [LinkedIn](https://www.linkedin.com/posts/avi-chawla_python-datascience-pandas-activity-7008390345101066240-3yJM?utm_source=share&utm_medium=member_desktop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3d30317f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:38:22.843723Z",
     "start_time": "2022-12-13T11:38:21.912724Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Company_Name</th>\n",
       "      <th>Employee_Job_Title</th>\n",
       "      <th>Employee_City</th>\n",
       "      <th>Employee_Country</th>\n",
       "      <th>Employee_Salary</th>\n",
       "      <th>Employment_Status</th>\n",
       "      <th>Employee_Rating</th>\n",
       "      <th>Credits</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Johnny Maynard</td>\n",
       "      <td>White, Mcclain and Cobb</td>\n",
       "      <td>Equities trader</td>\n",
       "      <td>New Cindychester</td>\n",
       "      <td>Mongolia</td>\n",
       "      <td>764340</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>3.3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Michael Williams</td>\n",
       "      <td>Scott Inc</td>\n",
       "      <td>Patent examiner</td>\n",
       "      <td>Ricardomouth</td>\n",
       "      <td>Guinea-Bissau</td>\n",
       "      <td>929230</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>4.6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Laura Flynn</td>\n",
       "      <td>Andrade LLC</td>\n",
       "      <td>Equities trader</td>\n",
       "      <td>North Melissafurt</td>\n",
       "      <td>Tonga</td>\n",
       "      <td>606900</td>\n",
       "      <td>Intern</td>\n",
       "      <td>0.3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Stefanie Archer</td>\n",
       "      <td>James and Sons</td>\n",
       "      <td>Retail merchandiser</td>\n",
       "      <td>Ricardomouth</td>\n",
       "      <td>San Marino</td>\n",
       "      <td>775380</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>2.8</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sierra Garcia</td>\n",
       "      <td>Matthews Inc</td>\n",
       "      <td>Armed forces logistics/support/administrative ...</td>\n",
       "      <td>Whitakerbury</td>\n",
       "      <td>Nepal</td>\n",
       "      <td>944480</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>4.5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Name             Company_Name  \\\n",
       "0    Johnny Maynard  White, Mcclain and Cobb   \n",
       "1  Michael Williams                Scott Inc   \n",
       "2       Laura Flynn              Andrade LLC   \n",
       "3   Stefanie Archer           James and Sons   \n",
       "4     Sierra Garcia             Matthews Inc   \n",
       "\n",
       "                                  Employee_Job_Title      Employee_City  \\\n",
       "0                                    Equities trader   New Cindychester   \n",
       "1                                    Patent examiner       Ricardomouth   \n",
       "2                                    Equities trader  North Melissafurt   \n",
       "3                                Retail merchandiser       Ricardomouth   \n",
       "4  Armed forces logistics/support/administrative ...       Whitakerbury   \n",
       "\n",
       "  Employee_Country  Employee_Salary Employment_Status  Employee_Rating  \\\n",
       "0         Mongolia           764340         Full Time              3.3   \n",
       "1    Guinea-Bissau           929230         Full Time              4.6   \n",
       "2            Tonga           606900            Intern              0.3   \n",
       "3       San Marino           775380         Full Time              2.8   \n",
       "4            Nepal           944480         Full Time              4.5   \n",
       "\n",
       "   Credits  \n",
       "0        1  \n",
       "1        4  \n",
       "2        3  \n",
       "3        5  \n",
       "4        4  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "df = pd.read_csv(\"../_Datasets/employee_dataset.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0c4d11c8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:38:23.308630Z",
     "start_time": "2022-12-13T11:38:22.844767Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2000, 9)\n",
      "(4000, 9)\n",
      "(8000, 9)\n",
      "(16000, 9)\n",
      "(32000, 9)\n",
      "(64000, 9)\n",
      "(128000, 9)\n",
      "(256000, 9)\n",
      "(512000, 9)\n",
      "(1024000, 9)\n",
      "(2048000, 9)\n",
      "(4096000, 9)\n",
      "(8192000, 9)\n"
     ]
    }
   ],
   "source": [
    "for i in range(13):\n",
    "    df = pd.concat((df, df))\n",
    "    print(df.shape)\n",
    "    \n",
    "df.reset_index(drop = True, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72645852",
   "metadata": {},
   "source": [
    "## Dump to CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "76bee2eb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:03.236827Z",
     "start_time": "2022-12-13T11:38:32.515567Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 28.8 s, sys: 675 ms, total: 29.5 s\n",
      "Wall time: 30.7 s\n"
     ]
    }
   ],
   "source": [
    "%time df.to_csv(\"../_Datasets/dump_to_csv.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "89842082",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:03.240557Z",
     "start_time": "2022-12-13T11:39:03.238117Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File Size is 862.112359046936 MB\n"
     ]
    }
   ],
   "source": [
    "file_size = os.path.getsize('../_Datasets/dump_to_csv.csv')\n",
    "print(f\"File Size is {file_size/(1024**2)} MB\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e551f486",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:03.420752Z",
     "start_time": "2022-12-13T11:39:03.241316Z"
    }
   },
   "outputs": [],
   "source": [
    "!rm ../_Datasets/dump_to_csv.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ca834e6",
   "metadata": {},
   "source": [
    "## Dump to Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bdbd7dfd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:08.637833Z",
     "start_time": "2022-12-13T11:39:07.076506Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.09 s, sys: 217 ms, total: 1.31 s\n",
      "Wall time: 1.56 s\n"
     ]
    }
   ],
   "source": [
    "%time df.to_pickle(\"../_Datasets/dump_to_pickle.pickle\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "477bb75b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:08.641238Z",
     "start_time": "2022-12-13T11:39:08.639277Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File Size is 416.6713104248047 MB\n"
     ]
    }
   ],
   "source": [
    "file_size = os.path.getsize('../_Datasets/dump_to_pickle.pickle')\n",
    "print(f\"File Size is {file_size/(1024**2)} MB\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f80bb47b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:08.806584Z",
     "start_time": "2022-12-13T11:39:08.642111Z"
    }
   },
   "outputs": [],
   "source": [
    "!rm ../_Datasets/dump_to_pickle.pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ba9fd30",
   "metadata": {},
   "source": [
    "## Dump to Parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "85b632e8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:17.437502Z",
     "start_time": "2022-12-13T11:39:12.367249Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.72 s, sys: 776 ms, total: 4.5 s\n",
      "Wall time: 5.06 s\n"
     ]
    }
   ],
   "source": [
    "%time df.to_parquet(\"../_Datasets/dump_to_parquet.parquet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1d874a81",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:17.442547Z",
     "start_time": "2022-12-13T11:39:17.439664Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File Size is 3.996185302734375 MB\n"
     ]
    }
   ],
   "source": [
    "file_size = os.path.getsize('../_Datasets/dump_to_parquet.parquet')\n",
    "print(f\"File Size is {file_size/(1024**2)} MB\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6219231f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:17.581678Z",
     "start_time": "2022-12-13T11:39:17.443505Z"
    }
   },
   "outputs": [],
   "source": [
    "!rm ../_Datasets/dump_to_parquet.parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1520b1ee",
   "metadata": {},
   "source": [
    "## Dump to Feather"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "dfeee365",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:58.462888Z",
     "start_time": "2022-12-13T11:39:56.070878Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.14 s, sys: 758 ms, total: 2.9 s\n",
      "Wall time: 2.39 s\n"
     ]
    }
   ],
   "source": [
    "%time df.to_feather(\"../_Datasets/dump_to_feather.feather\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "01641c90",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:58.466379Z",
     "start_time": "2022-12-13T11:39:58.464286Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File Size is 282.39537239074707 MB\n"
     ]
    }
   ],
   "source": [
    "file_size = os.path.getsize('../_Datasets/dump_to_feather.feather')\n",
    "print(f\"File Size is {file_size/(1024**2)} MB\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1a31a8a1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-13T11:39:59.512311Z",
     "start_time": "2022-12-13T11:39:59.325621Z"
    }
   },
   "outputs": [],
   "source": [
    "!rm ../_Datasets/dump_to_feather.feather"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcecd75d",
   "metadata": {},
   "source": [
    "### Further reading: [Why I Stopped Dumping DataFrames to a CSV and Why You Should Too.](https://towardsdatascience.com/why-i-stopped-dumping-dataframes-to-a-csv-and-why-you-should-too-c0954c410f8f)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}