Run-time Optimization/Parquet-Optimize.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ebbb7884",
   "metadata": {},
   "source": [
    "# Daily-Dose-of-Data-Science\n",
    "\n",
    "[Daily Dose of Data Science](https://avichawla.substack.com) is a publication on Substack that brings together intriguing frameworks, libraries, technologies, and tips that make the life cycle of a Data Science project effortless. \n",
    "\n",
    "Author: Avi Chawla\n",
    "\n",
    "[Medium](https://medium.com/@avi_chawla) | [LinkedIn](https://www.linkedin.com/in/avi-chawla/)\n",
    "\n",
    "# Speed-up Parquet I/O of Pandas by 5x\n",
    "\n",
    "Post Link: [Substack](https://avichawla.substack.com/p/speed-up-parquet-io-of-pandas-by)\n",
    "\n",
    "LinkedIn Post: [LinkedIn](https://www.linkedin.com/feed/update/urn:li:activity:7030853460430307328/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71b7587d",
   "metadata": {},
   "source": [
    "## Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6a46f0bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "53b71434",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.6 s, sys: 10.1 s, total: 21.7 s\n",
      "Wall time: 25.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "df = pd.read_parquet(\"../_Datasets/employee_dataset.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60734eb6",
   "metadata": {},
   "source": [
    "## fastparquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5a31aadd",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: fastparquet in /Users/avi/anaconda3/lib/python3.9/site-packages (2023.2.0)\n",
      "Requirement already satisfied: pandas>=1.5.0 in /Users/avi/anaconda3/lib/python3.9/site-packages (from fastparquet) (1.5.0)\n",
      "Requirement already satisfied: fsspec in /Users/avi/anaconda3/lib/python3.9/site-packages (from fastparquet) (2022.2.0)\n",
      "Requirement already satisfied: packaging in /Users/avi/anaconda3/lib/python3.9/site-packages (from fastparquet) (21.3)\n",
      "Requirement already satisfied: cramjam>=2.3 in /Users/avi/anaconda3/lib/python3.9/site-packages (from fastparquet) (2.6.2)\n",
      "Requirement already satisfied: numpy>=1.20.3 in /Users/avi/anaconda3/lib/python3.9/site-packages (from fastparquet) (1.24.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1.5.0->fastparquet) (2021.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /Users/avi/anaconda3/lib/python3.9/site-packages (from pandas>=1.5.0->fastparquet) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in /Users/avi/anaconda3/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas>=1.5.0->fastparquet) (1.16.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/avi/anaconda3/lib/python3.9/site-packages (from packaging->fastparquet) (3.0.4)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "!pip install fastparquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1870b21b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastparquet import ParquetFile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2935a8e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.27 s, sys: 753 ms, total: 4.02 s\n",
      "Wall time: 4.34 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "pf = ParquetFile(\"../_Datasets/employee_dataset.parquet\")\n",
    "df = pf.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95d8f8b8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}