Convert_netCDF_to_CSV_files/Convert_netCDF_to_PARQUET.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "36103330",
   "metadata": {},
   "source": [
    "## Objective:\n",
    "- Import required libraries data pre-processing; numpy, pyarrow\n",
    "- Convert Entire OCO2/OCO3 netCDF files to parquet format\n",
    "- Locate the path for the multiple files to convert into parquet\n",
    "- Additional columns like `verticies` of carbon footprints are extracted for polygon shape visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1d982df",
   "metadata": {},
   "source": [
    "### Why Parquet format?\n",
    "Parquet is an open source file format built to handle flat columnar storage data formats. Parquet operates well with complex data in large volumes.It is known for its both performant data compression and its ability to handle a wide variety of encoding types. \n",
    "\n",
    "- Parquet deploys Google's record-shredding and assembly algorithm that can address complex data structures within data storage. Some Parquet benefits include:\n",
    "\n",
    "    * Fast queries that can fetch specific column values without reading full row data\n",
    "    * Highly efficient column-wise compression\n",
    "    * High compatibility with with OLAP\n",
    "    \n",
    "### Benifits of using Parquet format over CSV\n",
    "- HOW IS PARQUET DIFFERENT FROM CSV?\n",
    "    - While CSV is simple and the most widely used data format (Excel, Google Sheets), there are several distinct advantages for Parquet, including:\n",
    "    - Parquet is column oriented and CSV is row oriented. Row-oriented formats are optimized for OLTP workloads while column-oriented formats are better suited for analytical workloads.\n",
    "    - Column-oriented databases such as AWS Redshift Spectrum bill by the amount data scanned per query\n",
    "    - Therefore, converting CSV to Parquet with partitioning and compression lowers overall costs and improves performance\n",
    "\n",
    "[source: Snowflake](https://www.snowflake.com/guides/what-parquet#:~:text=Parquet%20is%20an%20open%20source,wide%20variety%20of%20encoding%20types.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ee7cd134",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install pyarrow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "544eb55a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pyarrow as pa\n",
    "import netCDF4 as nc\n",
    "\n",
    "from pyarrow import parquet as parq\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "95e4f19a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6bd367bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Example: creating a parquet file using table using the array\n",
    "arr= np.arange(1.0, 20.0)\n",
    "pa_table= pa.table({\"float_data\": arr})\n",
    "parq.write_table(pa_table, 'test_table_data.parquet')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1de66a1",
   "metadata": {},
   "source": [
    "### NETCDF files\n",
    "- PATH: downloaded netCDF files from the source(Earthdata search)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69ba337b",
   "metadata": {},
   "source": [
    "### Direcotry with all files\n",
    "- this example shows files read from a single dir. for year 20021"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e0879b85",
   "metadata": {},
   "outputs": [],
   "source": [
    "path_= \"../../../Cluster_machine/OCO2/B_11_new_version/2021/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e23803b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['../../../Cluster_machine/OCO2/B_11_new_version/2021/01\\\\01\\\\LtCO2\\\\oco2_LtCO2_210101_B11014Ar_220729005635s.nc4',\n",
       " '../../../Cluster_machine/OCO2/B_11_new_version/2021/01\\\\02\\\\LtCO2\\\\oco2_LtCO2_210102_B11014Ar_220729005715s.nc4',\n",
       " '../../../Cluster_machine/OCO2/B_11_new_version/2021/01\\\\03\\\\LtCO2\\\\oco2_LtCO2_210103_B11014Ar_220729005741s.nc4']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_path_= []\n",
    "\n",
    "for root, dirs, files in os.walk(path_):\n",
    "    \n",
    "    for filename in files:\n",
    "        #print(os.path.join(root, filename))\n",
    "        \n",
    "        # Append the files into list\n",
    "        file_path_.append(os.path.join(root, filename))\n",
    "        \n",
    "file_path_[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e93e9044",
   "metadata": {},
   "outputs": [],
   "source": [
    "file_names= file_path_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "007576c9",
   "metadata": {},
   "source": [
    "## conv dateTime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "da133f36",
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv_date(d):\n",
    "    return datetime.strptime(str(d), '%Y%m%d%H%M%S%f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8441ef3c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 3min 25s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "countFiles=0\n",
    "\n",
    "lon_list= []\n",
    "lat_list= []\n",
    "xco2_list= []\n",
    "qual_flag_list= []\n",
    "lat=[]\n",
    "lon=[]\n",
    "dateTime= []\n",
    "\n",
    "for j in file_names:\n",
    "    if j.endswith(\".nc4\"):\n",
    "        var_= nc.Dataset(j)\n",
    "        lon_list.append(np.array(var_.variables['vertex_longitude'][:]).tolist())\n",
    "        lat_list.append(np.array(var_.variables['vertex_latitude'][:]).tolist())\n",
    "        xco2_list.append(np.array(var_.variables['xco2'][:]).tolist())\n",
    "        qual_flag_list.append(np.array(var_.variables['xco2_quality_flag'][:]).tolist())\n",
    "        \n",
    "        # DateTIme formating\n",
    "        dateTime.append(np.array(var_.variables['sounding_id'][:].tolist()))\n",
    "\n",
    "        # lat and long without vert\n",
    "        lat.append(np.array(var_.variables['latitude'][:].tolist()))\n",
    "        lon.append(np.array(var_.variables['longitude'][:].tolist()))\n",
    "\n",
    "#print('\\nTotalFiles: ', countFiles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1e2fcaa7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 40.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lon_list_a= [element for sublist in lon_list for element in sublist]\n",
    "lat_list_a= [element for sublist in lat_list for element in sublist]\n",
    "xco2_list_a= [element for sublist in xco2_list for element in sublist]\n",
    "lon_a= [element for sublist in lon for element in sublist]\n",
    "lat_a= [element for sublist in lat for element in sublist]\n",
    "xco2_qual_flag= [ element for sublist in qual_flag_list for element in sublist]\n",
    "dateTime_list= [ element for sublist in dateTime for element in sublist]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f6f7aa3",
   "metadata": {},
   "source": [
    "### Transformation to parquet format\n",
    "- Creating table format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "399151a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 3min 13s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_oco3= pa.table({\n",
    "    'Latitude_vertices': lat_list_a,\n",
    "    'Longitude_vertices': lon_list_a,\n",
    "    'Latitude': lat_a,\n",
    "    'Longitude': lon_a,\n",
    "    'Xco2': xco2_list_a,\n",
    "    'quality_flag': xco2_qual_flag,\n",
    "    'DateTime': dateTime_list\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70749bce",
   "metadata": {},
   "source": [
    "## Table from parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4d70791e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'oco3_20222_parq.parquet'\n",
    "#file_name= input(\"File Name\")\n",
    "file_name= \"oco2_2021_parq.parquet\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8feaffa0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 36.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "parq.write_table(df_oco3, file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a95655bb",
   "metadata": {},
   "source": [
    "# Pre-processing the original file\n",
    "- CONVERT the DATETIME -> Date and Month\n",
    "- Filter by quality flag\n",
    "    - Good Quality-> 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "54faca7f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2min 18s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Latitude_vertices</th>\n",
       "      <th>Longitude_vertices</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Xco2</th>\n",
       "      <th>quality_flag</th>\n",
       "      <th>DateTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[-84.71208953857422, -84.71642303466797, -84.7...</td>\n",
       "      <td>[-51.488616943359375, -51.684295654296875, -51...</td>\n",
       "      <td>-84.725594</td>\n",
       "      <td>-51.603241</td>\n",
       "      <td>409.039856</td>\n",
       "      <td>1</td>\n",
       "      <td>2021010100041302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[-84.9544906616211, -84.9550552368164, -84.976...</td>\n",
       "      <td>[-64.50091552734375, -64.71295166015625, -64.8...</td>\n",
       "      <td>-84.965218</td>\n",
       "      <td>-64.651619</td>\n",
       "      <td>418.155731</td>\n",
       "      <td>1</td>\n",
       "      <td>2021010100043437</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   Latitude_vertices  \\\n",
       "0  [-84.71208953857422, -84.71642303466797, -84.7...   \n",
       "1  [-84.9544906616211, -84.9550552368164, -84.976...   \n",
       "\n",
       "                                  Longitude_vertices   Latitude  Longitude  \\\n",
       "0  [-51.488616943359375, -51.684295654296875, -51... -84.725594 -51.603241   \n",
       "1  [-64.50091552734375, -64.71295166015625, -64.8... -84.965218 -64.651619   \n",
       "\n",
       "         Xco2  quality_flag          DateTime  \n",
       "0  409.039856             1  2021010100041302  \n",
       "1  418.155731             1  2021010100043437  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "df_parq= pd.read_parquet(\"oco2_2021_parq.parquet\", engine=\"pyarrow\")\n",
    "df_parq.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ed9118d",
   "metadata": {},
   "source": [
    "## Quality Flag filtering:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "641e0bba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 8.8 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "18"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "df_qual_= df_parq[df_parq[\"quality_flag\"]==0]\n",
    "del df_parq\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c4ec16ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33052890"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#len(df_parq), \n",
    "len(df_qual_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a9c0d46",
   "metadata": {},
   "source": [
    "# Convert the DATETIME format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3b2c7446",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 7min 20s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_qual_[\"DateTime\"]= pd.to_datetime(df_qual_[\"DateTime\"], format=\"%Y%m%d%H%M%S%f\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0f47ee1e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Latitude_vertices</th>\n",
       "      <th>Longitude_vertices</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Xco2</th>\n",
       "      <th>quality_flag</th>\n",
       "      <th>DateTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[-63.28132247924805, -63.26362991333008, -63.2...</td>\n",
       "      <td>[-147.940185546875, -147.949951171875, -147.98...</td>\n",
       "      <td>-63.273125</td>\n",
       "      <td>-147.962006</td>\n",
       "      <td>409.814789</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-01-01 00:12:44.060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[-63.28230285644531, -63.26461410522461, -63.2...</td>\n",
       "      <td>[-147.97442626953125, -147.98416137695312, -14...</td>\n",
       "      <td>-63.274101</td>\n",
       "      <td>-147.996048</td>\n",
       "      <td>409.309540</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-01-01 00:12:44.070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[-63.28309631347656, -63.26540756225586, -63.2...</td>\n",
       "      <td>[-148.0076904296875, -148.01739501953125, -148...</td>\n",
       "      <td>-63.274967</td>\n",
       "      <td>-148.030212</td>\n",
       "      <td>409.366913</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-01-01 00:12:44.080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[-63.2584342956543, -63.2407341003418, -63.242...</td>\n",
       "      <td>[-147.8147735595703, -147.82464599609375, -147...</td>\n",
       "      <td>-63.250282</td>\n",
       "      <td>-147.836777</td>\n",
       "      <td>410.673309</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-01-01 00:12:44.320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[-63.25991439819336, -63.242218017578125, -63....</td>\n",
       "      <td>[-147.84823608398438, -147.8580780029297, -147...</td>\n",
       "      <td>-63.251755</td>\n",
       "      <td>-147.870346</td>\n",
       "      <td>409.633270</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-01-01 00:12:44.330</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33052885</th>\n",
       "      <td>[-5.621041774749756, -5.602428913116455, -5.61...</td>\n",
       "      <td>[-156.93154907226562, -156.9356689453125, -156...</td>\n",
       "      <td>-5.617089</td>\n",
       "      <td>-156.932739</td>\n",
       "      <td>415.088226</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-12-31 23:59:52.370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33052886</th>\n",
       "      <td>[-5.631556034088135, -5.61294412612915, -5.624...</td>\n",
       "      <td>[-156.93019104003906, -156.934326171875, -156....</td>\n",
       "      <td>-5.627943</td>\n",
       "      <td>-156.931335</td>\n",
       "      <td>414.612823</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-12-31 23:59:52.380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33052887</th>\n",
       "      <td>[-5.612905979156494, -5.5942816734313965, -5.6...</td>\n",
       "      <td>[-156.934326171875, -156.93846130371094, -156....</td>\n",
       "      <td>-5.609292</td>\n",
       "      <td>-156.935471</td>\n",
       "      <td>414.930511</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-12-31 23:59:52.780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33052888</th>\n",
       "      <td>[-5.5837321281433105, -5.565078258514404, -5.5...</td>\n",
       "      <td>[-156.9398193359375, -156.94398498535156, -156...</td>\n",
       "      <td>-5.579757</td>\n",
       "      <td>-156.941040</td>\n",
       "      <td>413.658966</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-12-31 23:59:53.070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33052889</th>\n",
       "      <td>[-5.575551509857178, -5.55689811706543, -5.568...</td>\n",
       "      <td>[-156.942626953125, -156.94677734375, -156.944...</td>\n",
       "      <td>-5.571915</td>\n",
       "      <td>-156.943771</td>\n",
       "      <td>413.477753</td>\n",
       "      <td>0</td>\n",
       "      <td>2021-12-31 23:59:53.380</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>33052890 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          Latitude_vertices  \\\n",
       "0         [-63.28132247924805, -63.26362991333008, -63.2...   \n",
       "1         [-63.28230285644531, -63.26461410522461, -63.2...   \n",
       "2         [-63.28309631347656, -63.26540756225586, -63.2...   \n",
       "3         [-63.2584342956543, -63.2407341003418, -63.242...   \n",
       "4         [-63.25991439819336, -63.242218017578125, -63....   \n",
       "...                                                     ...   \n",
       "33052885  [-5.621041774749756, -5.602428913116455, -5.61...   \n",
       "33052886  [-5.631556034088135, -5.61294412612915, -5.624...   \n",
       "33052887  [-5.612905979156494, -5.5942816734313965, -5.6...   \n",
       "33052888  [-5.5837321281433105, -5.565078258514404, -5.5...   \n",
       "33052889  [-5.575551509857178, -5.55689811706543, -5.568...   \n",
       "\n",
       "                                         Longitude_vertices   Latitude  \\\n",
       "0         [-147.940185546875, -147.949951171875, -147.98... -63.273125   \n",
       "1         [-147.97442626953125, -147.98416137695312, -14... -63.274101   \n",
       "2         [-148.0076904296875, -148.01739501953125, -148... -63.274967   \n",
       "3         [-147.8147735595703, -147.82464599609375, -147... -63.250282   \n",
       "4         [-147.84823608398438, -147.8580780029297, -147... -63.251755   \n",
       "...                                                     ...        ...   \n",
       "33052885  [-156.93154907226562, -156.9356689453125, -156...  -5.617089   \n",
       "33052886  [-156.93019104003906, -156.934326171875, -156....  -5.627943   \n",
       "33052887  [-156.934326171875, -156.93846130371094, -156....  -5.609292   \n",
       "33052888  [-156.9398193359375, -156.94398498535156, -156...  -5.579757   \n",
       "33052889  [-156.942626953125, -156.94677734375, -156.944...  -5.571915   \n",
       "\n",
       "           Longitude        Xco2  quality_flag                DateTime  \n",
       "0        -147.962006  409.814789             0 2021-01-01 00:12:44.060  \n",
       "1        -147.996048  409.309540             0 2021-01-01 00:12:44.070  \n",
       "2        -148.030212  409.366913             0 2021-01-01 00:12:44.080  \n",
       "3        -147.836777  410.673309             0 2021-01-01 00:12:44.320  \n",
       "4        -147.870346  409.633270             0 2021-01-01 00:12:44.330  \n",
       "...              ...         ...           ...                     ...  \n",
       "33052885 -156.932739  415.088226             0 2021-12-31 23:59:52.370  \n",
       "33052886 -156.931335  414.612823             0 2021-12-31 23:59:52.380  \n",
       "33052887 -156.935471  414.930511             0 2021-12-31 23:59:52.780  \n",
       "33052888 -156.941040  413.658966             0 2021-12-31 23:59:53.070  \n",
       "33052889 -156.943771  413.477753             0 2021-12-31 23:59:53.380  \n",
       "\n",
       "[33052890 rows x 7 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_qual_= df_qual_.reset_index()\n",
    "df_qual_=df_qual_.drop(columns=[\"index\"])\n",
    "df_qual_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "599c7870",
   "metadata": {},
   "source": [
    "## Get month column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9086d0db",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_qual_[\"Month\"]= df_qual_[\"DateTime\"].dt.month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1fcc75e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12], dtype=int64)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_qual_[\"Month\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f73bea82",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2min 14s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "df_qual_.to_parquet(\"oco2_2021_parq_dateTime.parquet\", engine=\"pyarrow\")\n",
    "\n",
    "del df_qual_\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1e4f64a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}