notebooks/Formats/LoadSaveParquetFile.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![lop](../../images/logo_diive1_128px.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style='font-size:32px; display:block;'>\n",
    "<b>\n",
    "    Load and save Parquet file\n",
    "</b>\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Notebook version**: `2` (24 Oct 2023)  \n",
    "**Author**: Lukas Hörtnagl (holukas@ethz.ch)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Background**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk. Parquet is available in multiple languages including Java, C++, Python, etc...\n",
    "\n",
    "source: https://parquet.apache.org/\n",
    "\n",
    "- Parquet files are much smaller than e.g. CSV files, and faster to load and save"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Imports**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "diive version: v0.74.0\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import importlib.metadata\n",
    "from datetime import datetime\n",
    "from pathlib import Path\n",
    "from diive.core.io.files import save_parquet, load_parquet\n",
    "from diive.core.io.filereader import MultiDataFileReader\n",
    "\n",
    "version_diive = importlib.metadata.version(\"diive\")\n",
    "print(f\"diive version: v{version_diive}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Docstring** of `save_parquet`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function save_parquet in module diive.core.io.files:\n",
      "\n",
      "save_parquet(filename: str, data: pandas.core.frame.DataFrame, outpath: str = None) -> str\n",
      "    Save pandas Series or DataFrame as parquet file\n",
      "    \n",
      "    Args:\n",
      "        filename: str\n",
      "            Name of the generated parquet file.\n",
      "        data: pandas Series or DataFrame\n",
      "        outpath: str or None\n",
      "            If *None*, file is saved to system default folder. When used within\n",
      "            a notebook, the file is saved in the same location as the notebook.\n",
      "    \n",
      "    Returns:\n",
      "        str, filepath to parquet file\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(save_parquet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Docstring** of `load_parquet`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function load_parquet in module diive.core.io.files:\n",
      "\n",
      "load_parquet(filepath: str) -> pandas.core.frame.DataFrame\n",
      "    Load data from Parquet file to pandas DataFrame\n",
      "    \n",
      "    Args:\n",
      "        filepath: str\n",
      "            filepath to parquet file\n",
      "    \n",
      "    Returns:\n",
      "        pandas DataFrame, data from Parquet file as pandas DataFrame\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(load_parquet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Input data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Source and output folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DIR = Path(r'F:\\Sync\\luhk_work\\TMP\\FRU')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search files in folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F:\\Sync\\luhk_work\\TMP\\FRU\\2020_eddypro_CH-FRU_FR-20230425-205325_fluxnet_2023-04-26T220030_adv.csv\n",
      "F:\\Sync\\luhk_work\\TMP\\FRU\\2021_eddypro_CH-FRU_FR-20230425-205131_fluxnet_2023-04-26T214014_adv.csv\n",
      "F:\\Sync\\luhk_work\\TMP\\FRU\\2022_eddypro_CH-FRU_FR-20230425-205042_fluxnet_2023-04-26T222351_adv.csv\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None, None]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepaths = [f for f in os.listdir(DIR) if f.endswith(\".csv\")]\n",
    "filepaths = [DIR / f for f in filepaths]\n",
    "filepaths = [Path(f) for f in filepaths]\n",
    "filepaths = [f for f in filepaths if 'fluxnet' in str(f)]\n",
    "[print(f) for f in filepaths]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and merge data from files\n",
    "- Read datafiles in `filepaths` using `diive`\n",
    "- All datafiles are in the `EDDYPRO_FLUXNET_30MIN` format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading file 2020_eddypro_CH-FRU_FR-20230425-205325_fluxnet_2023-04-26T220030_adv.csv ...\n",
      "Reading file 2021_eddypro_CH-FRU_FR-20230425-205131_fluxnet_2023-04-26T214014_adv.csv ...\n",
      "Reading file 2022_eddypro_CH-FRU_FR-20230425-205042_fluxnet_2023-04-26T222351_adv.csv ...\n"
     ]
    }
   ],
   "source": [
    "loaddatafile = MultiDataFileReader(filetype='EDDYPRO-FLUXNET-CSV-30MIN', filepaths=filepaths)\n",
    "df = loaddatafile.data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AIR_MV</th>\n",
       "      <th>AIR_DENSITY</th>\n",
       "      <th>AIR_RHO_CP</th>\n",
       "      <th>AIR_CP</th>\n",
       "      <th>AOA_METHOD</th>\n",
       "      <th>AXES_ROTATION_METHOD</th>\n",
       "      <th>...</th>\n",
       "      <th>W_ITC</th>\n",
       "      <th>W_ITC_TEST</th>\n",
       "      <th>WBOOST_APPLIED</th>\n",
       "      <th>WPL_APPLIED</th>\n",
       "      <th>ZL</th>\n",
       "      <th>ZL_UNCORR</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIMESTAMP_MIDDLE</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-12-31 19:15:00</th>\n",
       "      <td>0.024907</td>\n",
       "      <td>1.16085</td>\n",
       "      <td>1169.83</td>\n",
       "      <td>1007.74</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.079130</td>\n",
       "      <td>3.006370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 19:45:00</th>\n",
       "      <td>0.024865</td>\n",
       "      <td>1.16271</td>\n",
       "      <td>1171.77</td>\n",
       "      <td>1007.79</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>47.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.096300</td>\n",
       "      <td>-0.099549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 20:15:00</th>\n",
       "      <td>0.024910</td>\n",
       "      <td>1.16061</td>\n",
       "      <td>1169.68</td>\n",
       "      <td>1007.81</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>27.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.780695</td>\n",
       "      <td>-0.771121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 20:45:00</th>\n",
       "      <td>0.024886</td>\n",
       "      <td>1.16183</td>\n",
       "      <td>1170.78</td>\n",
       "      <td>1007.70</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>21.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.540130</td>\n",
       "      <td>0.545173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 21:15:00</th>\n",
       "      <td>0.024874</td>\n",
       "      <td>1.16239</td>\n",
       "      <td>1171.37</td>\n",
       "      <td>1007.73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.135396</td>\n",
       "      <td>-0.095764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-12-31 22:45:00</th>\n",
       "      <td>0.025906</td>\n",
       "      <td>1.11442</td>\n",
       "      <td>1125.59</td>\n",
       "      <td>1010.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.316949</td>\n",
       "      <td>0.309181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-12-31 23:15:00</th>\n",
       "      <td>0.025770</td>\n",
       "      <td>1.12033</td>\n",
       "      <td>1131.50</td>\n",
       "      <td>1009.97</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>60.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.057874</td>\n",
       "      <td>-0.056127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-12-31 23:45:00</th>\n",
       "      <td>0.025757</td>\n",
       "      <td>1.12077</td>\n",
       "      <td>1132.08</td>\n",
       "      <td>1010.09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.716370</td>\n",
       "      <td>1.677440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-01-01 00:15:00</th>\n",
       "      <td>0.026678</td>\n",
       "      <td>1.08509</td>\n",
       "      <td>1096.57</td>\n",
       "      <td>1010.58</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.378159</td>\n",
       "      <td>0.374705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-01-01 00:45:00</th>\n",
       "      <td>0.026690</td>\n",
       "      <td>1.08463</td>\n",
       "      <td>1096.02</td>\n",
       "      <td>1010.50</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>34.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.257230</td>\n",
       "      <td>1.219240</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>52620 rows × 485 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       AIR_MV  AIR_DENSITY  AIR_RHO_CP   AIR_CP  AOA_METHOD  AXES_ROTATION_METHOD  ...  W_ITC  W_ITC_TEST  WBOOST_APPLIED  WPL_APPLIED        ZL  ZL_UNCORR\n",
       "TIMESTAMP_MIDDLE                                                                                   ...                                                                     \n",
       "2019-12-31 19:15:00  0.024907      1.16085     1169.83  1007.74         0.0                   1.0  ...    6.0         1.0             0.0          1.0  3.079130   3.006370\n",
       "2019-12-31 19:45:00  0.024865      1.16271     1171.77  1007.79         0.0                   1.0  ...   47.0         3.0             0.0          1.0 -0.096300  -0.099549\n",
       "2019-12-31 20:15:00  0.024910      1.16061     1169.68  1007.81         0.0                   1.0  ...   27.0         2.0             0.0          1.0 -0.780695  -0.771121\n",
       "2019-12-31 20:45:00  0.024886      1.16183     1170.78  1007.70         0.0                   1.0  ...   21.0         2.0             0.0          1.0  0.540130   0.545173\n",
       "2019-12-31 21:15:00  0.024874      1.16239     1171.37  1007.73         0.0                   1.0  ...    9.0         1.0             0.0          1.0 -0.135396  -0.095764\n",
       "...                       ...          ...         ...      ...         ...                   ...  ...    ...         ...             ...          ...       ...        ...\n",
       "2022-12-31 22:45:00  0.025906      1.11442     1125.59  1010.02         0.0                   1.0  ...    9.0         1.0             0.0          1.0  0.316949   0.309181\n",
       "2022-12-31 23:15:00  0.025770      1.12033     1131.50  1009.97         0.0                   1.0  ...   60.0         4.0             0.0          1.0 -0.057874  -0.056127\n",
       "2022-12-31 23:45:00  0.025757      1.12077     1132.08  1010.09         0.0                   1.0  ...    8.0         1.0             0.0          1.0  1.716370   1.677440\n",
       "2023-01-01 00:15:00  0.026678      1.08509     1096.57  1010.58         0.0                   1.0  ...   25.0         2.0             0.0          1.0  0.378159   0.374705\n",
       "2023-01-01 00:45:00  0.026690      1.08463     1096.02  1010.50         0.0                   1.0  ...   34.0         3.0             0.0          1.0  1.257230   1.219240\n",
       "\n",
       "[52620 rows x 485 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Save dataframe as parquet file**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (1) Save to specific folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved file F:\\Sync\\luhk_work\\TMP\\FRU\\CH-DAV_FP2022.5_1997-2022_ID20230206154316_30MIN.parquet (0.953 seconds).\n"
     ]
    }
   ],
   "source": [
    "filepath = save_parquet(outpath=DIR, filename=\"CH-DAV_FP2022.5_1997-2022_ID20230206154316_30MIN\", data=df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2) Save to same folder as notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# filepath = save_parquet(filename=\"data\", data=df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filepath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'F:\\\\Sync\\\\luhk_work\\\\TMP\\\\FRU\\\\CH-DAV_FP2022.5_1997-2022_ID20230206154316_30MIN.parquet'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepath"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Load data from parquet file**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded .parquet file F:\\Sync\\luhk_work\\TMP\\FRU\\CH-DAV_FP2022.5_1997-2022_ID20230206154316_30MIN.parquet (0.127 seconds). Detected time resolution of <30 * Minutes> / 30min \n"
     ]
    }
   ],
   "source": [
    "data_from_parquet = load_parquet(filepath=filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AIR_MV</th>\n",
       "      <th>AIR_DENSITY</th>\n",
       "      <th>AIR_RHO_CP</th>\n",
       "      <th>AIR_CP</th>\n",
       "      <th>AOA_METHOD</th>\n",
       "      <th>AXES_ROTATION_METHOD</th>\n",
       "      <th>...</th>\n",
       "      <th>W_ITC</th>\n",
       "      <th>W_ITC_TEST</th>\n",
       "      <th>WBOOST_APPLIED</th>\n",
       "      <th>WPL_APPLIED</th>\n",
       "      <th>ZL</th>\n",
       "      <th>ZL_UNCORR</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIMESTAMP_MIDDLE</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-12-31 19:15:00</th>\n",
       "      <td>0.024907</td>\n",
       "      <td>1.16085</td>\n",
       "      <td>1169.83</td>\n",
       "      <td>1007.74</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.079130</td>\n",
       "      <td>3.006370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 19:45:00</th>\n",
       "      <td>0.024865</td>\n",
       "      <td>1.16271</td>\n",
       "      <td>1171.77</td>\n",
       "      <td>1007.79</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>47.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.096300</td>\n",
       "      <td>-0.099549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 20:15:00</th>\n",
       "      <td>0.024910</td>\n",
       "      <td>1.16061</td>\n",
       "      <td>1169.68</td>\n",
       "      <td>1007.81</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>27.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.780695</td>\n",
       "      <td>-0.771121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 20:45:00</th>\n",
       "      <td>0.024886</td>\n",
       "      <td>1.16183</td>\n",
       "      <td>1170.78</td>\n",
       "      <td>1007.70</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>21.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.540130</td>\n",
       "      <td>0.545173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 21:15:00</th>\n",
       "      <td>0.024874</td>\n",
       "      <td>1.16239</td>\n",
       "      <td>1171.37</td>\n",
       "      <td>1007.73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.135396</td>\n",
       "      <td>-0.095764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-12-31 22:45:00</th>\n",
       "      <td>0.025906</td>\n",
       "      <td>1.11442</td>\n",
       "      <td>1125.59</td>\n",
       "      <td>1010.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.316949</td>\n",
       "      <td>0.309181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-12-31 23:15:00</th>\n",
       "      <td>0.025770</td>\n",
       "      <td>1.12033</td>\n",
       "      <td>1131.50</td>\n",
       "      <td>1009.97</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>60.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.057874</td>\n",
       "      <td>-0.056127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-12-31 23:45:00</th>\n",
       "      <td>0.025757</td>\n",
       "      <td>1.12077</td>\n",
       "      <td>1132.08</td>\n",
       "      <td>1010.09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.716370</td>\n",
       "      <td>1.677440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-01-01 00:15:00</th>\n",
       "      <td>0.026678</td>\n",
       "      <td>1.08509</td>\n",
       "      <td>1096.57</td>\n",
       "      <td>1010.58</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.378159</td>\n",
       "      <td>0.374705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-01-01 00:45:00</th>\n",
       "      <td>0.026690</td>\n",
       "      <td>1.08463</td>\n",
       "      <td>1096.02</td>\n",
       "      <td>1010.50</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>34.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.257230</td>\n",
       "      <td>1.219240</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>52620 rows × 485 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       AIR_MV  AIR_DENSITY  AIR_RHO_CP   AIR_CP  AOA_METHOD  AXES_ROTATION_METHOD  ...  W_ITC  W_ITC_TEST  WBOOST_APPLIED  WPL_APPLIED        ZL  ZL_UNCORR\n",
       "TIMESTAMP_MIDDLE                                                                                   ...                                                                     \n",
       "2019-12-31 19:15:00  0.024907      1.16085     1169.83  1007.74         0.0                   1.0  ...    6.0         1.0             0.0          1.0  3.079130   3.006370\n",
       "2019-12-31 19:45:00  0.024865      1.16271     1171.77  1007.79         0.0                   1.0  ...   47.0         3.0             0.0          1.0 -0.096300  -0.099549\n",
       "2019-12-31 20:15:00  0.024910      1.16061     1169.68  1007.81         0.0                   1.0  ...   27.0         2.0             0.0          1.0 -0.780695  -0.771121\n",
       "2019-12-31 20:45:00  0.024886      1.16183     1170.78  1007.70         0.0                   1.0  ...   21.0         2.0             0.0          1.0  0.540130   0.545173\n",
       "2019-12-31 21:15:00  0.024874      1.16239     1171.37  1007.73         0.0                   1.0  ...    9.0         1.0             0.0          1.0 -0.135396  -0.095764\n",
       "...                       ...          ...         ...      ...         ...                   ...  ...    ...         ...             ...          ...       ...        ...\n",
       "2022-12-31 22:45:00  0.025906      1.11442     1125.59  1010.02         0.0                   1.0  ...    9.0         1.0             0.0          1.0  0.316949   0.309181\n",
       "2022-12-31 23:15:00  0.025770      1.12033     1131.50  1009.97         0.0                   1.0  ...   60.0         4.0             0.0          1.0 -0.057874  -0.056127\n",
       "2022-12-31 23:45:00  0.025757      1.12077     1132.08  1010.09         0.0                   1.0  ...    8.0         1.0             0.0          1.0  1.716370   1.677440\n",
       "2023-01-01 00:15:00  0.026678      1.08509     1096.57  1010.58         0.0                   1.0  ...   25.0         2.0             0.0          1.0  0.378159   0.374705\n",
       "2023-01-01 00:45:00  0.026690      1.08463     1096.02  1010.50         0.0                   1.0  ...   34.0         3.0             0.0          1.0  1.257230   1.219240\n",
       "\n",
       "[52620 rows x 485 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_from_parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# **Finish**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-07-01T01:13:34.149472Z",
     "start_time": "2023-07-01T01:13:34.060892Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished 2024-04-20 19:05:25\n"
     ]
    }
   ],
   "source": [
    "dt_string = datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\")\n",
    "print(f\"Finished {dt_string}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}