tools_python/parquet_datasets/parquet_datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring parquet datasets\n",
    "\n",
    "Parquet files are a columinar data format we can use to store dataframes. They can be stored in partitions, which can allow us to load only a subset of the data. This is useful is we are filtering the data, as we can do that without loading it all into memory.\n",
    "\n",
    "Import stuff:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.style.use(\"seaborn-whitegrid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "Here we use the UCI heart disease data set found on Kaggle:  \n",
    "`https://www.kaggle.com/ronitf/heart-disease-uci`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n0   29    1   1       130   204    0        0      202      0      0.0      2   \n1   34    0   1       118   210    0        1      192      0      0.7      2   \n2   34    1   3       118   182    0        0      174      0      0.0      2   \n3   35    0   0       138   183    0        1      182      0      1.4      2   \n4   35    1   0       120   198    0        1      130      1      1.6      1   \n\n   ca  thal  target  \n0   0     2       1  \n1   0     2       1  \n2   0     2       1  \n3   0     2       1  \n4   0     3       0  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>sex</th>\n      <th>cp</th>\n      <th>trestbps</th>\n      <th>chol</th>\n      <th>fbs</th>\n      <th>restecg</th>\n      <th>thalach</th>\n      <th>exang</th>\n      <th>oldpeak</th>\n      <th>slope</th>\n      <th>ca</th>\n      <th>thal</th>\n      <th>target</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>29</td>\n      <td>1</td>\n      <td>1</td>\n      <td>130</td>\n      <td>204</td>\n      <td>0</td>\n      <td>0</td>\n      <td>202</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>2</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>34</td>\n      <td>0</td>\n      <td>1</td>\n      <td>118</td>\n      <td>210</td>\n      <td>0</td>\n      <td>1</td>\n      <td>192</td>\n      <td>0</td>\n      <td>0.7</td>\n      <td>2</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>34</td>\n      <td>1</td>\n      <td>3</td>\n      <td>118</td>\n      <td>182</td>\n      <td>0</td>\n      <td>0</td>\n      <td>174</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>2</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>35</td>\n      <td>0</td>\n      <td>0</td>\n      <td>138</td>\n      <td>183</td>\n      <td>0</td>\n      <td>1</td>\n      <td>182</td>\n      <td>0</td>\n      <td>1.4</td>\n      <td>2</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>35</td>\n      <td>1</td>\n      <td>0</td>\n      <td>120</td>\n      <td>198</td>\n      <td>0</td>\n      <td>1</td>\n      <td>130</td>\n      <td>1</td>\n      <td>1.6</td>\n      <td>1</td>\n      <td>0</td>\n      <td>3</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "source": [
    "dir_path = os.getcwd()\n",
    "heart_data = pd.read_csv(os.path.join(dir_path, \"data\", \"heart.csv\"))\n",
    "heart_data.sort_values(by=[\"age\", \"sex\", \"cp\"], inplace=True)\n",
    "heart_data.reset_index(inplace=True, drop=True)\n",
    "heart_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Partition data\n",
    "To split the data we need an appropriate column to split by. Based on it's histogram, we can use the age column to divide the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"257.747813pt\" version=\"1.1\" viewBox=\"0 0 377.084375 257.747813\" width=\"377.084375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 257.747813 \nL 377.084375 257.747813 \nL 377.084375 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 35.084375 224.64 \nL 369.884375 224.64 \nL 369.884375 7.2 \nL 35.084375 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 56.643466 224.64 \nL 56.643466 7.2 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_2\"/>\n     <g id=\"text_1\">\n      <!-- 30 -->\n      <defs>\n       <path d=\"M 4.203125 18.890625 \nL 12.984375 20.0625 \nQ 14.5 12.59375 18.140625 9.296875 \nQ 21.78125 6 27 6 \nQ 33.203125 6 37.46875 10.296875 \nQ 41.75 14.59375 41.75 20.953125 \nQ 41.75 27 37.796875 30.921875 \nQ 33.84375 34.859375 27.734375 34.859375 \nQ 25.25 34.859375 21.53125 33.890625 \nL 22.515625 41.609375 \nQ 23.390625 41.5 23.921875 41.5 \nQ 29.546875 41.5 34.03125 44.421875 \nQ 38.53125 47.359375 38.53125 53.46875 \nQ 38.53125 58.296875 35.25 61.46875 \nQ 31.984375 64.65625 26.8125 64.65625 \nQ 21.6875 64.65625 18.265625 61.421875 \nQ 14.84375 58.203125 13.875 51.765625 \nL 5.078125 53.328125 \nQ 6.6875 62.15625 12.390625 67.015625 \nQ 18.109375 71.875 26.609375 71.875 \nQ 32.46875 71.875 37.390625 69.359375 \nQ 42.328125 66.84375 44.9375 62.5 \nQ 47.5625 58.15625 47.5625 53.265625 \nQ 47.5625 48.640625 45.0625 44.828125 \nQ 42.578125 41.015625 37.703125 38.765625 \nQ 44.046875 37.3125 47.5625 32.6875 \nQ 51.078125 28.078125 51.078125 21.140625 \nQ 51.078125 11.765625 44.234375 5.25 \nQ 37.40625 -1.265625 26.953125 -1.265625 \nQ 17.53125 -1.265625 11.296875 4.34375 \nQ 5.078125 9.96875 4.203125 18.890625 \nz\n\" id=\"ArialMT-51\"/>\n       <path d=\"M 4.15625 35.296875 \nQ 4.15625 48 6.765625 55.734375 \nQ 9.375 63.484375 14.515625 67.671875 \nQ 19.671875 71.875 27.484375 71.875 \nQ 33.25 71.875 37.59375 69.546875 \nQ 41.9375 67.234375 44.765625 62.859375 \nQ 47.609375 58.5 49.21875 52.21875 \nQ 50.828125 45.953125 50.828125 35.296875 \nQ 50.828125 22.703125 48.234375 14.96875 \nQ 45.65625 7.234375 40.5 3 \nQ 35.359375 -1.21875 27.484375 -1.21875 \nQ 17.140625 -1.21875 11.234375 6.203125 \nQ 4.15625 15.140625 4.15625 35.296875 \nz\nM 13.1875 35.296875 \nQ 13.1875 17.671875 17.3125 11.828125 \nQ 21.4375 6 27.484375 6 \nQ 33.546875 6 37.671875 11.859375 \nQ 41.796875 17.71875 41.796875 35.296875 \nQ 41.796875 52.984375 37.671875 58.78125 \nQ 33.546875 64.59375 27.390625 64.59375 \nQ 21.34375 64.59375 17.71875 59.46875 \nQ 13.1875 52.9375 13.1875 35.296875 \nz\n\" id=\"ArialMT-48\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(51.082528 235.297813)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-51\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 120.052557 224.64 \nL 120.052557 7.2 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_4\"/>\n     <g id=\"text_2\">\n      <!-- 40 -->\n      <defs>\n       <path d=\"M 32.328125 0 \nL 32.328125 17.140625 \nL 1.265625 17.140625 \nL 1.265625 25.203125 \nL 33.9375 71.578125 \nL 41.109375 71.578125 \nL 41.109375 25.203125 \nL 50.78125 25.203125 \nL 50.78125 17.140625 \nL 41.109375 17.140625 \nL 41.109375 0 \nz\nM 32.328125 25.203125 \nL 32.328125 57.46875 \nL 9.90625 25.203125 \nz\n\" id=\"ArialMT-52\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(114.491619 235.297813)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-52\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 183.461648 224.64 \nL 183.461648 7.2 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_6\"/>\n     <g id=\"text_3\">\n      <!-- 50 -->\n      <defs>\n       <path d=\"M 4.15625 18.75 \nL 13.375 19.53125 \nQ 14.40625 12.796875 18.140625 9.390625 \nQ 21.875 6 27.15625 6 \nQ 33.5 6 37.890625 10.78125 \nQ 42.28125 15.578125 42.28125 23.484375 \nQ 42.28125 31 38.0625 35.34375 \nQ 33.84375 39.703125 27 39.703125 \nQ 22.75 39.703125 19.328125 37.765625 \nQ 15.921875 35.84375 13.96875 32.765625 \nL 5.71875 33.84375 \nL 12.640625 70.609375 \nL 48.25 70.609375 \nL 48.25 62.203125 \nL 19.671875 62.203125 \nL 15.828125 42.96875 \nQ 22.265625 47.46875 29.34375 47.46875 \nQ 38.71875 47.46875 45.15625 40.96875 \nQ 51.609375 34.46875 51.609375 24.265625 \nQ 51.609375 14.546875 45.953125 7.46875 \nQ 39.0625 -1.21875 27.15625 -1.21875 \nQ 17.390625 -1.21875 11.203125 4.25 \nQ 5.03125 9.71875 4.15625 18.75 \nz\n\" id=\"ArialMT-53\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(177.90071 235.297813)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-53\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 246.870739 224.64 \nL 246.870739 7.2 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_8\"/>\n     <g id=\"text_4\">\n      <!-- 60 -->\n      <defs>\n       <path d=\"M 49.75 54.046875 \nL 41.015625 53.375 \nQ 39.84375 58.546875 37.703125 60.890625 \nQ 34.125 64.65625 28.90625 64.65625 \nQ 24.703125 64.65625 21.53125 62.3125 \nQ 17.390625 59.28125 14.984375 53.46875 \nQ 12.59375 47.65625 12.5 36.921875 \nQ 15.671875 41.75 20.265625 44.09375 \nQ 24.859375 46.4375 29.890625 46.4375 \nQ 38.671875 46.4375 44.84375 39.96875 \nQ 51.03125 33.5 51.03125 23.25 \nQ 51.03125 16.5 48.125 10.71875 \nQ 45.21875 4.9375 40.140625 1.859375 \nQ 35.0625 -1.21875 28.609375 -1.21875 \nQ 17.625 -1.21875 10.6875 6.859375 \nQ 3.765625 14.9375 3.765625 33.5 \nQ 3.765625 54.25 11.421875 63.671875 \nQ 18.109375 71.875 29.4375 71.875 \nQ 37.890625 71.875 43.28125 67.140625 \nQ 48.6875 62.40625 49.75 54.046875 \nz\nM 13.875 23.1875 \nQ 13.875 18.65625 15.796875 14.5 \nQ 17.71875 10.359375 21.1875 8.171875 \nQ 24.65625 6 28.46875 6 \nQ 34.03125 6 38.03125 10.484375 \nQ 42.046875 14.984375 42.046875 22.703125 \nQ 42.046875 30.125 38.078125 34.390625 \nQ 34.125 38.671875 28.125 38.671875 \nQ 22.171875 38.671875 18.015625 34.390625 \nQ 13.875 30.125 13.875 23.1875 \nz\n\" id=\"ArialMT-54\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(241.309801 235.297813)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-54\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 310.27983 224.64 \nL 310.27983 7.2 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_10\"/>\n     <g id=\"text_5\">\n      <!-- 70 -->\n      <defs>\n       <path d=\"M 4.734375 62.203125 \nL 4.734375 70.65625 \nL 51.078125 70.65625 \nL 51.078125 63.8125 \nQ 44.234375 56.546875 37.515625 44.484375 \nQ 30.8125 32.421875 27.15625 19.671875 \nQ 24.515625 10.6875 23.78125 0 \nL 14.75 0 \nQ 14.890625 8.453125 18.0625 20.40625 \nQ 21.234375 32.375 27.171875 43.484375 \nQ 33.109375 54.59375 39.796875 62.203125 \nz\n\" id=\"ArialMT-55\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(304.718892 235.297813)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-55\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_6\">\n     <!-- Age -->\n     <defs>\n      <path d=\"M -0.140625 0 \nL 27.34375 71.578125 \nL 37.546875 71.578125 \nL 66.84375 0 \nL 56.0625 0 \nL 47.703125 21.6875 \nL 17.78125 21.6875 \nL 9.90625 0 \nz\nM 20.515625 29.390625 \nL 44.78125 29.390625 \nL 37.3125 49.21875 \nQ 33.890625 58.25 32.234375 64.0625 \nQ 30.859375 57.171875 28.375 50.390625 \nz\n\" id=\"ArialMT-65\"/>\n      <path d=\"M 4.984375 -4.296875 \nL 13.53125 -5.5625 \nQ 14.0625 -9.515625 16.5 -11.328125 \nQ 19.78125 -13.765625 25.4375 -13.765625 \nQ 31.546875 -13.765625 34.859375 -11.328125 \nQ 38.1875 -8.890625 39.359375 -4.5 \nQ 40.046875 -1.8125 39.984375 6.78125 \nQ 34.234375 0 25.640625 0 \nQ 14.9375 0 9.078125 7.71875 \nQ 3.21875 15.4375 3.21875 26.21875 \nQ 3.21875 33.640625 5.90625 39.90625 \nQ 8.59375 46.1875 13.6875 49.609375 \nQ 18.796875 53.03125 25.6875 53.03125 \nQ 34.859375 53.03125 40.828125 45.609375 \nL 40.828125 51.859375 \nL 48.921875 51.859375 \nL 48.921875 7.03125 \nQ 48.921875 -5.078125 46.453125 -10.125 \nQ 44 -15.1875 38.640625 -18.109375 \nQ 33.296875 -21.046875 25.484375 -21.046875 \nQ 16.21875 -21.046875 10.5 -16.875 \nQ 4.78125 -12.703125 4.984375 -4.296875 \nz\nM 12.25 26.859375 \nQ 12.25 16.65625 16.296875 11.96875 \nQ 20.359375 7.28125 26.46875 7.28125 \nQ 32.515625 7.28125 36.609375 11.9375 \nQ 40.71875 16.609375 40.71875 26.5625 \nQ 40.71875 36.078125 36.5 40.90625 \nQ 32.28125 45.75 26.3125 45.75 \nQ 20.453125 45.75 16.34375 40.984375 \nQ 12.25 36.234375 12.25 26.859375 \nz\n\" id=\"ArialMT-103\"/>\n      <path d=\"M 42.09375 16.703125 \nL 51.171875 15.578125 \nQ 49.03125 7.625 43.21875 3.21875 \nQ 37.40625 -1.171875 28.375 -1.171875 \nQ 17 -1.171875 10.328125 5.828125 \nQ 3.65625 12.84375 3.65625 25.484375 \nQ 3.65625 38.578125 10.390625 45.796875 \nQ 17.140625 53.03125 27.875 53.03125 \nQ 38.28125 53.03125 44.875 45.953125 \nQ 51.46875 38.875 51.46875 26.03125 \nQ 51.46875 25.25 51.421875 23.6875 \nL 12.75 23.6875 \nQ 13.234375 15.140625 17.578125 10.59375 \nQ 21.921875 6.0625 28.421875 6.0625 \nQ 33.25 6.0625 36.671875 8.59375 \nQ 40.09375 11.140625 42.09375 16.703125 \nz\nM 13.234375 30.90625 \nL 42.1875 30.90625 \nQ 41.609375 37.453125 38.875 40.71875 \nQ 34.671875 45.796875 27.984375 45.796875 \nQ 21.921875 45.796875 17.796875 41.75 \nQ 13.671875 37.703125 13.234375 30.90625 \nz\n\" id=\"ArialMT-101\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(193.588281 248.443125)scale(0.1 -0.1)\">\n      <use xlink:href=\"#ArialMT-65\"/>\n      <use x=\"66.699219\" xlink:href=\"#ArialMT-103\"/>\n      <use x=\"122.314453\" xlink:href=\"#ArialMT-101\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_11\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 35.084375 224.64 \nL 369.884375 224.64 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_12\"/>\n     <g id=\"text_7\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(26.023438 228.218906)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_13\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 35.084375 191.769252 \nL 369.884375 191.769252 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_14\"/>\n     <g id=\"text_8\">\n      <!-- 10 -->\n      <defs>\n       <path d=\"M 37.25 0 \nL 28.46875 0 \nL 28.46875 56 \nQ 25.296875 52.984375 20.140625 49.953125 \nQ 14.984375 46.921875 10.890625 45.40625 \nL 10.890625 53.90625 \nQ 18.265625 57.375 23.78125 62.296875 \nQ 29.296875 67.234375 31.59375 71.875 \nL 37.25 71.875 \nz\n\" id=\"ArialMT-49\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(20.4625 195.348158)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-49\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_15\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 35.084375 158.898503 \nL 369.884375 158.898503 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_16\"/>\n     <g id=\"text_9\">\n      <!-- 20 -->\n      <defs>\n       <path d=\"M 50.34375 8.453125 \nL 50.34375 0 \nL 3.03125 0 \nQ 2.9375 3.171875 4.046875 6.109375 \nQ 5.859375 10.9375 9.828125 15.625 \nQ 13.8125 20.3125 21.34375 26.46875 \nQ 33.015625 36.03125 37.109375 41.625 \nQ 41.21875 47.21875 41.21875 52.203125 \nQ 41.21875 57.421875 37.46875 61 \nQ 33.734375 64.59375 27.734375 64.59375 \nQ 21.390625 64.59375 17.578125 60.78125 \nQ 13.765625 56.984375 13.71875 50.25 \nL 4.6875 51.171875 \nQ 5.609375 61.28125 11.65625 66.578125 \nQ 17.71875 71.875 27.9375 71.875 \nQ 38.234375 71.875 44.234375 66.15625 \nQ 50.25 60.453125 50.25 52 \nQ 50.25 47.703125 48.484375 43.546875 \nQ 46.734375 39.40625 42.65625 34.8125 \nQ 38.578125 30.21875 29.109375 22.21875 \nQ 21.1875 15.578125 18.9375 13.203125 \nQ 16.703125 10.84375 15.234375 8.453125 \nz\n\" id=\"ArialMT-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(20.4625 162.47741)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-50\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_17\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 35.084375 126.027755 \nL 369.884375 126.027755 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_18\"/>\n     <g id=\"text_10\">\n      <!-- 30 -->\n      <g style=\"fill:#262626;\" transform=\"translate(20.4625 129.606661)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-51\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_19\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 35.084375 93.157007 \nL 369.884375 93.157007 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_20\"/>\n     <g id=\"text_11\">\n      <!-- 40 -->\n      <g style=\"fill:#262626;\" transform=\"translate(20.4625 96.735913)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-52\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 35.084375 60.286259 \nL 369.884375 60.286259 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_22\"/>\n     <g id=\"text_12\">\n      <!-- 50 -->\n      <g style=\"fill:#262626;\" transform=\"translate(20.4625 63.865165)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-53\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_23\">\n      <path clip-path=\"url(#pe629cf66bf)\" d=\"M 35.084375 27.41551 \nL 369.884375 27.41551 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:round;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_24\"/>\n     <g id=\"text_13\">\n      <!-- 60 -->\n      <g style=\"fill:#262626;\" transform=\"translate(20.4625 30.994416)scale(0.1 -0.1)\">\n       <use xlink:href=\"#ArialMT-54\"/>\n       <use x=\"55.615234\" xlink:href=\"#ArialMT-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_14\">\n     <!-- Frequency -->\n     <defs>\n      <path d=\"M 8.203125 0 \nL 8.203125 71.578125 \nL 56.5 71.578125 \nL 56.5 63.140625 \nL 17.671875 63.140625 \nL 17.671875 40.96875 \nL 51.265625 40.96875 \nL 51.265625 32.515625 \nL 17.671875 32.515625 \nL 17.671875 0 \nz\n\" id=\"ArialMT-70\"/>\n      <path d=\"M 6.5 0 \nL 6.5 51.859375 \nL 14.40625 51.859375 \nL 14.40625 44 \nQ 17.4375 49.515625 20 51.265625 \nQ 22.5625 53.03125 25.640625 53.03125 \nQ 30.078125 53.03125 34.671875 50.203125 \nL 31.640625 42.046875 \nQ 28.421875 43.953125 25.203125 43.953125 \nQ 22.3125 43.953125 20.015625 42.21875 \nQ 17.71875 40.484375 16.75 37.40625 \nQ 15.28125 32.71875 15.28125 27.15625 \nL 15.28125 0 \nz\n\" id=\"ArialMT-114\"/>\n      <path d=\"M 39.65625 -19.875 \nL 39.65625 5.515625 \nQ 37.59375 2.640625 33.90625 0.734375 \nQ 30.21875 -1.171875 26.078125 -1.171875 \nQ 16.84375 -1.171875 10.171875 6.203125 \nQ 3.515625 13.578125 3.515625 26.421875 \nQ 3.515625 34.234375 6.21875 40.421875 \nQ 8.9375 46.625 14.078125 49.828125 \nQ 19.234375 53.03125 25.390625 53.03125 \nQ 35.015625 53.03125 40.53125 44.921875 \nL 40.53125 51.859375 \nL 48.4375 51.859375 \nL 48.4375 -19.875 \nz\nM 12.546875 26.078125 \nQ 12.546875 16.0625 16.75 11.0625 \nQ 20.953125 6.0625 26.8125 6.0625 \nQ 32.421875 6.0625 36.46875 10.8125 \nQ 40.53125 15.578125 40.53125 25.296875 \nQ 40.53125 35.640625 36.25 40.859375 \nQ 31.984375 46.09375 26.21875 46.09375 \nQ 20.515625 46.09375 16.53125 41.234375 \nQ 12.546875 36.375 12.546875 26.078125 \nz\n\" id=\"ArialMT-113\"/>\n      <path d=\"M 40.578125 0 \nL 40.578125 7.625 \nQ 34.515625 -1.171875 24.125 -1.171875 \nQ 19.53125 -1.171875 15.546875 0.578125 \nQ 11.578125 2.34375 9.640625 5 \nQ 7.71875 7.671875 6.9375 11.53125 \nQ 6.390625 14.109375 6.390625 19.734375 \nL 6.390625 51.859375 \nL 15.1875 51.859375 \nL 15.1875 23.09375 \nQ 15.1875 16.21875 15.71875 13.8125 \nQ 16.546875 10.359375 19.234375 8.375 \nQ 21.921875 6.390625 25.875 6.390625 \nQ 29.828125 6.390625 33.296875 8.421875 \nQ 36.765625 10.453125 38.203125 13.9375 \nQ 39.65625 17.4375 39.65625 24.078125 \nL 39.65625 51.859375 \nL 48.4375 51.859375 \nL 48.4375 0 \nz\n\" id=\"ArialMT-117\"/>\n      <path d=\"M 6.59375 0 \nL 6.59375 51.859375 \nL 14.5 51.859375 \nL 14.5 44.484375 \nQ 20.21875 53.03125 31 53.03125 \nQ 35.6875 53.03125 39.625 51.34375 \nQ 43.5625 49.65625 45.515625 46.921875 \nQ 47.46875 44.1875 48.25 40.4375 \nQ 48.734375 37.984375 48.734375 31.890625 \nL 48.734375 0 \nL 39.9375 0 \nL 39.9375 31.546875 \nQ 39.9375 36.921875 38.90625 39.578125 \nQ 37.890625 42.234375 35.28125 43.8125 \nQ 32.671875 45.40625 29.15625 45.40625 \nQ 23.53125 45.40625 19.453125 41.84375 \nQ 15.375 38.28125 15.375 28.328125 \nL 15.375 0 \nz\n\" id=\"ArialMT-110\"/>\n      <path d=\"M 40.4375 19 \nL 49.078125 17.875 \nQ 47.65625 8.9375 41.8125 3.875 \nQ 35.984375 -1.171875 27.484375 -1.171875 \nQ 16.84375 -1.171875 10.375 5.78125 \nQ 3.90625 12.75 3.90625 25.734375 \nQ 3.90625 34.125 6.6875 40.421875 \nQ 9.46875 46.734375 15.15625 49.875 \nQ 20.84375 53.03125 27.546875 53.03125 \nQ 35.984375 53.03125 41.359375 48.75 \nQ 46.734375 44.484375 48.25 36.625 \nL 39.703125 35.296875 \nQ 38.484375 40.53125 35.375 43.15625 \nQ 32.28125 45.796875 27.875 45.796875 \nQ 21.234375 45.796875 17.078125 41.03125 \nQ 12.9375 36.28125 12.9375 25.984375 \nQ 12.9375 15.53125 16.9375 10.796875 \nQ 20.953125 6.0625 27.390625 6.0625 \nQ 32.5625 6.0625 36.03125 9.234375 \nQ 39.5 12.40625 40.4375 19 \nz\n\" id=\"ArialMT-99\"/>\n      <path d=\"M 6.203125 -19.96875 \nL 5.21875 -11.71875 \nQ 8.109375 -12.5 10.25 -12.5 \nQ 13.1875 -12.5 14.9375 -11.515625 \nQ 16.703125 -10.546875 17.828125 -8.796875 \nQ 18.65625 -7.46875 20.515625 -2.25 \nQ 20.75 -1.515625 21.296875 -0.09375 \nL 1.609375 51.859375 \nL 11.078125 51.859375 \nL 21.875 21.828125 \nQ 23.96875 16.109375 25.640625 9.8125 \nQ 27.15625 15.875 29.25 21.625 \nL 40.328125 51.859375 \nL 49.125 51.859375 \nL 29.390625 -0.875 \nQ 26.21875 -9.421875 24.46875 -12.640625 \nQ 22.125 -17 19.09375 -19.015625 \nQ 16.0625 -21.046875 11.859375 -21.046875 \nQ 9.328125 -21.046875 6.203125 -19.96875 \nz\n\" id=\"ArialMT-121\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(14.357813 139.541094)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#ArialMT-70\"/>\n      <use x=\"61.083984\" xlink:href=\"#ArialMT-114\"/>\n      <use x=\"94.384766\" xlink:href=\"#ArialMT-101\"/>\n      <use x=\"150\" xlink:href=\"#ArialMT-113\"/>\n      <use x=\"205.615234\" xlink:href=\"#ArialMT-117\"/>\n      <use x=\"261.230469\" xlink:href=\"#ArialMT-101\"/>\n      <use x=\"316.845703\" xlink:href=\"#ArialMT-110\"/>\n      <use x=\"372.460938\" xlink:href=\"#ArialMT-99\"/>\n      <use x=\"422.460938\" xlink:href=\"#ArialMT-121\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"patch_3\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 50.302557 224.64 \nL 80.73892 224.64 \nL 80.73892 221.352925 \nL 50.302557 221.352925 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 80.73892 224.64 \nL 111.175284 224.64 \nL 111.175284 188.482177 \nL 80.73892 188.482177 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 111.175284 224.64 \nL 141.611648 224.64 \nL 141.611648 116.166531 \nL 111.175284 116.166531 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 141.611648 224.64 \nL 172.048011 224.64 \nL 172.048011 99.731156 \nL 141.611648 99.731156 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_7\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 172.048011 224.64 \nL 202.484375 224.64 \nL 202.484375 103.018231 \nL 172.048011 103.018231 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_8\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 202.484375 224.64 \nL 232.920739 224.64 \nL 232.920739 27.41551 \nL 202.484375 27.41551 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_9\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 232.920739 224.64 \nL 263.357102 224.64 \nL 263.357102 17.554286 \nL 232.920739 17.554286 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_10\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 263.357102 224.64 \nL 293.793466 224.64 \nL 293.793466 83.295782 \nL 263.357102 83.295782 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_11\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 293.793466 224.64 \nL 324.22983 224.64 \nL 324.22983 178.620952 \nL 293.793466 178.620952 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_12\">\n    <path clip-path=\"url(#pe629cf66bf)\" d=\"M 324.22983 224.64 \nL 354.666193 224.64 \nL 354.666193 214.778776 \nL 324.22983 214.778776 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_13\">\n    <path d=\"M 35.084375 224.64 \nL 35.084375 7.2 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:square;stroke-linejoin:miter;\"/>\n   </g>\n   <g id=\"patch_14\">\n    <path d=\"M 369.884375 224.64 \nL 369.884375 7.2 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:square;stroke-linejoin:miter;\"/>\n   </g>\n   <g id=\"patch_15\">\n    <path d=\"M 35.084375 224.64 \nL 369.884375 224.64 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:square;stroke-linejoin:miter;\"/>\n   </g>\n   <g id=\"patch_16\">\n    <path d=\"M 35.084375 7.2 \nL 369.884375 7.2 \n\" style=\"fill:none;stroke:#cccccc;stroke-linecap:square;stroke-linejoin:miter;\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pe629cf66bf\">\n   <rect height=\"217.44\" width=\"334.8\" x=\"35.084375\" y=\"7.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEBCAYAAABysL6vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAV+UlEQVR4nO3df0xV9/3H8dcBBAVhjna1WVELFRXTdHUalASpW62sdbXqRKCtbtG6ajUVOzbQKWimgnXOTZZGbPfjOxCtbV1jZpNNsQkqlTm2uQ1v61REHWptrQp3IBTP9w/1DiviVe4B+dznIzHhnsP5fN4fz70vTj73/LBs27YFADBOQFcXAABwBgEPAIYi4AHAUAQ8ABiKgAcAQxHwAGCooK4uoLXKysquLgEAuqXhw4ffsOyuCnip7SI7wuVyKS4uzqdtdif+PH5/Hrvk3+P3t7Hf7OCYKRoAMBQBDwCGIuABwFAEPAAYioAHAEMR8ABgKAIeAAxFwAOAoe66C50AXO/B7O0d2ProHW95LH98B/rF3YAjeAAwFAEPAIYi4AHAUAQ8ABiKgAcAQxHwAGAox06TLCws1K5du9Tc3Kz09HTFx8crOztblmUpNjZWubm5Cgjg7wsAOMWRhK2oqNDf/vY3bdq0SUVFRTp9+rTy8vKUkZGhkpIS2bat0tJSJ7oGAFzlSMDv2bNHgwYN0ty5czV79myNGTNGVVVVio+PlyQlJSWpvLzcia4BAFc5MkXz2Wefqba2VuvXr9fJkyc1Z84c2bYty7IkSWFhYaqrq2tzW5fL5dNaGhsbfd5md+LP4/fl2J/8vzu/IrS76s7vG39+37fmSMD36dNHMTExCg4OVkxMjEJCQnT69GnPerfbrYiIiDa39fVzFP3t2Yxf5M/j9+3Y/S/gu/P7xt/e9536TNbhw4dr9+7dsm1bZ86cUUNDgxISElRRUSFJKisr04gRI5zoGgBwlSNH8N/4xje0f/9+TZkyRbZtKycnR1FRUVqyZIl+9rOfKSYmRsnJyU50DQC4yrHTJH/0ox/dsKy4uNip7gAAX8CJ6ABgKAIeAAxFwAOAoQh4ADAUAQ8AhiLgAcBQBDwAGIqABwBDEfAAYCgCHgAMRcADgKEIeAAwFAEPAIYi4AHAUAQ8ABiKgAcAQxHwAGAoAh4ADEXAA4ChCHgAMBQBDwCGIuABwFAEPAAYioAHAEMR8ABgqCCnGp40aZJ69+4tSYqKilJqaqpWrFihwMBAJSYmat68eU51DQCQQwF/6dIl2batoqIiz7JnnnlGBQUF6tevn77//e/r4MGDGjp0qBPdAwDk0BTNhx9+qIaGBs2YMUPTp0/X/v371dTUpP79+8uyLCUmJqq8vNyJrgEAVzlyBN+zZ0/NnDlTKSkpOnbsmGbNmqWIiAjP+rCwMJ04caLNbV0ul09raWxs9Hmb3Yk/j9+fx+4L3fn/jn1/hSMBHx0drQEDBsiyLEVHRys8PFznz5/3rHe73dcFfmtxcXE+rcXlcvm8ze7En8fv27Ef9VE73Ud3ft/42/u+srKyzeWOTNG8/fbbys/PlySdOXNGDQ0NCg0N1fHjx2Xbtvbs2aMRI0Y40TUA4CpHjuCnTJmihQsXKj09XZZlaeXKlQoICFBmZqZaWlqUmJior33ta050DQC4ypGADw4O1po1a25YvmXLFie6AwC0gQudAMBQBDwAGIqABwBDEfAAYCgCHgAMRcADgKEIeAAwFAEPAIYi4AHAUAQ8ABiKgAcAQxHwAGAoAh4ADEXAA4ChCHgAMBQBDwCGIuABwFAEPAAYypFH9gHo/h7M3t5lfR/LH99lfZuEI3gAMBQBDwCGIuABwFAEPAAYioAHAEMR8ABgKMcC/tNPP9Vjjz2mI0eOqKamRunp6Xr22WeVm5ury5cvO9UtAOAqRwK+ublZOTk56tmzpyQpLy9PGRkZKikpkW3bKi0tdaJbAEArjgT8qlWrlJaWpvvuu0+SVFVVpfj4eElSUlKSysvLnegWANCKz69k3bp1qyIjIzV69Ght2LBBkmTbtizLkiSFhYWprq7uptu7XC6f1tPY2OjzNrsTfx6/P4+9u+vofmPfX+FVwJ89e1Zf+cpXvGrwnXfekWVZ+uCDD+RyuZSVlaVz58551rvdbkVERNx0+7i4OK/68ZbL5fJ5m92JP4/ft2M/6qN24I2O7jd/e99XVla2udyrgH/55ZcVGRmpKVOm6LHHHlNAwM1ndjZu3Oj5edq0aVq6dKlWr16tiooKjRw5UmVlZRo1atRtlg8AuF1ezcFv2rRJCxYs0J///GelpaVp7dq1OnHihNedZGVlqaCgQKmpqWpublZycvIdFwwA8I7Xc/B9+/ZVv379VFVVpUOHDmnFihUaOHCgMjMzb7pNUVGR5+fi4uKOVQoAuC1eBfz8+fP173//WxMmTNDq1avVt29fSdLkyZMdLQ4AcOe8CvipU6fq0UcfVVhYmD7++GPP8k2bNjlWGO5e3Ccc6B68moP/61//qsLCQknS8uXLPac/hoSEOFcZAKBDvAr4999/X6+88ookad26ddq1a5ejRQEAOs6rgLcsS01NTZKu3IbAtm1HiwIAdJxXc/BpaWl6+umnNWjQIB09elQvvPCC03UBbbr9+X8uUIL/8irgU1JS9Pjjj+vEiRPq16+fIiMjna4LANBBXgW8y+XSm2++qUuXLnmW5eXlOVYUAKDjvAr47OxsPf/887r//vudrgcA4CNeBfy9996rlJQUp2sBAPiQVwH/wAMPaMOGDYqLi/Pc9jcxMdHRwgAAHeNVwDc3N6u6ulrV1dWeZQQ8ANzdvAr4vLw8VVdX6/jx4xo8eLDnSU0AgLuXVwFfXFysHTt26MKFC5o0aZJqamqUk5PjdG0AgA7w6krW7du36ze/+Y3Cw8P13e9+VwcOHHC6LgBAB3kV8NeeqXrtC9bg4GBHiwIAdJxXUzTf/va39dxzz6m2tlazZs3S2LFjna4LANBBXgX8888/r4SEBB06dEjR0dEaMmSI03UBADrIq4D/5S9/6fn5yJEj2rlzp+bNm+dYUfCO9zfe4oZbgD/y+kpW6cpc/MGDB3X58mVHiwIAdJzXtwtujdsFA8Ddz6uAb30F69mzZ1VbW+tYQQAA3/Aq4Ftf1BQSEqKsrCzHCgIA+IZXAV9UVOR0HQAAH/Mq4CdMmCC3262QkBDPQz+uXfxUWlrqaIEAgDvjVcAPGzZMEydO1LBhw/TRRx/pV7/6lZYvX37T329padHixYtVXV0ty7K0bNkyhYSEKDs7W5ZlKTY2Vrm5uQoI8OpCWgDAHfAq4I8cOaJhw4ZJkgYPHqxTp061e7uC999/X5K0efNmVVRUaO3atbJtWxkZGRo5cqRycnJUWlqqJ554wgdDAAC0xauADw8P189//nM98sgj+stf/qKvfvWr7f7+2LFjNWbMGElSbW2tIiIiVF5ervj4eElSUlKS9u7dS8ADgIO8Cvg1a9aopKREu3fv1uDBg/XKK6/cuuGgIGVlZWnHjh1at26d9u7d67lZWVhYmOrq6trczuVy3Ub5t9bY2OjzNgE4q6OfWT73V3gV8CEhIfrSl74kt9ut6OhoXbx4UZGRkbfcbtWqVcrMzNTUqVM9X85KktvtVkRERJvbxMXFeVm6d1wul8/bvHtwCwKYqaOfWbM/9zeqrKxsc7lX33Lm5OSotrZW5eXlcrvdtzwP/t1331VhYaEkqVevXrIsSw8//LAqKiokSWVlZRoxYsTt1A8AuE1eBfzx48c1f/58BQcH65vf/OZNp1euGTdunA4ePKjnnntOM2fO1KJFi5STk6OCggKlpqaqublZycnJPhkAAKBtXk3RtLS06Ny5c7IsS/X19bc8vTE0NFS/+MUvblheXFx8Z1UCAG6bVwG/YMECpaen6+zZs0pNTdWPf/xjp+sCAHSQVwF/6tQp/fGPf9S5c+f05S9/2XM2DADg7uXVHPyWLVskSZGRkYQ7AHQTXh3BNzU1aeLEiYqOjvbMv69Zs8bRwgAAHdNuwL/22mt66aWXlJmZqTNnzqhv376dVRcAoIPanaLZt2+fJCk+Pl5vvfWW4uPjPf8AAHe3dgPetu02fwYA3P3aDfjWX6jy5SoAdC/tzsFXVVUpLS1Ntm3r8OHDnp8ty9LmzZs7q0YAwB1oN+C3bdvWWXUAAHys3YB/4IEHOqsOAICP8cw8ADAUAQ8AhiLgAcBQBDwAGIqABwBDEfAAYCgCHgAMRcADgKEIeAAwFAEPAIYi4AHAUAQ8ABiKgAcAQ3n10O3b0dzcrEWLFuk///mPmpqaNGfOHA0cOFDZ2dmyLEuxsbHKzc31PLwbAOAMnwf8tm3b1KdPH61evVrnz5/XxIkTNWTIEGVkZGjkyJHKyclRaWmpnnjiCV93DQBoxeeH0d/61rc0f/58SVee4xoYGKiqqirPg7qTkpJUXl7u624BAF/g84APCwtT7969VV9fr5dfflkZGRmex/xdW19XV+frbgEAX+DzKRpJOnXqlObOnatnn31WTz/9tFavXu1Z53a7FRERcdNtXS6XT2tpbGz0eZsAnNXRzyyf+yt8HvCffPKJZsyYoZycHCUkJEiShg4dqoqKCo0cOVJlZWUaNWrUTbePi4vzaT0ul8vnbd49jnZ1AYAjOvqZNftzf6PKyso2l/t8imb9+vW6ePGiXnvtNU2bNk3Tpk1TRkaGCgoKlJqaqubmZiUnJ/u6WwDAF/j8CH7x4sVavHjxDcuLi4t93RUAoB2cjA4AhiLgAcBQBDwAGIqABwBDEfAAYCgCHgAMRcADgKEIeAAwFAEPAIYi4AHAUAQ8ABiKgAcAQxHwAGAoAh4ADEXAA4ChHHlkHwB0xIPZ233Qyu0/8exY/ngf9Hv34AgeAAxFwAOAoQh4ADAUAQ8AhiLgAcBQBDwAGIqABwBDEfAAYCgCHgAM5diVrAcOHNBPf/pTFRUVqaamRtnZ2bIsS7GxscrNzVVAgBl/W3xzxR0A+J4jKfv6669r8eLFunTpkiQpLy9PGRkZKikpkW3bKi0tdaJbAEArjgR8//79VVBQ4HldVVWl+Ph4SVJSUpLKy8ud6BYA0IojUzTJyck6efKk57Vt27IsS5IUFhamurq6m27rcrl8WktjY6PP2wRgJtOyolPuJtl6vt3tdisiIuKmvxsXF+fTvl0ul8/bvN7t37EOwN3J2axwTmVlZZvLO+WbzqFDh6qiokKSVFZWphEjRnRGtwDg1zol4LOyslRQUKDU1FQ1NzcrOTm5M7oFAL/m2BRNVFSUtmzZIkmKjo5WcXGxU10BANpgxsnoAIAbEPAAYCgCHgAMRcADgKEIeAAwFAEPAIYi4AHAUAQ8ABiKgAcAQxHwAGAoAh4ADEXAA4ChCHgAMBQBDwCGIuABwFAEPAAYioAHAEMR8ABgKMce2QcA3c2D2du7pN9j+eMdaZcjeAAwFAEPAIYi4AHAUAQ8ABiKgAcAQxHwAGCoTjtN8vLly1q6dKk++ugjBQcHa/ny5RowYIDP2m//9KajPusHALqLTjuC37lzp5qamvTmm2/qBz/4gfLz8zurawDwS50W8JWVlRo9erQk6dFHH9W//vWvzuoaAPxSp03R1NfXq3fv3p7XgYGB+vzzzxUUdH0JlZWVd9T+Oyn3d6g+AOgqd5p7t9JpAd+7d2+53W7P68uXL98Q7sOHD++scgDAeJ02RfP1r39dZWVlkqS///3vGjRoUGd1DQB+ybJt2+6Mjq6dRXPo0CHZtq2VK1fqoYce6oyuAcAvdVrAd4aWlhYtXrxY1dXVsixLy5YtU0hIiLKzs2VZlmJjY5Wbm6uAAHNP///00081efJk/frXv1ZQUJBfjX3SpEme73mioqKUmpqqFStWKDAwUImJiZo3b14XV+icwsJC7dq1S83NzUpPT1d8fLzf7PutW7fq97//vSTp0qVLcrlcKioq8pt93y7bIDt27LCzs7Nt27btffv22bNnz7ZffPFFe9++fbZt2/aSJUvsP/3pT11ZoqOamprsl156yR43bpx9+PBhvxp7Y2Oj/cwzz1y3bMKECXZNTY19+fJl+4UXXrCrqqq6qDpn7du3z37xxRftlpYWu76+3l63bp1f7fvWli5dam/evNlv9v2tGPUnfezYsfrJT34iSaqtrVVERISqqqoUHx8vSUpKSlJ5eXlXluioVatWKS0tTffdd58k+dXYP/zwQzU0NGjGjBmaPn269u/fr6amJvXv31+WZSkxMdHY8e/Zs0eDBg3S3LlzNXv2bI0ZM8av9v01//znP3X48GGNHz/eb/b9rRj3wI+goCBlZWVpx44dWrdunfbu3SvLsiRJYWFhqqur6+IKnbF161ZFRkZq9OjR2rBhgyTJtm2/GLsk9ezZUzNnzlRKSoqOHTumWbNmKSIiwrM+LCxMJ06c6MIKnfPZZ5+ptrZW69ev18mTJzVnzhy/2vfXFBYWau7cuTeckm3yvr8V4wJeunIkm5mZqalTp+rSpUue5W63+7oPvUneeecdWZalDz74QC6XS1lZWTp37pxnvcljl6To6GgNGDBAlmUpOjpa4eHhOn/+vGe9yePv06ePYmJiFBwcrJiYGIWEhOj06dOe9SaP/ZqLFy+qurpao0aNUn19/XWnZPvD+G/GqCmad999V4WFhZKkXr16ybIsPfzww6qoqJAklZWVacSIEV1ZomM2btyo4uJiFRUVKS4uTqtWrVJSUpJfjF2S3n77bc/tL86cOaOGhgaFhobq+PHjsm1be/bsMXb8w4cP1+7du2XbtmfsCQkJfrPvJWn//v1KSEiQdOWamx49evjFvr8Vo86i+e9//6uFCxfqk08+0eeff65Zs2bpoYce0pIlS9Tc3KyYmBgtX75cgYGBXV2qo6ZNm6alS5cqICDAb8be1NSkhQsXqra2VpZlKTMzUwEBAVq5cqVaWlqUmJioBQsWdHWZjnn11VdVUVEh27a1YMECRUVF+c2+l6Q33nhDQUFB+t73vifpyrU2/rLv22NUwAMA/seoKRoAwP8Q8ABgKAIeAAxFwAOAoQh4ADAUAQ+08vrrrysxMfG6C+SA7oqAB1rZtm2bnnrqKW3f3t5D3IHuwchbFQB3oqKiQv3791daWpp++MMfavLkyfrHP/6hZcuWKSwsTPfcc49CQkKUn5+voqIi/eEPf5BlWXrqqac0ffr0ri4fuAFH8MBVb731llJSUjz3dTlw4IByc3OVn5+v3/3ud+rfv78k6fDhw3rvvfdUUlKijRs3aufOnTp69GgXVw/ciCN4QNKFCxdUVlamc+fOqaioSPX19SouLtbHH3+s2NhYSVfu+fLee+/p0KFDqq2t9VwWf+HCBdXU1CgmJqYLRwDciIAHdGXu/Tvf+Y6ysrIkSQ0NDXr88cfVs2dPHT58WAMHDtSBAwckSTExMRo4cKDeeOMNWZal3/72txo8eHBXlg+0iYAHdGV65tVXX/W87tWrl8aNG6d7771XixYtUmhoqHr06KG+fftqyJAhSkhIUHp6upqamvTII4+ob9++XVg90DZuNga0Y+PGjXryyScVGRmptWvXqkePHv77fE90OxzBA+245557NGPGDIWGhio8PNxzz3mgO+AIHgAMxWmSAGAoAh4ADEXAA4ChCHgAMBQBDwCGIuABwFD/D2PSNa12TymWAAAAAElFTkSuQmCC\n"
     },
     "metadata": {}
    }
   ],
   "source": [
    "heart_data[\"age\"].plot(kind=\"hist\")\n",
    "plt.xlabel(\"Age\")\n",
    "plt.show()\n",
    "\n",
    "heart_data[\"partition\"] = (heart_data[\"age\"] / 10).round().astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save data\n",
    "We save the data to parquet files twice: once normally and once with partitioning.\n",
    "\n",
    "When saving the file normally we produce a single `heart.parquet` file. When saving with partitions we create a folder `heart_partition` as follows:\n",
    "```\n",
    "heart_partition\n",
    "├── bracket=3\n",
    "│   └── ___.parquet\n",
    "├── bracket=4\n",
    "│   └── ___.parquet\n",
    "...\n",
    "└── bracket=8\n",
    "    └── ___.parquet\n",
    "```\n",
    "\n",
    "Each column we use for partitioning the dataframe splits the folder structure, and the filtered data will be saved as a parquet file inside the respective folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to parquet\n",
    "heart_data.to_parquet(\n",
    "    os.path.join(dir_path, \"data\", \"heart.parquet\"),\n",
    "    index=False,\n",
    ")\n",
    "\n",
    "# Save to partitioned parquet\n",
    "heart_data.to_parquet(\n",
    "    os.path.join(dir_path, \"data\", \"heart_partition\"),\n",
    "    partition_cols=[\"partition\"],\n",
    "    index=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and testing\n",
    "Pandas also supports loading from partitioned parquet datasets. Instead of loading the `.parquet` files we can point to the directory to load the whole dataset.\n",
    "\n",
    "We resort the data after loading the partitioned dataset. When we load we concatenate the various partitions together so the ordering may not be preserved.\n",
    "The `partition` column is loaded as a categorical type, so we convert back to `int` to ensure its the same as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load single parquet file\n",
    "heart_data_1 = pd.read_parquet(os.path.join(dir_path, \"data\", \"heart.parquet\"))\n",
    "\n",
    "# Load parquet dataset\n",
    "heart_data_2 = pd.read_parquet(os.path.join(dir_path, \"data\", \"heart_partition\"))\n",
    "heart_data_2.sort_values(by=[\"age\", \"sex\", \"cp\"], inplace=True)\n",
    "heart_data_2[\"partition\"] = heart_data_2[\"partition\"].astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing the resulting dataframes to the original show that they are all identical:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.testing.assert_frame_equal(heart_data, heart_data_1)\n",
    "pd.testing.assert_frame_equal(heart_data, heart_data_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading a subset of data\n",
    "Usually if we wanted a subset of the data, we would load the whole dataframe and filter it in memory.\n",
    "One of the benefits of using partitioned datasets is that we can load a subset of data directly from disk without needing to load all the data first.\n",
    "This is possible when we are filtering on one of the partition columns. This can help reduce the memory footprint of a program.\n",
    "\n",
    "For example, in the following we filter for the 3rd parition. In this case only that partition is loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load and then filter single parquet file\n",
    "heart_data_1 = pd.read_parquet(os.path.join(dir_path, \"data\", \"heart.parquet\"))\n",
    "filt = heart_data_1[\"partition\"] == 3\n",
    "heart_data_1 = heart_data_1.loc[filt, :]\n",
    "\n",
    "# Filter and load parquet dataset\n",
    "heart_data_2 = pd.read_parquet(\n",
    "    os.path.join(dir_path, \"data\", \"heart_partition\"), filters=[(\"partition\", \"=\", \"3\")]\n",
    ")\n",
    "heart_data_2.sort_values(by=[\"age\", \"sex\", \"cp\"], inplace=True)\n",
    "heart_data_2[\"partition\"] = heart_data_2[\"partition\"].astype(int)\n",
    "\n",
    "# Results are identical\n",
    "pd.testing.assert_frame_equal(heart_data_1, heart_data_2)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python37664bitdataanalysisenvvenv63b40a3c1a484f6297148098436bce1d",
   "display_name": "Python 3.7.6 64-bit ('dataAnalysisEnv': venv)"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}