blogs/write-multiple-parquet-files-to-a-single-csv-using-python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a467f3e2-2e61-4b0a-b5c4-d8d0ae8d652a",
   "metadata": {},
   "source": [
    "# Write Multiple Parquet Files to a Single CSV using Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b2c8aa1-e9b5-4a52-89e8-d09f6bceb59c",
   "metadata": {},
   "source": [
    "This post shows you how to read multiple Parquet files into your Python session and write them to a single CSV.\n",
    "\n",
    "We will:\n",
    "- create multiple dummy Parquet files\n",
    "- read these Parquet files into a Dask Dataframe with a single `read_parquet` command\n",
    "- write the Dask DataFrame to a single CSV file\n",
    "\n",
    "Using Dask to convert your Parquet files to CSV means:\n",
    "1. You don't have to worry about file sizes or running out of memory, and\n",
    "2. You can read and write easily from cloud data storage like Amazon S3. \n",
    "\n",
    "Let's get to it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1490cf5-49e7-44cf-9a1f-117aa32ff3db",
   "metadata": {},
   "source": [
    "## Create Multiple Parquet Files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee290a7a-b907-4931-985a-391d60514e7f",
   "metadata": {},
   "source": [
    "We'll start by creating some dummy dataframes write these to 3 separate Parquet files using a simple `for` loop. Each dataframe will contain 3 rows and 4 columns, populated with random integers between 0 and 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "20bdfc27-c8dd-42c4-908b-8f21857a80b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# use the recommended method for generating random integers with NumPy\n",
    "rng = np.random.default_rng()\n",
    "\n",
    "# generate 3 dummy dataframes with similar filenames\n",
    "for i in range(3):\n",
    "    df = pd.DataFrame(rng.integers(0, 100, size=(3, 4)), columns=list('ABCD'))\n",
    "    df.to_parquet(f\"dummy_df_{i}.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ab0f95c-6bb0-49ca-8554-bb5c6f8ee2be",
   "metadata": {},
   "source": [
    "If you're working in an IPython session on Mac or Linux you can run `! ls` to confirm that the files have been created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7d8ad9fa-b2d8-4974-bfbf-cd2fb0b17c4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dummy_df_0.parquet        dummy_df_2.parquet        parquet-to-csv-dask.ipynb\n",
      "dummy_df_1.parquet        dummy_df_all.csv\n"
     ]
    }
   ],
   "source": [
    "! ls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9528e8ba-6de4-4fb1-a6c5-1c2f4060e063",
   "metadata": {},
   "source": [
    "## Load Multiple Parquet Files with Dask DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef4e903e-6fa5-488c-84dd-340816398953",
   "metadata": {},
   "source": [
    "We now have all the ingredients to run our experiment. \n",
    "\n",
    "Let's see how we can load multiple Parquet files into a DataFrame and write them to single CSV file using the Dask DataFrame API. Dask is a library for distributed computing that scales familiar Python APIs like pandas, NumPy and scikit-learn to arbitrarily large datasets. Read more about the basics of Dask here.\n",
    "\n",
    "We'll start by importing `dask.dataframe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "95f0b486-85a0-4318-a78e-65522826cc83",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.dataframe as dd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "718a59a7-e44c-4f59-bbae-a9fb09e17c98",
   "metadata": {},
   "source": [
    "We'll then use the `read_parquet` method to read all of our Parquet files at once. \n",
    "\n",
    "We can do this because Dask accepts an asterisk `*` as a glob / wildcard character that will match related filenames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b27200f9-ac84-4466-82dc-2f4223ac1932",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = dd.read_parquet('dummy_df_*.parquet')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "004f86ce-e944-43d0-b292-f9551b07d663",
   "metadata": {},
   "source": [
    "Let's have a look at the contents of our DataFrame by calling `ddf.compute()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "09ce6cc8-c739-4bcb-ad97-469ff1304961",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "      <td>85</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>92</td>\n",
       "      <td>53</td>\n",
       "      <td>30</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>61</td>\n",
       "      <td>79</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>13</td>\n",
       "      <td>74</td>\n",
       "      <td>5</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19</td>\n",
       "      <td>30</td>\n",
       "      <td>5</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>34</td>\n",
       "      <td>13</td>\n",
       "      <td>26</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>80</td>\n",
       "      <td>12</td>\n",
       "      <td>84</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23</td>\n",
       "      <td>57</td>\n",
       "      <td>7</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>44</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A   B   C   D\n",
       "0   0  41  85  11\n",
       "1  92  53  30  72\n",
       "2   6  61  79  29\n",
       "0  13  74   5  25\n",
       "1  19  30   5  89\n",
       "2  34  13  26  70\n",
       "0  80  12  84  40\n",
       "1  23  57   7  44\n",
       "2  44   2  18  64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ddf.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea37b2c2-0ef2-4582-8d40-5bc0cf77bafa",
   "metadata": {},
   "source": [
    "Our Dask DataFrame now contains all the data from our 3 separate Parquet files.\n",
    "\n",
    "## Write Parquet Files to CSV\n",
    "\n",
    "We can now write our multiple Parquet files out to a single CSV file using the `to_csv` method. Make sure to set `single_file` to `True` and `index` to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "150450fb-bff7-4190-b62e-8dafd7864f06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/Users/rpelgrim/Documents/git/coiled-resources/parquet-csv/dummy_df_all.csv']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ddf.to_csv(\"dummy_df_all.csv\", \n",
    "           single_file=True, \n",
    "           index=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8cad0e1-2dbc-455c-a3e8-c34093a2669a",
   "metadata": {},
   "source": [
    "Let's verify that this actually worked by reading the csv file into a pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "c6a68f15-2516-4c3a-8bf5-bba69267566d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_csv = pd.read_csv(\"dummy_df_all.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "fab90e2a-cfe8-4c37-9202-49c0d61e6d9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "      <td>85</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>92</td>\n",
       "      <td>53</td>\n",
       "      <td>30</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>61</td>\n",
       "      <td>79</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>74</td>\n",
       "      <td>5</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>19</td>\n",
       "      <td>30</td>\n",
       "      <td>5</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>34</td>\n",
       "      <td>13</td>\n",
       "      <td>26</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>80</td>\n",
       "      <td>12</td>\n",
       "      <td>84</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>23</td>\n",
       "      <td>57</td>\n",
       "      <td>7</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>44</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A   B   C   D\n",
       "0   0  41  85  11\n",
       "1  92  53  30  72\n",
       "2   6  61  79  29\n",
       "3  13  74   5  25\n",
       "4  19  30   5  89\n",
       "5  34  13  26  70\n",
       "6  80  12  84  40\n",
       "7  23  57   7  44\n",
       "8  44   2  18  64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e048b6a-828f-4b8c-af1e-8e5714d6a6c4",
   "metadata": {},
   "source": [
    "## About the Dask API\n",
    "\n",
    "Dask follows the pandas API as much as possible, but there are two important differences to note in the example above:\n",
    "1. using the `.compute()` method to inspect the DataFrame, and\n",
    "2. setting the index "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "084162a9-4a73-4bcf-ba4e-11a12a9de906",
   "metadata": {},
   "source": [
    "### 1. Using .compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60cd59cd-78c1-44c6-ae3c-f265b2fe769a",
   "metadata": {},
   "source": [
    "Dask uses 'lazy evaluation' to optimize performance. This means that results are not computed until you explicitly tell Dask to do so. This allows Dask to find the quickest way to get you your result *when you actually need it*. \n",
    "\n",
    "Simply calling `ddf` will get you some basisc information *about* the DataFrame but not the actual contents. To view the content of the DataFrame, tell Dask to go and run the computation by calling `ddf.compute()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e084edf3-0f59-4135-b7ec-89347778d659",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><strong>Dask DataFrame Structure:</strong></div>\n",
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>npartitions=3</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>int64</td>\n",
       "      <td>int64</td>\n",
       "      <td>int64</td>\n",
       "      <td>int64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "<div>Dask Name: read-parquet, 3 tasks</div>"
      ],
      "text/plain": [
       "Dask DataFrame Structure:\n",
       "                   A      B      C      D\n",
       "npartitions=3                            \n",
       "               int64  int64  int64  int64\n",
       "                 ...    ...    ...    ...\n",
       "                 ...    ...    ...    ...\n",
       "                 ...    ...    ...    ...\n",
       "Dask Name: read-parquet, 3 tasks"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ddf without .compute()\n",
    "ddf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "0015713e-892e-4f37-a9d9-c9731666413f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "      <td>85</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>92</td>\n",
       "      <td>53</td>\n",
       "      <td>30</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>61</td>\n",
       "      <td>79</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>13</td>\n",
       "      <td>74</td>\n",
       "      <td>5</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19</td>\n",
       "      <td>30</td>\n",
       "      <td>5</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>34</td>\n",
       "      <td>13</td>\n",
       "      <td>26</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>80</td>\n",
       "      <td>12</td>\n",
       "      <td>84</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23</td>\n",
       "      <td>57</td>\n",
       "      <td>7</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>44</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A   B   C   D\n",
       "0   0  41  85  11\n",
       "1  92  53  30  72\n",
       "2   6  61  79  29\n",
       "0  13  74   5  25\n",
       "1  19  30   5  89\n",
       "2  34  13  26  70\n",
       "0  80  12  84  40\n",
       "1  23  57   7  44\n",
       "2  44   2  18  64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ddf with .compute()\n",
    "ddf.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8217e063-c33e-4822-a71f-6d0b6a9b17e3",
   "metadata": {},
   "source": [
    "### 2. Dask DataFrame Index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8590e399-4777-4657-acbf-240b9029510e",
   "metadata": {},
   "source": [
    "You might have noticed that the index for the Dask DataFrame runs from 0 to 3 and then repeats. This is because a Dask DataFrame is divided into partitions (3 in this case). Each partition is a pandas DataFrame that has its own index starting from 0. This may seem odd but helps Dask speed up your indexing operations when working with very large datasets.\n",
    "\n",
    "For most purposes - including ours - this is not a problem at all, just something to be aware of. Read this blog if you want to learn more about setting indexes in Dask. https://coiled.io/blog/dask-set-index-dataframe/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c293b30-b8f6-4183-9855-9a5f6f87c311",
   "metadata": {},
   "source": [
    "## Why Use Dask to Write Parquet Files to CSV\n",
    "\n",
    "The benefits of using Dask to write your Parquet files to CSV are that you:\n",
    "1. Don't have to worry about file sizes or running out of memory\n",
    "2. Can easily read and write data to/from cloud-based data storage\n",
    "\n",
    "To demonstrate, let's load 2750 Parquet files (104 GB) into a Dask DataFrame and write them to an S3 bucket as CSV. We'll use a Coiled cluster to access additional hardware resources in the cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f028e046-4adc-4f19-909c-8b0eb9a5c918",
   "metadata": {},
   "outputs": [],
   "source": [
    "import coiled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d0916f14-d0aa-486e-8633-7439eb4887d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.0.60'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coiled.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab4d5427-3dc9-4e7f-85a4-6158e66f900c",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = coiled.Cluster(\n",
    "    name=\"parquet-to-csv\",\n",
    "    software=\"coiled-examples/numpy-zarr\",\n",
    "    n_workers=25,\n",
    "    scheduler_options={'idle_timeout':'1 hour'},\n",
    "    backend_options={'spot': True}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a0405356-cc96-4ec0-b645-ae7c59fc84e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from distributed import Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "48be763c-7735-46f7-bc5b-c0718b653163",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-9eed7d8c-6ed0-11ec-a164-1e008a1e9c26</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "\n",
       "        <tr>\n",
       "        \n",
       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Cluster object</td>\n",
       "            <td style=\"text-align: left;\"><strong>Cluster type:</strong> coiled.Cluster</td>\n",
       "        \n",
       "        </tr>\n",
       "\n",
       "        \n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard: </strong> <a href=\"http://3.234.205.253:8787\" target=\"_blank\">http://3.234.205.253:8787</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\"></td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        </table>\n",
       "\n",
       "        \n",
       "            <details>\n",
       "            <summary style=\"margin-bottom: 20px;\"><h3 style=\"display: inline;\">Cluster Info</h3></summary>\n",
       "            <div class=\"jp-RenderedHTMLCommon jp-RenderedHTML jp-mod-trusted jp-OutputArea-output\">\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\">\n",
       "    </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px; margin-top: 0px;\">Cluster</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">parquet-to-csv</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard:</strong> <a href=\"http://3.234.205.253:8787\" target=\"_blank\">http://3.234.205.253:8787</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Workers:</strong> 20\n",
       "                </td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Total threads:</strong> 40\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Total memory:</strong> 153.34 GiB\n",
       "                </td>\n",
       "            </tr>\n",
       "            \n",
       "        </table>\n",
       "\n",
       "        <details>\n",
       "            <summary style=\"margin-bottom: 20px;\">\n",
       "                <h3 style=\"display: inline;\">Scheduler Info</h3>\n",
       "            </summary>\n",
       "\n",
       "            <div style=\"\">\n",
       "    <div>\n",
       "        <div style=\"width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;\"> </div>\n",
       "        <div style=\"margin-left: 48px;\">\n",
       "            <h3 style=\"margin-bottom: 0px;\">Scheduler</h3>\n",
       "            <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Scheduler-0a13763a-600b-4b2a-9fca-6c91b2179fef</p>\n",
       "            <table style=\"width: 100%; text-align: left;\">\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Comm:</strong> tls://10.4.6.65:8786\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Workers:</strong> 20\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Dashboard:</strong> <a href=\"http://10.4.6.65:8787/status\" target=\"_blank\">http://10.4.6.65:8787/status</a>\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total threads:</strong> 40\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Started:</strong> 1 minute ago\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total memory:</strong> 153.34 GiB\n",
       "                    </td>\n",
       "                </tr>\n",
       "            </table>\n",
       "        </div>\n",
       "    </div>\n",
       "\n",
       "    <details style=\"margin-left: 48px;\">\n",
       "        <summary style=\"margin-bottom: 20px;\">\n",
       "            <h3 style=\"display: inline;\">Workers</h3>\n",
       "        </summary>\n",
       "\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-0c9cef3e49</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.3.243:40737\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.3.243:43009/status\" target=\"_blank\">http://10.4.3.243:43009/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.3.243:39729\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-nnxn5xkv\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-109c0d4b77</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.12.24:43825\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.12.24:34143/status\" target=\"_blank\">http://10.4.12.24:34143/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.12.24:36161\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-me5rozyz\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-1ca7aca039</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.12.200:43295\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.12.200:40745/status\" target=\"_blank\">http://10.4.12.200:40745/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.12.200:38289\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-8a09cy2i\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-246c75c95d</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.7.122:45091\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.7.122:34109/status\" target=\"_blank\">http://10.4.7.122:34109/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.7.122:44729\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-fld6jb9n\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-31a943d6c1</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.5.243:35983\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.5.243:40477/status\" target=\"_blank\">http://10.4.5.243:40477/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.5.243:35389\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-2vc8ueav\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-329f8af185</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.2.110:36139\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.2.110:44913/status\" target=\"_blank\">http://10.4.2.110:44913/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.2.110:41075\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-p82attr5\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-6a0619b34b</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.15.11:37965\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.15.11:32917/status\" target=\"_blank\">http://10.4.15.11:32917/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.15.11:45851\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-kikzv1v8\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-730f0bde0c</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.3.152:43079\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.3.152:36355/status\" target=\"_blank\">http://10.4.3.152:36355/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.3.152:44015\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-b2cruq7y\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-8e71b0094b</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.10.37:43759\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.10.37:41787/status\" target=\"_blank\">http://10.4.10.37:41787/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.10.37:38533\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-fccmoblm\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-96f19380b9</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.6.226:37063\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.6.226:46261/status\" target=\"_blank\">http://10.4.6.226:46261/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.6.226:44595\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-mo3l7axs\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-98640a80e4</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.14.210:41659\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.14.210:46173/status\" target=\"_blank\">http://10.4.14.210:46173/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.14.210:37315\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-mya7gnn0\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-a106c11cdb</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.8.67:39093\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.8.67:35339/status\" target=\"_blank\">http://10.4.8.67:35339/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.8.67:46569\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-990ssr80\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-a8986ea54b</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.7.80:42303\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.7.80:37925/status\" target=\"_blank\">http://10.4.7.80:37925/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.7.80:41627\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-uropr1cs\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-a9f0894b2d</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.8.19:41755\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.8.19:46853/status\" target=\"_blank\">http://10.4.8.19:46853/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.8.19:44507\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-mjr_fbsk\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-ad60ed8f2d</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.8.123:40409\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.8.123:35075/status\" target=\"_blank\">http://10.4.8.123:35075/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.8.123:39801\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-j0zfnzmm\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-dab0b4bebf</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.15.49:39095\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.15.49:46445/status\" target=\"_blank\">http://10.4.15.49:46445/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.15.49:39669\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-oedclo3m\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-dd2b6f65c6</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.1.159:37589\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.1.159:36781/status\" target=\"_blank\">http://10.4.1.159:36781/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.1.159:36877\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-5tb9fwwy\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-e14183ac14</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.3.74:40709\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.3.74:37821/status\" target=\"_blank\">http://10.4.3.74:37821/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.3.74:42263\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-dt5j43n_\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-eb20be7506</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.4.251:38251\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.4.251:39789/status\" target=\"_blank\">http://10.4.4.251:39789/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.4.251:43965\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-y2f8okq2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: coiled-dask-rrpelgr71-94456-worker-f7a0659554</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tls://10.4.9.42:34819\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.4.9.42:43873/status\" target=\"_blank\">http://10.4.9.42:43873/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 7.67 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tls://10.4.9.42:44139\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /dask-worker-space/worker-dv_k3p8k\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "\n",
       "        </details>\n",
       "    </div>\n",
       "</div>\n",
       "            </details>\n",
       "        \n",
       "\n",
       "    </div>\n",
       "</div>"
      ],
      "text/plain": [
       "<Client: 'tls://10.4.6.65:8786' processes=20 threads=40, memory=153.34 GiB>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9be04973-ca06-4d64-b033-19598a321568",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = dd.read_parquet(\n",
    "    \"s3://coiled-datasets/synthetic-data/synth-reg-104GB.parquet/\",\n",
    "    storage_options={'anon':'True', 'use_ssl':'True'}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c5f9e1c4-d442-4ace-849a-8d2485a9c332",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><strong>Dask DataFrame Structure:</strong></div>\n",
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>npartitions=2750</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "      <td>float64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100000</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>274900000</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>274999999</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "<div>Dask Name: read-parquet, 2750 tasks</div>"
      ],
      "text/plain": [
       "Dask DataFrame Structure:\n",
       "                        0        1        2        3        4        5        6        7        8        9       10       11       12       13       14       15       16       17       18       19       20       21       22       23       24       25       26       27       28       29       30       31       32       33       34       35       36       37       38       39       40       41       42       43       44       45       46       47       48       49   target\n",
       "npartitions=2750                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "0                 float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64  float64\n",
       "100000                ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...\n",
       "...                   ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...\n",
       "274900000             ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...\n",
       "274999999             ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...      ...\n",
       "Dask Name: read-parquet, 2750 tasks"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ddf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "73388a5c-76eb-4645-86a8-8cf509fecb5e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.083516</td>\n",
       "      <td>0.173372</td>\n",
       "      <td>-0.973546</td>\n",
       "      <td>-1.465443</td>\n",
       "      <td>1.973955</td>\n",
       "      <td>-0.922526</td>\n",
       "      <td>1.058072</td>\n",
       "      <td>0.302878</td>\n",
       "      <td>1.160762</td>\n",
       "      <td>-0.690999</td>\n",
       "      <td>...</td>\n",
       "      <td>0.478698</td>\n",
       "      <td>-1.286906</td>\n",
       "      <td>0.037474</td>\n",
       "      <td>-0.448159</td>\n",
       "      <td>-0.652509</td>\n",
       "      <td>-1.205982</td>\n",
       "      <td>0.166634</td>\n",
       "      <td>2.526275</td>\n",
       "      <td>-0.890744</td>\n",
       "      <td>223.602485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.077819</td>\n",
       "      <td>-0.507675</td>\n",
       "      <td>1.188347</td>\n",
       "      <td>-0.958974</td>\n",
       "      <td>0.666332</td>\n",
       "      <td>0.699718</td>\n",
       "      <td>0.416365</td>\n",
       "      <td>-0.006916</td>\n",
       "      <td>-0.561665</td>\n",
       "      <td>-0.535323</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.406144</td>\n",
       "      <td>-0.122424</td>\n",
       "      <td>1.623143</td>\n",
       "      <td>0.438106</td>\n",
       "      <td>-1.510411</td>\n",
       "      <td>-0.909098</td>\n",
       "      <td>-0.416044</td>\n",
       "      <td>0.169660</td>\n",
       "      <td>-1.343285</td>\n",
       "      <td>-63.876627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.545396</td>\n",
       "      <td>-1.001309</td>\n",
       "      <td>-0.185548</td>\n",
       "      <td>-0.507883</td>\n",
       "      <td>1.223005</td>\n",
       "      <td>0.405486</td>\n",
       "      <td>-0.838138</td>\n",
       "      <td>-0.521867</td>\n",
       "      <td>1.164290</td>\n",
       "      <td>0.566665</td>\n",
       "      <td>...</td>\n",
       "      <td>1.341402</td>\n",
       "      <td>-0.206474</td>\n",
       "      <td>-1.203585</td>\n",
       "      <td>0.796500</td>\n",
       "      <td>-2.083753</td>\n",
       "      <td>0.670345</td>\n",
       "      <td>1.243194</td>\n",
       "      <td>-0.513658</td>\n",
       "      <td>-1.388109</td>\n",
       "      <td>182.856379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.548436</td>\n",
       "      <td>-0.754629</td>\n",
       "      <td>1.628490</td>\n",
       "      <td>0.954295</td>\n",
       "      <td>0.190117</td>\n",
       "      <td>-0.359459</td>\n",
       "      <td>1.901831</td>\n",
       "      <td>-0.137075</td>\n",
       "      <td>-0.005027</td>\n",
       "      <td>0.918249</td>\n",
       "      <td>...</td>\n",
       "      <td>1.214883</td>\n",
       "      <td>-0.115838</td>\n",
       "      <td>0.287735</td>\n",
       "      <td>-0.115192</td>\n",
       "      <td>-0.499330</td>\n",
       "      <td>0.349165</td>\n",
       "      <td>-1.618127</td>\n",
       "      <td>1.421938</td>\n",
       "      <td>-0.439240</td>\n",
       "      <td>-211.527657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.981102</td>\n",
       "      <td>0.993449</td>\n",
       "      <td>-0.173022</td>\n",
       "      <td>0.503123</td>\n",
       "      <td>0.823864</td>\n",
       "      <td>0.083351</td>\n",
       "      <td>0.242027</td>\n",
       "      <td>0.661806</td>\n",
       "      <td>0.463781</td>\n",
       "      <td>-0.799858</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.988890</td>\n",
       "      <td>-0.541225</td>\n",
       "      <td>-0.298992</td>\n",
       "      <td>0.306095</td>\n",
       "      <td>0.351885</td>\n",
       "      <td>2.269911</td>\n",
       "      <td>0.465673</td>\n",
       "      <td>0.909917</td>\n",
       "      <td>0.513545</td>\n",
       "      <td>-165.464021</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 51 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4         5         6  \\\n",
       "0 -1.083516  0.173372 -0.973546 -1.465443  1.973955 -0.922526  1.058072   \n",
       "1  2.077819 -0.507675  1.188347 -0.958974  0.666332  0.699718  0.416365   \n",
       "2 -1.545396 -1.001309 -0.185548 -0.507883  1.223005  0.405486 -0.838138   \n",
       "3 -0.548436 -0.754629  1.628490  0.954295  0.190117 -0.359459  1.901831   \n",
       "4 -0.981102  0.993449 -0.173022  0.503123  0.823864  0.083351  0.242027   \n",
       "\n",
       "          7         8         9  ...        41        42        43        44  \\\n",
       "0  0.302878  1.160762 -0.690999  ...  0.478698 -1.286906  0.037474 -0.448159   \n",
       "1 -0.006916 -0.561665 -0.535323  ... -0.406144 -0.122424  1.623143  0.438106   \n",
       "2 -0.521867  1.164290  0.566665  ...  1.341402 -0.206474 -1.203585  0.796500   \n",
       "3 -0.137075 -0.005027  0.918249  ...  1.214883 -0.115838  0.287735 -0.115192   \n",
       "4  0.661806  0.463781 -0.799858  ... -0.988890 -0.541225 -0.298992  0.306095   \n",
       "\n",
       "         45        46        47        48        49      target  \n",
       "0 -0.652509 -1.205982  0.166634  2.526275 -0.890744  223.602485  \n",
       "1 -1.510411 -0.909098 -0.416044  0.169660 -1.343285  -63.876627  \n",
       "2 -2.083753  0.670345  1.243194 -0.513658 -1.388109  182.856379  \n",
       "3 -0.499330  0.349165 -1.618127  1.421938 -0.439240 -211.527657  \n",
       "4  0.351885  2.269911  0.465673  0.909917  0.513545 -165.464021  \n",
       "\n",
       "[5 rows x 51 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ddf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdd549cc-1502-42f7-8e17-988ca7f25e71",
   "metadata": {},
   "source": [
    "We'll use the same call to `ddf.to_csv()` we used earlier, just changing the file path to point to our S3 bucket. We'll also set `single_file` to `False` this time to make the best use of Dask's parallel writing capabilities. Technically you can write the Parquet files to a single CSV file, but it will take much longer and the resulting CSV file will be quiet unwieldy and you'll be losing many of the performance benefits that come with parallel read/write."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "44dfaf7d-bbe2-455d-a4b3-5bb48e3ea7d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13 s, sys: 1.21 s, total: 14.2 s\n",
      "Wall time: 21min 54s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['coiled-datasets/synthetic-data/synth-reg-104GB.csv/0000.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0001.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0002.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0003.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0004.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0005.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0006.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0007.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0008.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0009.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0010.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0011.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0012.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0013.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0014.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0015.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0016.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0017.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0018.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0019.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0020.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0021.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0022.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0023.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0024.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0025.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0026.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0027.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0028.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0029.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0030.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0031.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0032.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0033.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0034.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0035.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0036.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0037.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0038.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0039.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0040.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0041.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0042.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0043.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0044.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0045.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0046.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0047.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0048.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0049.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0050.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0051.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0052.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0053.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0054.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0055.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0056.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0057.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0058.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0059.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0060.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0061.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0062.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0063.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0064.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0065.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0066.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0067.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0068.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0069.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0070.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0071.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0072.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0073.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0074.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0075.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0076.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0077.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0078.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0079.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0080.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0081.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0082.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0083.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0084.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0085.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0086.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0087.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0088.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0089.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0090.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0091.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0092.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0093.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0094.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0095.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0096.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0097.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0098.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0099.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0100.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0101.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0102.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0103.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0104.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0105.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0106.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0107.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0108.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0109.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0110.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0111.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0112.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0113.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0114.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0115.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0116.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0117.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0118.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0119.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0120.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0121.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0122.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0123.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0124.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0125.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0126.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0127.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0128.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0129.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0130.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0131.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0132.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0133.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0134.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0135.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0136.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0137.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0138.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0139.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0140.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0141.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0142.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0143.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0144.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0145.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0146.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0147.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0148.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0149.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0150.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0151.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0152.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0153.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0154.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0155.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0156.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0157.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0158.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0159.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0160.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0161.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0162.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0163.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0164.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0165.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0166.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0167.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0168.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0169.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0170.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0171.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0172.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0173.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0174.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0175.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0176.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0177.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0178.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0179.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0180.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0181.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0182.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0183.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0184.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0185.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0186.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0187.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0188.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0189.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0190.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0191.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0192.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0193.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0194.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0195.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0196.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0197.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0198.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0199.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0200.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0201.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0202.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0203.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0204.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0205.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0206.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0207.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0208.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0209.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0210.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0211.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0212.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0213.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0214.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0215.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0216.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0217.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0218.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0219.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0220.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0221.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0222.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0223.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0224.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0225.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0226.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0227.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0228.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0229.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0230.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0231.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0232.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0233.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0234.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0235.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0236.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0237.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0238.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0239.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0240.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0241.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0242.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0243.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0244.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0245.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0246.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0247.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0248.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0249.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0250.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0251.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0252.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0253.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0254.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0255.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0256.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0257.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0258.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0259.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0260.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0261.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0262.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0263.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0264.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0265.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0266.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0267.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0268.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0269.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0270.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0271.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0272.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0273.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0274.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0275.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0276.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0277.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0278.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0279.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0280.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0281.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0282.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0283.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0284.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0285.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0286.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0287.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0288.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0289.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0290.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0291.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0292.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0293.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0294.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0295.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0296.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0297.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0298.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0299.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0300.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0301.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0302.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0303.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0304.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0305.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0306.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0307.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0308.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0309.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0310.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0311.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0312.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0313.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0314.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0315.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0316.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0317.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0318.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0319.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0320.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0321.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0322.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0323.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0324.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0325.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0326.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0327.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0328.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0329.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0330.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0331.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0332.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0333.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0334.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0335.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0336.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0337.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0338.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0339.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0340.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0341.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0342.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0343.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0344.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0345.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0346.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0347.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0348.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0349.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0350.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0351.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0352.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0353.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0354.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0355.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0356.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0357.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0358.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0359.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0360.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0361.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0362.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0363.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0364.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0365.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0366.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0367.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0368.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0369.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0370.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0371.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0372.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0373.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0374.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0375.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0376.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0377.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0378.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0379.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0380.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0381.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0382.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0383.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0384.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0385.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0386.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0387.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0388.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0389.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0390.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0391.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0392.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0393.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0394.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0395.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0396.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0397.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0398.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0399.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0400.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0401.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0402.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0403.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0404.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0405.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0406.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0407.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0408.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0409.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0410.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0411.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0412.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0413.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0414.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0415.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0416.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0417.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0418.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0419.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0420.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0421.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0422.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0423.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0424.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0425.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0426.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0427.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0428.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0429.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0430.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0431.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0432.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0433.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0434.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0435.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0436.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0437.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0438.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0439.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0440.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0441.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0442.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0443.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0444.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0445.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0446.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0447.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0448.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0449.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0450.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0451.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0452.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0453.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0454.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0455.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0456.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0457.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0458.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0459.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0460.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0461.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0462.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0463.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0464.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0465.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0466.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0467.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0468.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0469.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0470.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0471.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0472.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0473.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0474.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0475.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0476.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0477.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0478.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0479.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0480.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0481.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0482.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0483.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0484.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0485.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0486.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0487.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0488.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0489.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0490.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0491.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0492.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0493.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0494.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0495.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0496.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0497.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0498.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0499.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0500.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0501.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0502.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0503.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0504.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0505.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0506.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0507.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0508.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0509.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0510.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0511.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0512.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0513.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0514.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0515.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0516.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0517.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0518.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0519.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0520.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0521.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0522.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0523.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0524.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0525.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0526.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0527.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0528.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0529.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0530.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0531.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0532.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0533.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0534.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0535.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0536.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0537.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0538.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0539.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0540.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0541.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0542.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0543.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0544.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0545.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0546.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0547.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0548.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0549.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0550.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0551.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0552.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0553.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0554.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0555.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0556.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0557.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0558.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0559.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0560.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0561.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0562.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0563.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0564.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0565.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0566.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0567.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0568.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0569.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0570.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0571.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0572.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0573.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0574.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0575.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0576.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0577.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0578.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0579.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0580.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0581.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0582.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0583.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0584.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0585.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0586.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0587.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0588.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0589.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0590.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0591.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0592.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0593.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0594.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0595.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0596.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0597.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0598.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0599.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0600.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0601.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0602.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0603.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0604.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0605.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0606.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0607.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0608.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0609.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0610.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0611.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0612.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0613.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0614.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0615.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0616.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0617.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0618.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0619.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0620.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0621.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0622.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0623.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0624.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0625.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0626.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0627.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0628.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0629.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0630.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0631.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0632.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0633.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0634.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0635.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0636.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0637.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0638.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0639.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0640.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0641.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0642.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0643.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0644.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0645.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0646.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0647.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0648.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0649.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0650.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0651.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0652.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0653.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0654.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0655.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0656.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0657.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0658.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0659.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0660.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0661.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0662.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0663.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0664.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0665.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0666.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0667.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0668.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0669.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0670.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0671.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0672.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0673.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0674.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0675.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0676.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0677.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0678.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0679.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0680.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0681.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0682.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0683.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0684.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0685.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0686.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0687.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0688.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0689.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0690.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0691.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0692.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0693.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0694.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0695.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0696.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0697.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0698.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0699.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0700.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0701.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0702.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0703.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0704.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0705.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0706.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0707.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0708.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0709.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0710.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0711.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0712.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0713.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0714.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0715.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0716.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0717.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0718.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0719.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0720.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0721.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0722.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0723.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0724.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0725.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0726.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0727.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0728.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0729.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0730.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0731.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0732.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0733.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0734.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0735.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0736.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0737.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0738.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0739.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0740.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0741.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0742.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0743.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0744.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0745.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0746.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0747.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0748.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0749.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0750.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0751.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0752.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0753.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0754.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0755.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0756.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0757.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0758.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0759.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0760.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0761.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0762.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0763.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0764.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0765.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0766.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0767.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0768.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0769.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0770.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0771.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0772.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0773.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0774.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0775.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0776.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0777.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0778.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0779.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0780.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0781.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0782.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0783.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0784.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0785.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0786.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0787.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0788.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0789.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0790.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0791.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0792.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0793.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0794.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0795.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0796.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0797.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0798.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0799.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0800.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0801.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0802.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0803.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0804.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0805.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0806.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0807.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0808.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0809.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0810.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0811.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0812.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0813.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0814.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0815.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0816.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0817.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0818.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0819.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0820.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0821.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0822.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0823.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0824.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0825.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0826.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0827.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0828.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0829.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0830.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0831.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0832.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0833.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0834.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0835.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0836.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0837.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0838.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0839.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0840.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0841.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0842.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0843.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0844.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0845.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0846.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0847.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0848.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0849.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0850.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0851.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0852.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0853.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0854.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0855.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0856.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0857.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0858.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0859.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0860.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0861.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0862.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0863.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0864.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0865.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0866.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0867.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0868.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0869.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0870.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0871.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0872.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0873.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0874.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0875.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0876.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0877.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0878.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0879.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0880.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0881.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0882.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0883.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0884.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0885.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0886.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0887.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0888.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0889.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0890.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0891.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0892.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0893.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0894.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0895.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0896.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0897.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0898.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0899.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0900.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0901.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0902.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0903.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0904.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0905.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0906.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0907.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0908.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0909.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0910.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0911.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0912.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0913.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0914.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0915.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0916.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0917.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0918.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0919.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0920.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0921.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0922.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0923.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0924.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0925.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0926.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0927.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0928.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0929.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0930.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0931.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0932.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0933.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0934.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0935.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0936.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0937.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0938.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0939.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0940.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0941.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0942.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0943.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0944.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0945.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0946.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0947.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0948.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0949.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0950.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0951.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0952.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0953.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0954.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0955.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0956.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0957.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0958.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0959.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0960.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0961.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0962.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0963.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0964.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0965.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0966.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0967.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0968.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0969.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0970.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0971.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0972.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0973.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0974.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0975.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0976.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0977.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0978.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0979.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0980.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0981.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0982.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0983.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0984.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0985.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0986.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0987.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0988.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0989.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0990.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0991.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0992.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0993.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0994.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0995.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0996.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0997.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0998.part',\n",
       " 'coiled-datasets/synthetic-data/synth-reg-104GB.csv/0999.part',\n",
       " ...]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "ddf.to_csv(\"s3://coiled-datasets/synthetic-data/synth-reg-104GB.csv\", \n",
    "           single_file=False, \n",
    "           index=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fee760d-77e2-4f3a-9aa0-e5097935a430",
   "metadata": {},
   "source": [
    "Note that writing this DataFrame to Parquet is much faster. Generally speaking, we recommend working with Parquet files when using Dask unless you have very strong reasons not to do so. Read this blog to learn more about the benefits of the Parquet file format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "68088a09-7517-4d08-abe1-276b5bc1b345",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.27 s, sys: 243 ms, total: 2.51 s\n",
      "Wall time: 4min 35s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "distributed.client - ERROR - Failed to reconnect to scheduler after 30.00 seconds, closing client\n",
      "distributed.client - ERROR - Failed to reconnect to scheduler after 30.00 seconds, closing client\n",
      "distributed.client - ERROR - Failed to reconnect to scheduler after 30.00 seconds, closing client\n",
      "_GatheringFuture exception was never retrieved\n",
      "future: <_GatheringFuture finished exception=CancelledError()>\n",
      "asyncio.exceptions.CancelledError\n",
      "_GatheringFuture exception was never retrieved\n",
      "future: <_GatheringFuture finished exception=CancelledError()>\n",
      "asyncio.exceptions.CancelledError\n",
      "_GatheringFuture exception was never retrieved\n",
      "future: <_GatheringFuture finished exception=CancelledError()>\n",
      "asyncio.exceptions.CancelledError\n",
      "distributed.deploy.cluster - WARNING - Failed to sync cluster info multiple times - perhaps there's a connection issue? Error:\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/site-packages/distributed/comm/tcp.py\", line 398, in connect\n",
      "    stream = await self.client.connect(\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/site-packages/tornado/tcpclient.py\", line 275, in connect\n",
      "    af, addr, stream = await connector.start(connect_timeout=timeout)\n",
      "asyncio.exceptions.CancelledError\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/asyncio/tasks.py\", line 492, in wait_for\n",
      "    fut.result()\n",
      "asyncio.exceptions.CancelledError\n",
      "\n",
      "The above exception was the direct cause of the following exception:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/site-packages/distributed/comm/core.py\", line 284, in connect\n",
      "    comm = await asyncio.wait_for(\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/asyncio/tasks.py\", line 494, in wait_for\n",
      "    raise exceptions.TimeoutError() from exc\n",
      "asyncio.exceptions.TimeoutError\n",
      "\n",
      "The above exception was the direct cause of the following exception:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/site-packages/distributed/deploy/cluster.py\", line 115, in _sync_cluster_info\n",
      "    await self.scheduler_comm.set_metadata(\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/site-packages/distributed/core.py\", line 817, in send_recv_from_rpc\n",
      "    comm = await self.live_comm()\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/site-packages/distributed/core.py\", line 774, in live_comm\n",
      "    comm = await connect(\n",
      "  File \"/Users/rpelgrim/mambaforge/envs/coiled-base/lib/python3.9/site-packages/distributed/comm/core.py\", line 308, in connect\n",
      "    raise OSError(\n",
      "OSError: Timed out trying to connect to tls://3.234.205.253:8786 after 30 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ddf.to_parquet(\"s3://coiled-datasets/synthetic-data/synth-reg-104GB_2.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efcfdb1e-37cc-498c-8b1d-fb751c644460",
   "metadata": {},
   "source": [
    "## Writing Parquet Files to CSV with Dask Summary\n",
    "\n",
    "- You can use Dask to convert multiple Parquet files into a single CSV file\n",
    "- With Dask you don’t have to worry about file sizes or memory errors\n",
    "- Dask supports reading and writing from cloud-based data storage\n",
    "- Dask enables you to work faster by reading and writing in parallel\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6ca9c62-c6c5-4c94-bfb7-070223419b02",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "coiled-base",
   "language": "python",
   "name": "coiled-base"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}