Julia/CSV/Read_CSV.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading a CSV file to a DataFrame in Julia (programing lang)\n",
    "\n",
    "Article: https://towardsdatascience.com/read-csv-to-data-frame-in-julia-programming-lang-77f3d0081c14\n",
    "\n",
    "Julia often offer several ways how to do the same thing and reading CSV is an example. In all cases, you will need the `CSV` and `DataFrames` package. If you don't have them installed, in the Julia REPL run: `import Pkg; Pkg.add(\"CSV\"); Pkg.add(\"DataFrames\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "v\"1.4.1\""
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "VERSION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries; you can also use import, but there's slight difference in the behavior. \n",
    "# for example you would have to import DataFrames.DataFrame\n",
    "using CSV\n",
    "using DataFrames\n",
    "using StringEncodings\n",
    "using Dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th></tr><tr><th></th><th>Int64</th><th>String</th></tr></thead><tbody><p>4 rows × 2 columns</p><tr><th>1</th><td>1</td><td>M</td></tr><tr><th>2</th><td>2</td><td>F</td></tr><tr><th>3</th><td>3</td><td>F</td></tr><tr><th>4</th><td>4</td><td>M</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& A & B\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & M \\\\\n",
       "\t2 & 2 & F \\\\\n",
       "\t3 & 3 & F \\\\\n",
       "\t4 & 4 & M \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "4×2 DataFrame\n",
       "│ Row │ A     │ B      │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼───────┼────────┤\n",
       "│ 1   │ 1     │ M      │\n",
       "│ 2   │ 2     │ F      │\n",
       "│ 3   │ 3     │ F      │\n",
       "│ 4   │ 4     │ M      │"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DataFrame(A = 1:4, B = [\"M\", \"F\", \"F\", \"M\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `CSV.File` will return `CSV.File` object, which you can iterate to get `CSV.Row`s. See complete documentation of the [Julia CSV parser](https://csv.juliadata.org/stable/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CSV.File{false}\n"
     ]
    }
   ],
   "source": [
    "csv_reader = CSV.File(\"file.csv\")\n",
    "println(typeof(csv_reader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CSV.Row\n",
      "CSV.Row\n"
     ]
    }
   ],
   "source": [
    "for row in csv_reader\n",
    "    println(typeof(row))\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great thing is that CSV support Table.jl interface which allow to easilly access the columns using dot notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "values: A, 12, 2.0\n",
      "values: B, 22, 5.1\n"
     ]
    }
   ],
   "source": [
    "for row in csv_reader\n",
    "    println(\"values: $(row.col1), $(row.col2), $(row.col3)\")\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CSV.File `header` parameter has default value `1` (in Julia it mean the first row) and `\",\"` for the delimiter. But you can always specify them manually. By convention kwargs (key-value parameters) are passed after the semicolon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element CSV.File{false}:\n",
       " CSV.Row: (col1 = \"A\", col2 = 12, col3 = 2.0)\n",
       " CSV.Row: (col1 = \"B\", col2 = 22, col3 = 5.1)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.File(\"file.csv\"; header=1, delim=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comma is also acceptable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element CSV.File{false}:\n",
       " CSV.Row: (col1 = \"A\", col2 = 12, col3 = 2.0)\n",
       " CSV.Row: (col1 = \"B\", col2 = 22, col3 = 5.1)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.File(\"file.csv\", header=1, delim=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read CSV option 1\n",
    "Load the data from CSV.File (Julia reader) and pass to the DataFrame function.\n",
    "\n",
    "`DataFrame(CSV.File(\"file.csv\"); kwargs)`\n",
    "\n",
    "By default the CSV.File will try to detect the delimiter from the first 10 lines. Default is `\",\"`. Notice that keyword argumets (kwargs) are separated by semicolon `;` by convention, though comma `,` works too. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>col1</th><th>col2</th><th>col3</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>A</td><td>12</td><td>2.0</td></tr><tr><th>2</th><td>B</td><td>22</td><td>5.1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& col1 & col2 & col3\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & A & 12 & 2.0 \\\\\n",
       "\t2 & B & 22 & 5.1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrame\n",
       "│ Row │ col1   │ col2  │ col3    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼─────────┤\n",
       "│ 1   │ A      │ 12    │ 2.0     │\n",
       "│ 2   │ B      │ 22    │ 5.1     │"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = DataFrame(CSV.File(\"file.csv\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read CSV option 2\n",
    "You can also read the csv file and pipe it to the `DataFrames.DataFrame` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>col1</th><th>col2</th><th>col3</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>A</td><td>12</td><td>2.0</td></tr><tr><th>2</th><td>B</td><td>22</td><td>5.1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& col1 & col2 & col3\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & A & 12 & 2.0 \\\\\n",
       "\t2 & B & 22 & 5.1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrame\n",
       "│ Row │ col1   │ col2  │ col3    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼─────────┤\n",
       "│ 1   │ A      │ 12    │ 2.0     │\n",
       "│ 2   │ B      │ 22    │ 5.1     │"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = CSV.File(\"file.csv\") |> DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read CSV option 3\n",
    "To be similar to other languages, there's `CSV.read(file,DataFrame; kwargs)`, syntactic sugar. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>col1</th><th>col2</th><th>col3</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>A</td><td>12</td><td>2.0</td></tr><tr><th>2</th><td>B</td><td>22</td><td>5.1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& col1 & col2 & col3\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & A & 12 & 2.0 \\\\\n",
       "\t2 & B & 22 & 5.1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrame\n",
       "│ Row │ col1   │ col2  │ col3    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼─────────┤\n",
       "│ 1   │ A      │ 12    │ 2.0     │\n",
       "│ 2   │ B      │ 22    │ 5.1     │"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = CSV.read(\"file.csv\", DataFrame; ignoreemptylines=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some older tutorials show `.read()` method without `DataFrame` argument which now leads to an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "ArgumentError",
     "evalue": "ArgumentError: provide a valid sink argument, like `using DataFrames; CSV.read(source, DataFrame)`",
     "output_type": "error",
     "traceback": [
      "ArgumentError: provide a valid sink argument, like `using DataFrames; CSV.read(source, DataFrame)`",
      "",
      "Stacktrace:",
      " [1] read(::String, ::Nothing; copycols::Bool, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/vaclav/.julia/packages/CSV/la2cd/src/CSV.jl:43",
      " [2] read(::String, ::Nothing) at /home/vaclav/.julia/packages/CSV/la2cd/src/CSV.jl:42 (repeats 2 times)",
      " [3] top-level scope at In[12]:1"
     ]
    }
   ],
   "source": [
    "CSV.read(\"file.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding\n",
    "You can deal with the non-UTF-8 encoding, by using `read` method and specifying the encoding. Encoding fall under the [StringEncodings.jl](https://github.com/JuliaStrings/StringEncodings.jl) which must first be imported by `using StringEncodings`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>col1</th><th>col2</th><th>col3</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>\\xc8</td><td>12</td><td>2.0</td></tr><tr><th>2</th><td>\\xf8</td><td>22</td><td>5.1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& col1 & col2 & col3\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & \\textbackslash{}xc8 & 12 & 2.0 \\\\\n",
       "\t2 & \\textbackslash{}xf8 & 22 & 5.1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrame\n",
       "│ Row │ col1   │ col2  │ col3    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼─────────┤\n",
       "│ 1   │ \\xc8   │ 12    │ 2.0     │\n",
       "│ 2   │ \\xf8   │ 22    │ 5.1     │"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# letter in the column `col1` are encoded in windows-1250\n",
    "DataFrame(CSV.File(\"file_encoding.csv\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to create an instance of Encoding type, you can use `enc` shortcut in front of encodings string name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>col1</th><th>col2</th><th>col3</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>Č</td><td>12</td><td>2.0</td></tr><tr><th>2</th><td>ř</td><td>22</td><td>5.1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& col1 & col2 & col3\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & Č & 12 & 2.0 \\\\\n",
       "\t2 & ř & 22 & 5.1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrame\n",
       "│ Row │ col1   │ col2  │ col3    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼─────────┤\n",
       "│ 1   │ Č      │ 12    │ 2.0     │\n",
       "│ 2   │ ř      │ 22    │ 5.1     │"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DataFrame(CSV.File(open(read,\"file_encoding.csv\", enc\"windows-1250\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>col1</th><th>col2</th><th>col3</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>Č</td><td>12</td><td>2.0</td></tr><tr><th>2</th><td>ř</td><td>22</td><td>5.1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& col1 & col2 & col3\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & Č & 12 & 2.0 \\\\\n",
       "\t2 & ř & 22 & 5.1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrame\n",
       "│ Row │ col1   │ col2  │ col3    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼─────────┤\n",
       "│ 1   │ Č      │ 12    │ 2.0     │\n",
       "│ 2   │ ř      │ 22    │ 5.1     │"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open(read,\"file_encoding.csv\", enc\"windows-1250\") |> CSV.File |> DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>col1</th><th>col2</th><th>col3</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>Č</td><td>12</td><td>2.0</td></tr><tr><th>2</th><td>ř</td><td>22</td><td>5.1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& col1 & col2 & col3\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & Č & 12 & 2.0 \\\\\n",
       "\t2 & ř & 22 & 5.1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrame\n",
       "│ Row │ col1   │ col2  │ col3    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼─────────┤\n",
       "│ 1   │ Č      │ 12    │ 2.0     │\n",
       "│ 2   │ ř      │ 22    │ 5.1     │"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DataFrame(CSV.File(read(\"file_encoding.csv\", enc\"windows-1250\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>col1</th><th>col2</th><th>col3</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>Č</td><td>12</td><td>2.0</td></tr><tr><th>2</th><td>ř</td><td>22</td><td>5.1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& col1 & col2 & col3\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & Č & 12 & 2.0 \\\\\n",
       "\t2 & ř & 22 & 5.1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrame\n",
       "│ Row │ col1   │ col2  │ col3    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼─────────┤\n",
       "│ 1   │ Č      │ 12    │ 2.0     │\n",
       "│ 2   │ ř      │ 22    │ 5.1     │"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using full syntax\n",
    "DataFrame(CSV.File(read(\"file_encoding.csv\", Encoding(\"windows-1250\"))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV.File parameters\n",
    "Often you don't have goldplated csv and you have to define a few parameters of the reader. You can read about all the parameters in the [CSV.jl documentation](https://csv.juliadata.org/stable/). Key parameters are:\n",
    "* `delim` - the delimiter (separator), which can be char or string.\n",
    "* `header` - which row is header or used to rename the headers\n",
    "* `select` - include only these columns (identify by int, string or symbol)\n",
    "* `drop` - inverse of select, which column to drop\n",
    "* `skipto` or `datarow` - at which which row the data start\n",
    "* `footerskip` - rows to skip at the end of the file\n",
    "* `limit` - how many rows to read (only reliable if `threaded=false`)\n",
    "* `type` - single type to use to parse whole file\n",
    "* `types` - specify type for each column\n",
    "\n",
    "There are few other methods, mainly related to error handling in [CSV.jl documentation](https://csv.juliadata.org/stable/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c1 & c2 & c3 & c4\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & 1.5 \\\\\n",
       "\t2 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c1     │ c2    │ c3     │ c4      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2     │ c      │ 1.5     │\n",
       "│ 2   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"\n",
    "\n",
    "c1|c2|c3|c4\n",
    "\"1\"|2|c|1.5\n",
    "\"C|D\"|16|x|2.33\n",
    "\"\"\"\n",
    "\n",
    "CSV.read(IOBuffer(data), DataFrame; delim='|', skipto=2, quotechar='\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select paramter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c2</th></tr><tr><th></th><th>Int64</th></tr></thead><tbody><p>2 rows × 1 columns</p><tr><th>1</th><td>2</td></tr><tr><th>2</th><td>16</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|c}\n",
       "\t& c2\\\\\n",
       "\t\\hline\n",
       "\t& Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 2 \\\\\n",
       "\t2 & 16 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×1 DataFrame\n",
       "│ Row │ c2    │\n",
       "│     │ \u001b[90mInt64\u001b[39m │\n",
       "├─────┼───────┤\n",
       "│ 1   │ 2     │\n",
       "│ 2   │ 16    │"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select only the second column (Julia list start with 1)\n",
    "CSV.read(IOBuffer(data), DataFrame;select=[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c2</th><th>c4</th></tr><tr><th></th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>2</td><td>1.5</td></tr><tr><th>2</th><td>16</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& c2 & c4\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 2 & 1.5 \\\\\n",
       "\t2 & 16 & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×2 DataFrame\n",
       "│ Row │ c2    │ c4      │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼───────┼─────────┤\n",
       "│ 1   │ 2     │ 1.5     │\n",
       "│ 2   │ 16    │ 2.33    │"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select using string\n",
    "CSV.read(IOBuffer(data), DataFrame;select=[\"c2\",\"c4\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th></tr><tr><th></th><th>String</th><th>Int64</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>1</td><td>2</td></tr><tr><th>2</th><td>C|D</td><td>16</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& c1 & c2\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 \\\\\n",
       "\t2 & C|D & 16 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×2 DataFrame\n",
       "│ Row │ c1     │ c2    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │\n",
       "├─────┼────────┼───────┤\n",
       "│ 1   │ 1      │ 2     │\n",
       "│ 2   │ C|D    │ 16    │"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select using symbol (symbols identify the columns and have : notation in Julia)\n",
    "CSV.read(IOBuffer(data), DataFrame;select=[:c1,:c2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c2</th><th>c3</th></tr><tr><th></th><th>Int64</th><th>String</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>2</td><td>c</td></tr><tr><th>2</th><td>16</td><td>x</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& c2 & c3\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 2 & c \\\\\n",
       "\t2 & 16 & x \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×2 DataFrame\n",
       "│ Row │ c2    │ c3     │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼───────┼────────┤\n",
       "│ 1   │ 2     │ c      │\n",
       "│ 2   │ 16    │ x      │"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select using symbol with Symbol constuctor\n",
    "CSV.read(IOBuffer(data), DataFrame;select=[Symbol(\"c3\"),:c2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drop parameter\n",
    "Drop is the opposite of select. It says which columns to drop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c4</th></tr><tr><th></th><th>String</th><th>Float64</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>1</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& c1 & c4\\\\\n",
       "\t\\hline\n",
       "\t& String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 1.5 \\\\\n",
       "\t2 & C|D & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×2 DataFrame\n",
       "│ Row │ c1     │ c4      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 1.5     │\n",
       "│ 2   │ C|D    │ 2.33    │"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.read(IOBuffer(data), DataFrame;drop=[Symbol(\"c3\"),:c2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Header parameter\n",
    "You can specify \n",
    "\n",
    "* row which is header (it can be more than 1 row, e.g. range 1:2)\n",
    "* there's no header\n",
    "* rename the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Header parameter of the Julia CSV parser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will parse the following data using the IOBuffer to pass them to the CSV reader. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"c|c|c|d\\n1|2|3|4\\n\\\"1\\\"|2|c|1.5\\n\\\"C|D\\\"|16|x|2.33\\n\""
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"\n",
    "c|c|c|d\n",
    "1|2|3|4\n",
    "\"1\"|2|c|1.5\n",
    "\"C|D\"|16|x|2.33\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Default header on the first row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c</th><th>c_1</th><th>c_2</th><th>d</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th></tr></thead><tbody><p>3 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>3</td><td>4.0</td></tr><tr><th>2</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>3</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c & c\\_1 & c\\_2 & d\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & 3 & 4.0 \\\\\n",
       "\t2 & 1 & 2 & c & 1.5 \\\\\n",
       "\t3 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "3×4 DataFrame\n",
       "│ Row │ c      │ c_1   │ c_2    │ d       │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2     │ 3      │ 4.0     │\n",
       "│ 2   │ 1      │ 2     │ c      │ 1.5     │\n",
       "│ 3   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# default header is the first row, duplicated column names are postfixed\n",
    "df = CSV.read(IOBuffer(data), DataFrame)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### No header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>Column1</th><th>Column2</th><th>Column3</th><th>Column4</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>String</th></tr></thead><tbody><p>4 rows × 4 columns</p><tr><th>1</th><td>c</td><td>c</td><td>c</td><td>d</td></tr><tr><th>2</th><td>1</td><td>2</td><td>3</td><td>4</td></tr><tr><th>3</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>4</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& Column1 & Column2 & Column3 & Column4\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & String\\\\\n",
       "\t\\hline\n",
       "\t1 & c & c & c & d \\\\\n",
       "\t2 & 1 & 2 & 3 & 4 \\\\\n",
       "\t3 & 1 & 2 & c & 1.5 \\\\\n",
       "\t4 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "4×4 DataFrame\n",
       "│ Row │ Column1 │ Column2 │ Column3 │ Column4 │\n",
       "│     │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │\n",
       "├─────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│ 1   │ c       │ c       │ c       │ d       │\n",
       "│ 2   │ 1       │ 2       │ 3       │ 4       │\n",
       "│ 3   │ 1       │ 2       │ c       │ 1.5     │\n",
       "│ 4   │ C|D     │ 16      │ x       │ 2.33    │"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Julia convention is to separate the key arguments (kwargs) by semicolon (;). Comma though works as well\n",
    "CSV.read(IOBuffer(data), DataFrame; header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>Column1</th><th>Column2</th><th>Column3</th><th>Column4</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>String</th></tr></thead><tbody><p>4 rows × 4 columns</p><tr><th>1</th><td>c</td><td>c</td><td>c</td><td>d</td></tr><tr><th>2</th><td>1</td><td>2</td><td>3</td><td>4</td></tr><tr><th>3</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>4</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& Column1 & Column2 & Column3 & Column4\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & String\\\\\n",
       "\t\\hline\n",
       "\t1 & c & c & c & d \\\\\n",
       "\t2 & 1 & 2 & 3 & 4 \\\\\n",
       "\t3 & 1 & 2 & c & 1.5 \\\\\n",
       "\t4 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "4×4 DataFrame\n",
       "│ Row │ Column1 │ Column2 │ Column3 │ Column4 │\n",
       "│     │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │\n",
       "├─────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│ 1   │ c       │ c       │ c       │ d       │\n",
       "│ 2   │ 1       │ 2       │ 3       │ 4       │\n",
       "│ 3   │ 1       │ 2       │ c       │ 1.5     │\n",
       "│ 4   │ C|D     │ 16      │ x       │ 2.33    │"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.read(IOBuffer(data), DataFrame; header=false)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Specify own column names using headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass the vector of column names (as strings or symbols) to specify the headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>first</th><th>second</th><th>third</th><th>fourth</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>String</th></tr></thead><tbody><p>4 rows × 4 columns</p><tr><th>1</th><td>c</td><td>c</td><td>c</td><td>d</td></tr><tr><th>2</th><td>1</td><td>2</td><td>3</td><td>4</td></tr><tr><th>3</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>4</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& first & second & third & fourth\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & String\\\\\n",
       "\t\\hline\n",
       "\t1 & c & c & c & d \\\\\n",
       "\t2 & 1 & 2 & 3 & 4 \\\\\n",
       "\t3 & 1 & 2 & c & 1.5 \\\\\n",
       "\t4 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "4×4 DataFrame\n",
       "│ Row │ first  │ second │ third  │ fourth │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼────────┼────────┼────────┼────────┤\n",
       "│ 1   │ c      │ c      │ c      │ d      │\n",
       "│ 2   │ 1      │ 2      │ 3      │ 4      │\n",
       "│ 3   │ 1      │ 2      │ c      │ 1.5    │\n",
       "│ 4   │ C|D    │ 16     │ x      │ 2.33   │"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.read(IOBuffer(data), DataFrame; header=[\"first\",\"second\",\"third\",\"fourth\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th><th>c</th><th>d</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>String</th></tr></thead><tbody><p>4 rows × 4 columns</p><tr><th>1</th><td>c</td><td>c</td><td>c</td><td>d</td></tr><tr><th>2</th><td>1</td><td>2</td><td>3</td><td>4</td></tr><tr><th>3</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>4</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& a & b & c & d\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & String\\\\\n",
       "\t\\hline\n",
       "\t1 & c & c & c & d \\\\\n",
       "\t2 & 1 & 2 & 3 & 4 \\\\\n",
       "\t3 & 1 & 2 & c & 1.5 \\\\\n",
       "\t4 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "4×4 DataFrame\n",
       "│ Row │ a      │ b      │ c      │ d      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼────────┼────────┼────────┼────────┤\n",
       "│ 1   │ c      │ c      │ c      │ d      │\n",
       "│ 2   │ 1      │ 2      │ 3      │ 4      │\n",
       "│ 3   │ 1      │ 2      │ c      │ 1.5    │\n",
       "│ 4   │ C|D    │ 16     │ x      │ 2.33   │"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.read(IOBuffer(data), DataFrame; header=[:a,:b,:c,:d])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>z</th><th>zy</th><th>zyx</th><th>zyxw</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>String</th></tr></thead><tbody><p>4 rows × 4 columns</p><tr><th>1</th><td>c</td><td>c</td><td>c</td><td>d</td></tr><tr><th>2</th><td>1</td><td>2</td><td>3</td><td>4</td></tr><tr><th>3</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>4</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& z & zy & zyx & zyxw\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & String\\\\\n",
       "\t\\hline\n",
       "\t1 & c & c & c & d \\\\\n",
       "\t2 & 1 & 2 & 3 & 4 \\\\\n",
       "\t3 & 1 & 2 & c & 1.5 \\\\\n",
       "\t4 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "4×4 DataFrame\n",
       "│ Row │ z      │ zy     │ zyx    │ zyxw   │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼────────┼────────┼────────┼────────┤\n",
       "│ 1   │ c      │ c      │ c      │ d      │\n",
       "│ 2   │ 1      │ 2      │ 3      │ 4      │\n",
       "│ 3   │ 1      │ 2      │ c      │ 1.5    │\n",
       "│ 4   │ C|D    │ 16     │ x      │ 2.33   │"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.read(IOBuffer(data), DataFrame; header=[Symbol(\"z\"),Symbol(\"zy\"),Symbol(\"zyx\"),Symbol(\"zyxw\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Header on the first row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c</th><th>c_1</th><th>c_2</th><th>d</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th></tr></thead><tbody><p>3 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>3</td><td>4.0</td></tr><tr><th>2</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>3</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c & c\\_1 & c\\_2 & d\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & 3 & 4.0 \\\\\n",
       "\t2 & 1 & 2 & c & 1.5 \\\\\n",
       "\t3 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "3×4 DataFrame\n",
       "│ Row │ c      │ c_1   │ c_2    │ d       │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2     │ 3      │ 4.0     │\n",
       "│ 2   │ 1      │ 2     │ c      │ 1.5     │\n",
       "│ 3   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remember that Julia stasts indexing at 1\n",
    "CSV.read(IOBuffer(data), DataFrame; header=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Header on the x-th row\n",
    "Everything above the header is ignored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>1</th><th>2</th><th>3</th><th>4</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& 1 & 2 & 3 & 4\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & 1.5 \\\\\n",
       "\t2 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ 1      │ 2     │ 3      │ 4       │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2     │ c      │ 1.5     │\n",
       "│ 2   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.read(IOBuffer(data), DataFrame; header=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Multirow header\n",
    "Can be specified using range, for example `1:2` or list `[1,2]`. Some rows can even be skipped, e.g. `[1,3]`. The columns names are concatenation of the values on these rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>Column1</th><th>Column2</th><th>Column3</th><th>Column4</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>String</th></tr></thead><tbody><p>4 rows × 4 columns</p><tr><th>1</th><td>c</td><td>c</td><td>c</td><td>d</td></tr><tr><th>2</th><td>1</td><td>2</td><td>3</td><td>4</td></tr><tr><th>3</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>4</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& Column1 & Column2 & Column3 & Column4\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & String\\\\\n",
       "\t\\hline\n",
       "\t1 & c & c & c & d \\\\\n",
       "\t2 & 1 & 2 & 3 & 4 \\\\\n",
       "\t3 & 1 & 2 & c & 1.5 \\\\\n",
       "\t4 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "4×4 DataFrame\n",
       "│ Row │ Column1 │ Column2 │ Column3 │ Column4 │\n",
       "│     │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │ \u001b[90mString\u001b[39m  │\n",
       "├─────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│ 1   │ c       │ c       │ c       │ d       │\n",
       "│ 2   │ 1       │ 2       │ 3       │ 4       │\n",
       "│ 3   │ 1       │ 2       │ c       │ 1.5     │\n",
       "│ 4   │ C|D     │ 16      │ x       │ 2.33    │"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# no headers, to remind you the data\n",
    "df = CSV.read(IOBuffer(data), DataFrame; header=false)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c_1</th><th>c_2</th><th>c_3</th><th>d_4</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c\\_1 & c\\_2 & c\\_3 & d\\_4\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & 1.5 \\\\\n",
       "\t2 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c_1    │ c_2   │ c_3    │ d_4     │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2     │ c      │ 1.5     │\n",
       "│ 2   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# headers are formed as concat of row 1 till row 2\n",
    "CSV.read(IOBuffer(data), DataFrame; header=1:2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c_1</th><th>c_2</th><th>c_3</th><th>d_4</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c\\_1 & c\\_2 & c\\_3 & d\\_4\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & 1.5 \\\\\n",
       "\t2 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c_1    │ c_2   │ c_3    │ d_4     │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2     │ c      │ 1.5     │\n",
       "│ 2   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the same using the list\n",
    "CSV.read(IOBuffer(data), DataFrame; header=[1,2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c_1</th><th>c_2</th><th>c_c</th><th>d_1.5</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th></tr></thead><tbody><p>1 rows × 4 columns</p><tr><th>1</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c\\_1 & c\\_2 & c\\_c & d\\_1.5\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "1×4 DataFrame\n",
       "│ Row │ c_1    │ c_2   │ c_c    │ d_1.5   │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# skipping row number 2\n",
    "CSV.read(IOBuffer(data), DataFrame; header=[1,3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Type and Types\n",
    "* `type` - set the same type to all columns\n",
    "* `types` - Vector or Dict of types for each column\n",
    "\n",
    "More about [Julia types](https://docs.julialang.org/en/v1/manual/types/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All examples are based on string input, which is passed to Julia's CSV reader through `IOBuffer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"c1|c2|c3|c4\\n\\\"1\\\"|2|c|1.5\\n\\\"C|D\\\"|16|x|2.33\\n\""
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"c1|c2|c3|c4\n",
    "\"1\"|2|c|1.5\n",
    "\"C|D\"|16|x|2.33\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can set the same type for all columns using `type` parameter, e.g. string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>String</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c1 & c2 & c3 & c4\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & 1.5 \\\\\n",
       "\t2 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c1     │ c2     │ c3     │ c4     │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼────────┼────────┼────────┼────────┤\n",
       "│ 1   │ 1      │ 2      │ c      │ 1.5    │\n",
       "│ 2   │ C|D    │ 16     │ x      │ 2.33   │"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# type turns all the columns to the same type\n",
    "CSV.read(IOBuffer(data), DataFrame; type=String)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or specify type for each or just some columns using a Dict. If the data cannot be parsed to the type, it's turned to `missing` type, equivalent of pandas's `Nan`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: thread = 1 warning: error parsing Int64 around row = 2, col = 4: \"1.5\n",
      "│ \", error=INVALID: OK | NEWLINE | INVALID_DELIMITER \n",
      "└ @ CSV /home/vaclav/.julia/packages/CSV/la2cd/src/file.jl:606\n",
      "┌ Warning: thread = 1 warning: error parsing Int64 around row = 3, col = 4: \"2.33\n",
      "│ \", error=INVALID: OK | NEWLINE | EOF | INVALID_DELIMITER \n",
      "└ @ CSV /home/vaclav/.julia/packages/CSV/la2cd/src/file.jl:606\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CSV.Row:\n",
      " :c1  \"1\"\n",
      " :c2  \"2\"\n",
      " :c3  \"c\"\n",
      " :c4  missing\n",
      "CSV.Row:\n",
      " :c1  \"C|D\"\n",
      " :c2  \"16\"\n",
      " :c3  \"x\"\n",
      " :c4  missing\n"
     ]
    }
   ],
   "source": [
    "for r in CSV.File(IOBuffer(data), types=Dict(:c2=>String, :c4=>Int64))\n",
    "    println(r)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: thread = 1 warning: error parsing Int64 around row = 2, col = 4: \"1.5\n",
      "│ \", error=INVALID: OK | NEWLINE | INVALID_DELIMITER \n",
      "└ @ CSV /home/vaclav/.julia/packages/CSV/la2cd/src/file.jl:606\n",
      "┌ Warning: thread = 1 warning: error parsing Int64 around row = 3, col = 4: \"2.33\n",
      "│ \", error=INVALID: OK | NEWLINE | EOF | INVALID_DELIMITER \n",
      "└ @ CSV /home/vaclav/.julia/packages/CSV/la2cd/src/file.jl:606\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>Int64?</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td><em>missing</em></td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td><em>missing</em></td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c1 & c2 & c3 & c4\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & Int64?\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & \\emph{missing} \\\\\n",
       "\t2 & C|D & 16 & x & \\emph{missing} \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c1     │ c2     │ c3     │ c4      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mInt64?\u001b[39m  │\n",
       "├─────┼────────┼────────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2      │ c      │ \u001b[90mmissing\u001b[39m │\n",
       "│ 2   │ C|D    │ 16     │ x      │ \u001b[90mmissing\u001b[39m │"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# specify types of the columns\n",
    "CSV.read(IOBuffer(data), DataFrame; types=Dict(:c2=>String, :c4=>Int64))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can silence these warnings by `silencewarnings=true`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>Int64?</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td><em>missing</em></td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td><em>missing</em></td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c1 & c2 & c3 & c4\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & Int64?\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & \\emph{missing} \\\\\n",
       "\t2 & C|D & 16 & x & \\emph{missing} \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c1     │ c2     │ c3     │ c4      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mInt64?\u001b[39m  │\n",
       "├─────┼────────┼────────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2      │ c      │ \u001b[90mmissing\u001b[39m │\n",
       "│ 2   │ C|D    │ 16     │ x      │ \u001b[90mmissing\u001b[39m │"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# specify types of the columns\n",
    "CSV.read(IOBuffer(data), DataFrame; types=Dict(:c2=>String, :c4=>Int64), silencewarnings=true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th></tr><tr><th></th><th>String</th><th>String</th><th>String</th><th>Float32</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c1 & c2 & c3 & c4\\\\\n",
       "\t\\hline\n",
       "\t& String & String & String & Float32\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & 1.5 \\\\\n",
       "\t2 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c1     │ c2     │ c3     │ c4      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat32\u001b[39m │\n",
       "├─────┼────────┼────────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2      │ c      │ 1.5     │\n",
       "│ 2   │ C|D    │ 16     │ x      │ 2.33    │"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# specify valid type for columns\n",
    "CSV.read(IOBuffer(data), DataFrame; types=Dict(:c2=>String, :c4=>Float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or specify types for all columns using a **Vector**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c1 & c2 & c3 & c4\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & 1.5 \\\\\n",
       "\t2 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c1     │ c2    │ c3     │ c4      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2     │ c      │ 1.5     │\n",
       "│ 2   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# specify Array with type\n",
    "types = Array{DataType,1}([String, Int, String, Float64])\n",
    "CSV.read(IOBuffer(data), DataFrame; types=types)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{DataType,1}:\n",
       " String\n",
       " Int32\n",
       " String\n",
       " Float32"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[String, Int32, String, Float32]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th></tr><tr><th></th><th>String</th><th>Int32</th><th>String</th><th>Float32</th></tr></thead><tbody><p>2 rows × 4 columns</p><tr><th>1</th><td>1</td><td>2</td><td>c</td><td>1.5</td></tr><tr><th>2</th><td>C|D</td><td>16</td><td>x</td><td>2.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& c1 & c2 & c3 & c4\\\\\n",
       "\t\\hline\n",
       "\t& String & Int32 & String & Float32\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2 & c & 1.5 \\\\\n",
       "\t2 & C|D & 16 & x & 2.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×4 DataFrame\n",
       "│ Row │ c1     │ c2    │ c3     │ c4      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt32\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat32\u001b[39m │\n",
       "├─────┼────────┼───────┼────────┼─────────┤\n",
       "│ 1   │ 1      │ 2     │ c      │ 1.5     │\n",
       "│ 2   │ C|D    │ 16    │ x      │ 2.33    │"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or just pass the array\n",
    "CSV.read(IOBuffer(data), DataFrame; types=[String, Int32, String, Float32])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Date Formats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th><th>d1</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th><th>Date</th></tr></thead><tbody><p>2 rows × 5 columns</p><tr><th>1</th><td>XY</td><td>2</td><td>c</td><td>1.5</td><td>2020-01-05</td></tr><tr><th>2</th><td>AB</td><td>16</td><td>x</td><td>2.33</td><td>2021-01-05</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& c1 & c2 & c3 & c4 & d1\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64 & Date\\\\\n",
       "\t\\hline\n",
       "\t1 & XY & 2 & c & 1.5 & 2020-01-05 \\\\\n",
       "\t2 & AB & 16 & x & 2.33 & 2021-01-05 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×5 DataFrame\n",
       "│ Row │ c1     │ c2    │ c3     │ c4      │ d1         │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mDate\u001b[39m       │\n",
       "├─────┼────────┼───────┼────────┼─────────┼────────────┤\n",
       "│ 1   │ XY     │ 2     │ c      │ 1.5     │ 2020-01-05 │\n",
       "│ 2   │ AB     │ 16    │ x      │ 2.33    │ 2021-01-05 │"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"c1|c2|c3|c4|d1\n",
    "\"XY\"|2|c|1.5|2020-01-05\n",
    "\"AB\"|16|x|2.33|2021-01-05\n",
    "\"\"\"\n",
    "\n",
    "CSV.read(IOBuffer(data), DataFrame; \n",
    "    dateformat=\"yyyy-mm-dd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th><th>d1</th><th>d2</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th><th>Date</th><th>Date</th></tr></thead><tbody><p>2 rows × 6 columns</p><tr><th>1</th><td>XY</td><td>2</td><td>c</td><td>1.5</td><td>2020-01-05</td><td>0020-12-01</td></tr><tr><th>2</th><td>AB</td><td>16</td><td>x</td><td>2.33</td><td>2021-01-05</td><td>0020-10-15</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccc}\n",
       "\t& c1 & c2 & c3 & c4 & d1 & d2\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64 & Date & Date\\\\\n",
       "\t\\hline\n",
       "\t1 & XY & 2 & c & 1.5 & 2020-01-05 & 0020-12-01 \\\\\n",
       "\t2 & AB & 16 & x & 2.33 & 2021-01-05 & 0020-10-15 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×6 DataFrame\n",
       "│ Row │ c1     │ c2    │ c3     │ c4      │ d1         │ d2         │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mDate\u001b[39m       │ \u001b[90mDate\u001b[39m       │\n",
       "├─────┼────────┼───────┼────────┼─────────┼────────────┼────────────┤\n",
       "│ 1   │ XY     │ 2     │ c      │ 1.5     │ 2020-01-05 │ 0020-12-01 │\n",
       "│ 2   │ AB     │ 16    │ x      │ 2.33    │ 2021-01-05 │ 0020-10-15 │"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"c1|c2|c3|c4|d1|d2\n",
    "\"XY\"|2|c|1.5|2020-01-05|01/12/20\n",
    "\"AB\"|16|x|2.33|2021-01-05|15/10/20\n",
    "\"\"\"\n",
    "\n",
    "# specify that columns are dates and then specify the dateformat\n",
    "df = CSV.read(IOBuffer(data), DataFrame; \n",
    "    types=Dict(\"d1\"=>Date, \"d2\"=>Date), \n",
    "    dateformats=Dict(\n",
    "        \"d1\"=>\"yyyy-mm-dd\",\n",
    "        \"d2\"=>\"dd/mm/yy\"\n",
    "    )\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th><th>d1</th><th>d2</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th><th>Date</th><th>Date</th></tr></thead><tbody><p>2 rows × 6 columns</p><tr><th>1</th><td>XY</td><td>2</td><td>c</td><td>1.5</td><td>2020-01-05</td><td>2020-12-01</td></tr><tr><th>2</th><td>AB</td><td>16</td><td>x</td><td>2.33</td><td>2021-01-05</td><td>2020-10-15</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccc}\n",
       "\t& c1 & c2 & c3 & c4 & d1 & d2\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64 & Date & Date\\\\\n",
       "\t\\hline\n",
       "\t1 & XY & 2 & c & 1.5 & 2020-01-05 & 2020-12-01 \\\\\n",
       "\t2 & AB & 16 & x & 2.33 & 2021-01-05 & 2020-10-15 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×6 DataFrame\n",
       "│ Row │ c1     │ c2    │ c3     │ c4      │ d1         │ d2         │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mDate\u001b[39m       │ \u001b[90mDate\u001b[39m       │\n",
       "├─────┼────────┼───────┼────────┼─────────┼────────────┼────────────┤\n",
       "│ 1   │ XY     │ 2     │ c      │ 1.5     │ 2020-01-05 │ 2020-12-01 │\n",
       "│ 2   │ AB     │ 16    │ x      │ 2.33    │ 2021-01-05 │ 2020-10-15 │"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add 2000 years to the column d2 containing 0020-MM-DD\n",
    "# caregul to run this only once, since both df[:, :d2] or df[!, :d2] modifies the column\n",
    "df[!, :d2] += Dates.Year(2000)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Treustrings and Falsestrings\n",
    "These arguments let you set the list of inputs to be considered as true and false boolean values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>b01</th><th>b002</th><th>c1</th><th>c2</th><th>c3</th><th>c4</th><th>d1</th></tr><tr><th></th><th>Bool</th><th>Bool</th><th>String</th><th>Int64</th><th>String</th><th>Float64</th><th>Date</th></tr></thead><tbody><p>2 rows × 7 columns</p><tr><th>1</th><td>1</td><td>0</td><td>XY</td><td>2</td><td>c</td><td>1.5</td><td>2020-01-05</td></tr><tr><th>2</th><td>0</td><td>1</td><td>AB</td><td>16</td><td>x</td><td>2.33</td><td>2021-01-05</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccc}\n",
       "\t& b01 & b002 & c1 & c2 & c3 & c4 & d1\\\\\n",
       "\t\\hline\n",
       "\t& Bool & Bool & String & Int64 & String & Float64 & Date\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 0 & XY & 2 & c & 1.5 & 2020-01-05 \\\\\n",
       "\t2 & 0 & 1 & AB & 16 & x & 2.33 & 2021-01-05 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×7 DataFrame\n",
       "│ Row │ b01  │ b002 │ c1     │ c2    │ c3     │ c4      │ d1         │\n",
       "│     │ \u001b[90mBool\u001b[39m │ \u001b[90mBool\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mDate\u001b[39m       │\n",
       "├─────┼──────┼──────┼────────┼───────┼────────┼─────────┼────────────┤\n",
       "│ 1   │ 1    │ 0    │ XY     │ 2     │ c      │ 1.5     │ 2020-01-05 │\n",
       "│ 2   │ 0    │ 1    │ AB     │ 16    │ x      │ 2.33    │ 2021-01-05 │"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"\n",
    "b01|b002|c1|c2|c3|c4|d1\n",
    "\"t\"|\"fa\"|\"XY\"|2|c|1.5|2020-01-05\n",
    "\"f\"|\"tr\"|\"AB\"|16|x|2.33|2021-01-05\n",
    "\"\"\"\n",
    "\n",
    "CSV.read(IOBuffer(data), DataFrame; \n",
    "    truestrings=[\"t\",\"tr\"],\n",
    "    falsestrings=[\"f\",\"fa\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## missingstrings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>m01</th><th>b002</th><th>c1</th><th>c2</th><th>c3</th><th>c4</th><th>d1</th></tr><tr><th></th><th>Int64?</th><th>String</th><th>String</th><th>Int64</th><th>String</th><th>Float64</th><th>Date</th></tr></thead><tbody><p>2 rows × 7 columns</p><tr><th>1</th><td><em>missing</em></td><td>fa</td><td>XY</td><td>2</td><td>c</td><td>1.5</td><td>2020-01-05</td></tr><tr><th>2</th><td>-1</td><td>tr</td><td>AB</td><td>16</td><td>x</td><td>2.33</td><td>2021-01-05</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccc}\n",
       "\t& m01 & b002 & c1 & c2 & c3 & c4 & d1\\\\\n",
       "\t\\hline\n",
       "\t& Int64? & String & String & Int64 & String & Float64 & Date\\\\\n",
       "\t\\hline\n",
       "\t1 & \\emph{missing} & fa & XY & 2 & c & 1.5 & 2020-01-05 \\\\\n",
       "\t2 & -1 & tr & AB & 16 & x & 2.33 & 2021-01-05 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×7 DataFrame\n",
       "│ Row │ m01     │ b002   │ c1     │ c2    │ c3     │ c4      │ d1         │\n",
       "│     │ \u001b[90mInt64?\u001b[39m  │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mDate\u001b[39m       │\n",
       "├─────┼─────────┼────────┼────────┼───────┼────────┼─────────┼────────────┤\n",
       "│ 1   │ \u001b[90mmissing\u001b[39m │ fa     │ XY     │ 2     │ c      │ 1.5     │ 2020-01-05 │\n",
       "│ 2   │ -1      │ tr     │ AB     │ 16    │ x      │ 2.33    │ 2021-01-05 │"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"\n",
    "m01|b002|c1|c2|c3|c4|d1\n",
    "999|\"fa\"|\"XY\"|2|c|1.5|2020-01-05\n",
    "-1|\"tr\"|\"AB\"|16|x|2.33|2021-01-05\n",
    "\"\"\"\n",
    "\n",
    "CSV.read(IOBuffer(data), DataFrame; \n",
    "    missingstrings=[\"999\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pool\n",
    "Pooling is similar to pandas catagory. The strings are stored to `PooledArrays.PooledArray` which can make some operations much faster. `Pool` argument set the treshold when the string column is turned to the pooled array. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"unique,cat\\nA18E9,AT\\nBF392,GC\\n93EBC,AT\\n54EE1,AT\\n8CD2E,GC\\n3A42E,GC\""
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"unique,cat\n",
    "A18E9,AT\n",
    "BF392,GC\n",
    "93EBC,AT\n",
    "54EE1,AT\n",
    "8CD2E,GC\n",
    "3A42E,GC\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>unique</th><th>cat</th></tr><tr><th></th><th>String</th><th>String</th></tr></thead><tbody><p>6 rows × 2 columns</p><tr><th>1</th><td>A18E9</td><td>AT</td></tr><tr><th>2</th><td>BF392</td><td>GC</td></tr><tr><th>3</th><td>93EBC</td><td>AT</td></tr><tr><th>4</th><td>54EE1</td><td>AT</td></tr><tr><th>5</th><td>8CD2E</td><td>GC</td></tr><tr><th>6</th><td>3A42E</td><td>GC</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& unique & cat\\\\\n",
       "\t\\hline\n",
       "\t& String & String\\\\\n",
       "\t\\hline\n",
       "\t1 & A18E9 & AT \\\\\n",
       "\t2 & BF392 & GC \\\\\n",
       "\t3 & 93EBC & AT \\\\\n",
       "\t4 & 54EE1 & AT \\\\\n",
       "\t5 & 8CD2E & GC \\\\\n",
       "\t6 & 3A42E & GC \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "6×2 DataFrame\n",
       "│ Row │ unique │ cat    │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼────────┼────────┤\n",
       "│ 1   │ A18E9  │ AT     │\n",
       "│ 2   │ BF392  │ GC     │\n",
       "│ 3   │ 93EBC  │ AT     │\n",
       "│ 4   │ 54EE1  │ AT     │\n",
       "│ 5   │ 8CD2E  │ GC     │\n",
       "│ 6   │ 3A42E  │ GC     │"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = CSV.read(IOBuffer(data), DataFrame; pool=0.29)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6-element Array{String,1}:\n",
       " \"A18E9\"\n",
       " \"BF392\"\n",
       " \"93EBC\"\n",
       " \"54EE1\"\n",
       " \"8CD2E\"\n",
       " \"3A42E\""
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# columns `unique` has different value on each row so it's not pooled\n",
    "df[:,:unique]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6-element PooledArrays.PooledArray{String,UInt32,1,Array{UInt32,1}}:\n",
       " \"AT\"\n",
       " \"GC\"\n",
       " \"AT\"\n",
       " \"AT\"\n",
       " \"GC\"\n",
       " \"GC\""
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# column can contain 2 values on 6 rows; 2/6 = 0.33 so the column is pooled. \n",
    "df[:,:cat]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comment parameter\n",
    "Sometimes your file contains comment lines. You can exclude them using `comment` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"c1|c2|c3|c4|d1\\n\\\"XY\\\"|2|c|1.5|2020-01-05\\n~ this is a comment\\n\\\"AB\\\"|16|x|2.33|2021-01-05\\n\""
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"c1|c2|c3|c4|d1\n",
    "\"XY\"|2|c|1.5|2020-01-05\n",
    "~ this is a comment\n",
    "\"AB\"|16|x|2.33|2021-01-05\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th><th>d1</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th><th>Date</th></tr></thead><tbody><p>2 rows × 5 columns</p><tr><th>1</th><td>XY</td><td>2</td><td>c</td><td>1.5</td><td>2020-01-05</td></tr><tr><th>2</th><td>AB</td><td>16</td><td>x</td><td>2.33</td><td>2021-01-05</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& c1 & c2 & c3 & c4 & d1\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64 & Date\\\\\n",
       "\t\\hline\n",
       "\t1 & XY & 2 & c & 1.5 & 2020-01-05 \\\\\n",
       "\t2 & AB & 16 & x & 2.33 & 2021-01-05 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×5 DataFrame\n",
       "│ Row │ c1     │ c2    │ c3     │ c4      │ d1         │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mDate\u001b[39m       │\n",
       "├─────┼────────┼───────┼────────┼─────────┼────────────┤\n",
       "│ 1   │ XY     │ 2     │ c      │ 1.5     │ 2020-01-05 │\n",
       "│ 2   │ AB     │ 16    │ x      │ 2.33    │ 2021-01-05 │"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rows starting with `~` will be skipped\n",
    "CSV.read(IOBuffer(data), DataFrame; comment=\"~\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th><th>c3</th><th>c4</th><th>d1</th></tr><tr><th></th><th>String</th><th>Int64</th><th>String</th><th>Float64</th><th>Date</th></tr></thead><tbody><p>2 rows × 5 columns</p><tr><th>1</th><td>XY</td><td>2</td><td>c</td><td>1.5</td><td>2020-01-05</td></tr><tr><th>2</th><td>AB</td><td>16</td><td>x</td><td>2.33</td><td>2021-01-05</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& c1 & c2 & c3 & c4 & d1\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & String & Float64 & Date\\\\\n",
       "\t\\hline\n",
       "\t1 & XY & 2 & c & 1.5 & 2020-01-05 \\\\\n",
       "\t2 & AB & 16 & x & 2.33 & 2021-01-05 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×5 DataFrame\n",
       "│ Row │ c1     │ c2    │ c3     │ c4      │ d1         │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mDate\u001b[39m       │\n",
       "├─────┼────────┼───────┼────────┼─────────┼────────────┤\n",
       "│ 1   │ XY     │ 2     │ c      │ 1.5     │ 2020-01-05 │\n",
       "│ 2   │ AB     │ 16    │ x      │ 2.33    │ 2021-01-05 │"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# comment line can start by more than one character\n",
    "data = \"\"\"c1|c2|c3|c4|d1\n",
    "\"XY\"|2|c|1.5|2020-01-05\n",
    "!! this is a comment\n",
    "\"AB\"|16|x|2.33|2021-01-05\n",
    "\"\"\"\n",
    "CSV.read(IOBuffer(data), DataFrame; comment=\"!!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "TypeError: in keyword argument comment, expected Union{Nothing, String}, got Array{String,1}",
     "output_type": "error",
     "traceback": [
      "TypeError: in keyword argument comment, expected Union{Nothing, String}, got Array{String,1}",
      "",
      "Stacktrace:",
      " [1] read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{T} where T; copycols::Bool, kwargs::Base.Iterators.Pairs{Symbol,Array{String,1},Tuple{Symbol},NamedTuple{(:comment,),Tuple{Array{String,1}}}}) at /home/vaclav/.julia/packages/CSV/la2cd/src/CSV.jl:45",
      " [2] top-level scope at In[59]:1"
     ]
    }
   ],
   "source": [
    "# you can only specify one string to mark the comments\n",
    "CSV.read(IOBuffer(data), DataFrame; comment=[\"~\",\"#\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transpose parameter\n",
    "When you can to transpose your data and turn columns into rows, you can set `transpose=true`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>c1</th><th>c2</th></tr><tr><th></th><th>String</th><th>Float64</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>X</td><td>2.0</td></tr><tr><th>2</th><td>Y</td><td>3.5</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& c1 & c2\\\\\n",
       "\t\\hline\n",
       "\t& String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & X & 2.0 \\\\\n",
       "\t2 & Y & 3.5 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×2 DataFrame\n",
       "│ Row │ c1     │ c2      │\n",
       "│     │ \u001b[90mString\u001b[39m │ \u001b[90mFloat64\u001b[39m │\n",
       "├─────┼────────┼─────────┤\n",
       "│ 1   │ X      │ 2.0     │\n",
       "│ 2   │ Y      │ 3.5     │"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"\n",
    "c1|X|Y\n",
    "c2|2.0|3.5\n",
    "\"\"\"\n",
    "CSV.read(IOBuffer(data), DataFrame; transpose=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fixed Width File with ignorerepeated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr><tr><th></th><th>Int64</th><th>Float64</th><th>String</th></tr></thead><tbody><p>1 rows × 3 columns</p><tr><th>1</th><td>1</td><td>2.0</td><td>X</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& A & B & C\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Float64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2.0 & X \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "1×3 DataFrame\n",
       "│ Row │ A     │ B       │ C      │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼───────┼─────────┼────────┤\n",
       "│ 1   │ 1     │ 2.0     │ X      │"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"\n",
    "A   B   C  \n",
    "1   2.0 \"X\"\n",
    "\"\"\"\n",
    "\n",
    "df = CSV.read(IOBuffer(data), DataFrame; \n",
    "    delim=\" \",\n",
    "    ignorerepeated=true)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{String,1}:\n",
       " \"A\"\n",
       " \"B\"\n",
       " \"C\""
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr><tr><th></th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>1 rows × 3 columns</p><tr><th>1</th><td>1</td><td>1</td><td>1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& A & B & C\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 1 & 1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "1×3 DataFrame\n",
       "│ Row │ A     │ B     │ C     │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │\n",
       "├─────┼───────┼───────┼───────┤\n",
       "│ 1   │ 1     │ 1     │ 1     │"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length.(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataframes with different delimiter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th> A </th><th>   B  </th><th> C  </th></tr><tr><th></th><th>Int64</th><th>Float64</th><th>String</th></tr></thead><tbody><p>1 rows × 3 columns</p><tr><th>1</th><td>1</td><td>2.0</td><td> X </td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t&  A  &    B   &  C  \\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Float64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2.0 &  X  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "1×3 DataFrame\n",
       "│ Row │  A    │    B    │  C     │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼───────┼─────────┼────────┤\n",
       "│ 1   │ 1     │ 2.0     │  X     │"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\"\n",
    "| A |   B  | C  \n",
    "| 1 |  2.0 | X \n",
    "\"\"\"\n",
    "\n",
    "df = CSV.read(IOBuffer(data), DataFrame; \n",
    "    delim=\"|\",\n",
    "    ignorerepeated=true)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the extra strings are blank spaces \" \" and not the pipe delimiters, nothing is removed using `ignorerepeated=true`. We need to remove the extra spaces manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{String,1}:\n",
       " \" A \"\n",
       " \"   B  \"\n",
       " \" C  \""
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# header names contain blank spaces\n",
    "names(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th> A </th><th>   B  </th><th> C  </th></tr><tr><th></th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>1 rows × 3 columns</p><tr><th>1</th><td>1</td><td>1</td><td>3</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t&  A  &    B   &  C  \\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 1 & 3 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "1×3 DataFrame\n",
       "│ Row │  A    │    B   │  C    │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m  │ \u001b[90mInt64\u001b[39m │\n",
       "├─────┼───────┼────────┼───────┤\n",
       "│ 1   │ 1     │ 1      │ 3     │"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# also the lenght of string values can be increated by the blank spaces\n",
    "length.(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{SubString{String},1}:\n",
       " \"A\"\n",
       " \"B\"\n",
       " \"C\""
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# applying strip (trim) element-wise (using . dot operator)\n",
    "strip.(names(df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr><tr><th></th><th>Int64</th><th>Float64</th><th>String</th></tr></thead><tbody><p>1 rows × 3 columns</p><tr><th>1</th><td>1</td><td>2.0</td><td> X </td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& A & B & C\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Float64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2.0 &  X  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "1×3 DataFrame\n",
       "│ Row │ A     │ B       │ C      │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mString\u001b[39m │\n",
       "├─────┼───────┼─────────┼────────┤\n",
       "│ 1   │ 1     │ 2.0     │  X     │"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rename the names, by applying `strip` element wise to the column names vector\n",
    "rename!(df, strip.(names(df)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{String,1}:\n",
       " \"A\"\n",
       " \"B\"\n",
       " \"C\""
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now the names are stripped\n",
    "names(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr><tr><th></th><th>Int64</th><th>Float64</th><th>SubStri…</th></tr></thead><tbody><p>1 rows × 3 columns</p><tr><th>1</th><td>1</td><td>2.0</td><td>X</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& A & B & C\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Float64 & SubStri…\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 2.0 & X \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "1×3 DataFrame\n",
       "│ Row │ A     │ B       │ C        │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mSubStri…\u001b[39m │\n",
       "├─────┼───────┼─────────┼──────────┤\n",
       "│ 1   │ 1     │ 2.0     │ X        │"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# apply strip to specific column\n",
    "transform!(df, :C => ByRow(x -> strip(x)) => :C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr><tr><th></th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>1 rows × 3 columns</p><tr><th>1</th><td>1</td><td>1</td><td>1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& A & B & C\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 1 & 1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "1×3 DataFrame\n",
       "│ Row │ A     │ B     │ C     │\n",
       "│     │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │\n",
       "├─────┼───────┼───────┼───────┤\n",
       "│ 1   │ 1     │ 1     │ 1     │"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length.(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "ename": "MethodError",
     "evalue": "MethodError: no method matching strip(::Int64)\nClosest candidates are:\n  strip(::Any, !Matched::AbstractString) at strings/util.jl:222\n  strip(::Any, !Matched::CategoricalValue{String,R} where R<:Integer) at deprecated.jl:65\n  strip(!Matched::AbstractString) at strings/util.jl:220\n  ...",
     "output_type": "error",
     "traceback": [
      "MethodError: no method matching strip(::Int64)\nClosest candidates are:\n  strip(::Any, !Matched::AbstractString) at strings/util.jl:222\n  strip(::Any, !Matched::CategoricalValue{String,R} where R<:Integer) at deprecated.jl:65\n  strip(!Matched::AbstractString) at strings/util.jl:220\n  ...",
      "",
      "Stacktrace:",
      " [1] _broadcast_getindex_evalf at ./broadcast.jl:631 [inlined]",
      " [2] _broadcast_getindex at ./broadcast.jl:604 [inlined]",
      " [3] getindex(::Base.Broadcast.Broadcasted{DataFrames.DataFrameStyle,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},typeof(strip),Tuple{Base.Broadcast.Extruded{Array{Int64,1},Tuple{Bool},Tuple{Int64}}}}, ::CartesianIndex{2}) at ./broadcast.jl:564",
      " [4] copy(::Base.Broadcast.Broadcasted{DataFrames.DataFrameStyle,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},typeof(strip),Tuple{DataFrame}}) at /home/vaclav/.julia/packages/DataFrames/GtZ1l/src/other/broadcasting.jl:71",
      " [5] materialize(::Base.Broadcast.Broadcasted{DataFrames.DataFrameStyle,Nothing,typeof(strip),Tuple{DataFrame}}) at ./broadcast.jl:820",
      " [6] top-level scope at In[72]:1"
     ]
    }
   ],
   "source": [
    "# applying strip to whole dataframe fails, because there's no strip method for ints or floats\n",
    "strip.(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select string columns\n",
    "# ... will be covered in some other tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Speed\n",
    "Test the speed for example on this ~600MB file available on kaggle [IEEE-CIS Fraud Detection competition](https://www.kaggle.com/c/ieee-fraud-detection/data?select=train_transaction.csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"/home/vaclav/Data/Kaggle/EEE-CIS_Fraud_Detection/train_transaction.csv\""
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = \"/home/vaclav/Data/Kaggle/EEE-CIS_Fraud_Detection/train_transaction.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 11.997074 seconds (3.77 M allocations: 2.056 GiB, 4.11% gc time)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>TransactionID</th><th>isFraud</th><th>TransactionDT</th><th>TransactionAmt</th><th>ProductCD</th><th>card1</th><th>card2</th></tr><tr><th></th><th>Int64</th><th>Int64</th><th>Int64</th><th>Float64</th><th>String</th><th>Int64</th><th>Float64?</th></tr></thead><tbody><p>590,540 rows × 394 columns (omitted printing of 387 columns)</p><tr><th>1</th><td>2987000</td><td>0</td><td>86400</td><td>68.5</td><td>W</td><td>13926</td><td><em>missing</em></td></tr><tr><th>2</th><td>2987001</td><td>0</td><td>86401</td><td>29.0</td><td>W</td><td>2755</td><td>404.0</td></tr><tr><th>3</th><td>2987002</td><td>0</td><td>86469</td><td>59.0</td><td>W</td><td>4663</td><td>490.0</td></tr><tr><th>4</th><td>2987003</td><td>0</td><td>86499</td><td>50.0</td><td>W</td><td>18132</td><td>567.0</td></tr><tr><th>5</th><td>2987004</td><td>0</td><td>86506</td><td>50.0</td><td>H</td><td>4497</td><td>514.0</td></tr><tr><th>6</th><td>2987005</td><td>0</td><td>86510</td><td>49.0</td><td>W</td><td>5937</td><td>555.0</td></tr><tr><th>7</th><td>2987006</td><td>0</td><td>86522</td><td>159.0</td><td>W</td><td>12308</td><td>360.0</td></tr><tr><th>8</th><td>2987007</td><td>0</td><td>86529</td><td>422.5</td><td>W</td><td>12695</td><td>490.0</td></tr><tr><th>9</th><td>2987008</td><td>0</td><td>86535</td><td>15.0</td><td>H</td><td>2803</td><td>100.0</td></tr><tr><th>10</th><td>2987009</td><td>0</td><td>86536</td><td>117.0</td><td>W</td><td>17399</td><td>111.0</td></tr><tr><th>11</th><td>2987010</td><td>0</td><td>86549</td><td>75.887</td><td>C</td><td>16496</td><td>352.0</td></tr><tr><th>12</th><td>2987011</td><td>0</td><td>86555</td><td>16.495</td><td>C</td><td>4461</td><td>375.0</td></tr><tr><th>13</th><td>2987012</td><td>0</td><td>86564</td><td>50.0</td><td>W</td><td>3786</td><td>418.0</td></tr><tr><th>14</th><td>2987013</td><td>0</td><td>86585</td><td>40.0</td><td>W</td><td>12866</td><td>303.0</td></tr><tr><th>15</th><td>2987014</td><td>0</td><td>86596</td><td>10.5</td><td>W</td><td>11839</td><td>490.0</td></tr><tr><th>16</th><td>2987015</td><td>0</td><td>86618</td><td>57.95</td><td>W</td><td>7055</td><td>555.0</td></tr><tr><th>17</th><td>2987016</td><td>0</td><td>86620</td><td>30.0</td><td>H</td><td>1790</td><td>555.0</td></tr><tr><th>18</th><td>2987017</td><td>0</td><td>86668</td><td>100.0</td><td>H</td><td>11492</td><td>111.0</td></tr><tr><th>19</th><td>2987018</td><td>0</td><td>86725</td><td>47.95</td><td>W</td><td>4663</td><td>490.0</td></tr><tr><th>20</th><td>2987019</td><td>0</td><td>86730</td><td>186.0</td><td>W</td><td>7005</td><td>111.0</td></tr><tr><th>21</th><td>2987020</td><td>0</td><td>86761</td><td>39.0</td><td>W</td><td>7875</td><td>314.0</td></tr><tr><th>22</th><td>2987021</td><td>0</td><td>86769</td><td>159.95</td><td>W</td><td>11401</td><td>543.0</td></tr><tr><th>23</th><td>2987022</td><td>0</td><td>86786</td><td>50.0</td><td>H</td><td>1724</td><td>583.0</td></tr><tr><th>24</th><td>2987023</td><td>0</td><td>86808</td><td>107.95</td><td>W</td><td>2392</td><td>360.0</td></tr><tr><th>25</th><td>2987024</td><td>0</td><td>86821</td><td>73.95</td><td>W</td><td>10112</td><td>360.0</td></tr><tr><th>26</th><td>2987025</td><td>0</td><td>86944</td><td>107.95</td><td>W</td><td>15385</td><td>111.0</td></tr><tr><th>27</th><td>2987026</td><td>0</td><td>86945</td><td>184.0</td><td>W</td><td>17868</td><td>148.0</td></tr><tr><th>28</th><td>2987027</td><td>0</td><td>86972</td><td>47.95</td><td>W</td><td>11307</td><td>321.0</td></tr><tr><th>29</th><td>2987028</td><td>0</td><td>86973</td><td>20.0</td><td>W</td><td>8431</td><td>269.0</td></tr><tr><th>30</th><td>2987029</td><td>0</td><td>86979</td><td>36.99</td><td>W</td><td>12932</td><td>361.0</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccc}\n",
       "\t& TransactionID & isFraud & TransactionDT & TransactionAmt & ProductCD & card1 & card2 & \\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Int64 & Int64 & Float64 & String & Int64 & Float64? & \\\\\n",
       "\t\\hline\n",
       "\t1 & 2987000 & 0 & 86400 & 68.5 & W & 13926 & \\emph{missing} & $\\dots$ \\\\\n",
       "\t2 & 2987001 & 0 & 86401 & 29.0 & W & 2755 & 404.0 & $\\dots$ \\\\\n",
       "\t3 & 2987002 & 0 & 86469 & 59.0 & W & 4663 & 490.0 & $\\dots$ \\\\\n",
       "\t4 & 2987003 & 0 & 86499 & 50.0 & W & 18132 & 567.0 & $\\dots$ \\\\\n",
       "\t5 & 2987004 & 0 & 86506 & 50.0 & H & 4497 & 514.0 & $\\dots$ \\\\\n",
       "\t6 & 2987005 & 0 & 86510 & 49.0 & W & 5937 & 555.0 & $\\dots$ \\\\\n",
       "\t7 & 2987006 & 0 & 86522 & 159.0 & W & 12308 & 360.0 & $\\dots$ \\\\\n",
       "\t8 & 2987007 & 0 & 86529 & 422.5 & W & 12695 & 490.0 & $\\dots$ \\\\\n",
       "\t9 & 2987008 & 0 & 86535 & 15.0 & H & 2803 & 100.0 & $\\dots$ \\\\\n",
       "\t10 & 2987009 & 0 & 86536 & 117.0 & W & 17399 & 111.0 & $\\dots$ \\\\\n",
       "\t11 & 2987010 & 0 & 86549 & 75.887 & C & 16496 & 352.0 & $\\dots$ \\\\\n",
       "\t12 & 2987011 & 0 & 86555 & 16.495 & C & 4461 & 375.0 & $\\dots$ \\\\\n",
       "\t13 & 2987012 & 0 & 86564 & 50.0 & W & 3786 & 418.0 & $\\dots$ \\\\\n",
       "\t14 & 2987013 & 0 & 86585 & 40.0 & W & 12866 & 303.0 & $\\dots$ \\\\\n",
       "\t15 & 2987014 & 0 & 86596 & 10.5 & W & 11839 & 490.0 & $\\dots$ \\\\\n",
       "\t16 & 2987015 & 0 & 86618 & 57.95 & W & 7055 & 555.0 & $\\dots$ \\\\\n",
       "\t17 & 2987016 & 0 & 86620 & 30.0 & H & 1790 & 555.0 & $\\dots$ \\\\\n",
       "\t18 & 2987017 & 0 & 86668 & 100.0 & H & 11492 & 111.0 & $\\dots$ \\\\\n",
       "\t19 & 2987018 & 0 & 86725 & 47.95 & W & 4663 & 490.0 & $\\dots$ \\\\\n",
       "\t20 & 2987019 & 0 & 86730 & 186.0 & W & 7005 & 111.0 & $\\dots$ \\\\\n",
       "\t21 & 2987020 & 0 & 86761 & 39.0 & W & 7875 & 314.0 & $\\dots$ \\\\\n",
       "\t22 & 2987021 & 0 & 86769 & 159.95 & W & 11401 & 543.0 & $\\dots$ \\\\\n",
       "\t23 & 2987022 & 0 & 86786 & 50.0 & H & 1724 & 583.0 & $\\dots$ \\\\\n",
       "\t24 & 2987023 & 0 & 86808 & 107.95 & W & 2392 & 360.0 & $\\dots$ \\\\\n",
       "\t25 & 2987024 & 0 & 86821 & 73.95 & W & 10112 & 360.0 & $\\dots$ \\\\\n",
       "\t26 & 2987025 & 0 & 86944 & 107.95 & W & 15385 & 111.0 & $\\dots$ \\\\\n",
       "\t27 & 2987026 & 0 & 86945 & 184.0 & W & 17868 & 148.0 & $\\dots$ \\\\\n",
       "\t28 & 2987027 & 0 & 86972 & 47.95 & W & 11307 & 321.0 & $\\dots$ \\\\\n",
       "\t29 & 2987028 & 0 & 86973 & 20.0 & W & 8431 & 269.0 & $\\dots$ \\\\\n",
       "\t30 & 2987029 & 0 & 86979 & 36.99 & W & 12932 & 361.0 & $\\dots$ \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ &  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "590540×394 DataFrame. Omitted printing of 390 columns\n",
       "│ Row    │ TransactionID │ isFraud │ TransactionDT │ TransactionAmt │\n",
       "│        │ \u001b[90mInt64\u001b[39m         │ \u001b[90mInt64\u001b[39m   │ \u001b[90mInt64\u001b[39m         │ \u001b[90mFloat64\u001b[39m        │\n",
       "├────────┼───────────────┼─────────┼───────────────┼────────────────┤\n",
       "│ 1      │ 2987000       │ 0       │ 86400         │ 68.5           │\n",
       "│ 2      │ 2987001       │ 0       │ 86401         │ 29.0           │\n",
       "│ 3      │ 2987002       │ 0       │ 86469         │ 59.0           │\n",
       "│ 4      │ 2987003       │ 0       │ 86499         │ 50.0           │\n",
       "│ 5      │ 2987004       │ 0       │ 86506         │ 50.0           │\n",
       "│ 6      │ 2987005       │ 0       │ 86510         │ 49.0           │\n",
       "│ 7      │ 2987006       │ 0       │ 86522         │ 159.0          │\n",
       "│ 8      │ 2987007       │ 0       │ 86529         │ 422.5          │\n",
       "│ 9      │ 2987008       │ 0       │ 86535         │ 15.0           │\n",
       "│ 10     │ 2987009       │ 0       │ 86536         │ 117.0          │\n",
       "⋮\n",
       "│ 590530 │ 3577529       │ 0       │ 15810912      │ 73.838         │\n",
       "│ 590531 │ 3577530       │ 0       │ 15810926      │ 400.78         │\n",
       "│ 590532 │ 3577531       │ 0       │ 15810935      │ 400.0          │\n",
       "│ 590533 │ 3577532       │ 0       │ 15811007      │ 204.97         │\n",
       "│ 590534 │ 3577533       │ 0       │ 15811029      │ 107.95         │\n",
       "│ 590535 │ 3577534       │ 0       │ 15811030      │ 67.505         │\n",
       "│ 590536 │ 3577535       │ 0       │ 15811047      │ 49.0           │\n",
       "│ 590537 │ 3577536       │ 0       │ 15811049      │ 39.5           │\n",
       "│ 590538 │ 3577537       │ 0       │ 15811079      │ 30.95          │\n",
       "│ 590539 │ 3577538       │ 0       │ 15811088      │ 117.0          │\n",
       "│ 590540 │ 3577539       │ 0       │ 15811131      │ 279.95         │"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# @time wrapper of julia\n",
    "@time begin\n",
    "    CSV.read(path, DataFrame)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of threads\n",
    "Threads.nthreads() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start julia with multiple threads, start julia with `julia --threads n` (`n` is the number of threads) or set the enviromental variable `JULIA_NUM_THREADS` to the number of treads. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "start julia with 4 threads\n",
    "\n",
    "```$ julia --threads 4```\n",
    "\n",
    "open the jupyter notebook from julia\n",
    "\n",
    "```using IJulia\n",
    "notebook()```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.4.1",
   "language": "julia",
   "name": "julia-1.4"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}