2016-12-06_data.tableWhereHaveYouBeen.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# `data.table`: Where Have You Been All My Life? \n",
    "\n",
    "# °₊·ˈ∗♡( ˃̶᷇ ‧̫ ˂̶᷆ )♡∗ˈ‧₊°"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`data.table` is an easy to use R package for querying and manipulating data. Dataset updates run much faster than with the base R paradigm, and specifying complicated calculations is more intuitive. This talk will show some of the package's features as well as point out a couple of gotchas.\n",
    "\n",
    "* Add/update/delete columns by reference by group using no copies at all\n",
    "\n",
    "* Really simplified data manipulation for me, especially on longitudinal data.\n",
    "\n",
    "* Focus on the add/update/delete columns and indexing\n",
    "\n",
    "* Many more capabilities not covered here!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "setwd(\"/Users/ruddjm/Dropbox/miscWork/presentations/dataDotTable\")\n",
    "library(data.table)\n",
    "library(rbenchmark)\n",
    "library(Hmisc)\n",
    "options(repr.matrix.max.rows = 40, repr.matrix.max.cols = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The data.table class\n",
    "\n",
    "* data.table is a package, but it's also a **class**.\n",
    "* 'Enhanced data.frame'\n",
    "* You can turn a data frame into a data table, and it will still do all the things data frames do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's make a data.table\n",
    "\n",
    "* Need to get your data in a data.table to use data.table's capabilities\n",
    "* How to turn a data frame into a data.table\n",
    "* Can also read in using a data.table function that automatically makes it a data.table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Can use `data.table` function with columns input directly "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>a</th><th scope=col>b</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>A</td><td>4</td></tr>\n",
       "\t<tr><td>A</td><td>5</td></tr>\n",
       "\t<tr><td>B</td><td>6</td></tr>\n",
       "\t<tr><td>C</td><td>7</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " a & b\\\\\n",
       "\\hline\n",
       "\t A & 4\\\\\n",
       "\t A & 5\\\\\n",
       "\t B & 6\\\\\n",
       "\t C & 7\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  a b\n",
       "1 A 4\n",
       "2 A 5\n",
       "3 B 6\n",
       "4 C 7"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "DT = data.table(a=LETTERS[c(1,1:3)],b=4:7,key=\"a\")\n",
    "DT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'data.table'</li>\n",
       "\t<li>'data.frame'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'data.table'\n",
       "\\item 'data.frame'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'data.table'\n",
       "2. 'data.frame'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"data.table\" \"data.frame\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class(DT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's \"unmake\" it a data.table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class(DT) = 'data.frame'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And back to a data.table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "DT = data.table(DT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example data: plants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#http://plants.usda.gov/adv_search.html\n",
    "plantsDF = read.csv(file = 'exampleData/usdaQuery.txt',\n",
    "  stringsAsFactors = FALSE,\n",
    "  na.strings = c(\"\", \"NA\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Accepted.Symbol</th><th scope=col>Synonym.Symbol</th><th scope=col>Scientific.Name</th><th scope=col>State.and.Province</th><th scope=col>Duration</th><th scope=col>⋯</th><th scope=col>Foliage.Color</th><th scope=col>Height..Mature..feet.</th><th scope=col>Drought.Tolerance</th><th scope=col>Shade.Tolerance</th><th scope=col>Bloom.Period</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>ABAM                                                                                                                                 </td><td>NA                                                                                                                                   </td><td>Abies amabilis                                                                                                                       </td><td>USA (AK, CA, OR, WA), CAN (BC)                                                                                                       </td><td>Perennial                                                                                                                            </td><td>⋯                                                                                                                                    </td><td>Gray-Green                                                                                                                           </td><td>165                                                                                                                                  </td><td>Low                                                                                                                                  </td><td>Tolerant                                                                                                                             </td><td>Late Spring                                                                                                                          </td></tr>\n",
       "\t<tr><td>ABBA                                                                                                                                 </td><td>NA                                                                                                                                   </td><td>Abies balsamea                                                                                                                       </td><td>USA (CT, IA, IN, MA, MD, ME, MI, MN, NH, NJ, NY, OH, PA, RI, VA, VT, WI, WV), CAN (AB, LB, MB, NB, NF, NS, ON, PE, QC, SK), FRA (SPM)</td><td>Perennial                                                                                                                            </td><td>⋯                                                                                                                                    </td><td>Green                                                                                                                                </td><td> 60                                                                                                                                  </td><td>Low                                                                                                                                  </td><td>Tolerant                                                                                                                             </td><td>Mid Summer                                                                                                                           </td></tr>\n",
       "\t<tr><td>ABCO                                                                                                                                 </td><td>NA                                                                                                                                   </td><td>Abies concolor                                                                                                                       </td><td>USA (AZ, CO, ID, MA, ME, NM, NV, OR, UT, WY)                                                                                         </td><td>Perennial                                                                                                                            </td><td>⋯                                                                                                                                    </td><td>Green                                                                                                                                </td><td>120                                                                                                                                  </td><td>Medium                                                                                                                               </td><td>Intermediate                                                                                                                         </td><td>Late Spring                                                                                                                          </td></tr>\n",
       "\t<tr><td>ABFR                                                                                                                                 </td><td>NA                                                                                                                                   </td><td>Abies fraseri                                                                                                                        </td><td>USA (GA, NC, TN, VA)                                                                                                                 </td><td>Perennial                                                                                                                            </td><td>⋯                                                                                                                                    </td><td>Dark Green                                                                                                                           </td><td> 55                                                                                                                                  </td><td>Low                                                                                                                                  </td><td>Tolerant                                                                                                                             </td><td>Mid Spring                                                                                                                           </td></tr>\n",
       "\t<tr><td>ABGR                                                                                                                                 </td><td>NA                                                                                                                                   </td><td>Abies grandis                                                                                                                        </td><td>USA (CA, ID, MT, OR, WA), CAN (BC)                                                                                                   </td><td>Perennial                                                                                                                            </td><td>⋯                                                                                                                                    </td><td>Green                                                                                                                                </td><td>150                                                                                                                                  </td><td>Medium                                                                                                                               </td><td>Tolerant                                                                                                                             </td><td>Late Spring                                                                                                                          </td></tr>\n",
       "\t<tr><td>ABLA                                                                                                                                 </td><td>NA                                                                                                                                   </td><td>Abies lasiocarpa                                                                                                                     </td><td>USA (AK, AZ, CA, CO, ID, MT, NM, NV, OR, UT, WA, WY), CAN (AB, BC, NT, YT)                                                           </td><td>Perennial                                                                                                                            </td><td>⋯                                                                                                                                    </td><td>Green                                                                                                                                </td><td> 90                                                                                                                                  </td><td>Low                                                                                                                                  </td><td>Tolerant                                                                                                                             </td><td>Late Spring                                                                                                                          </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllllllllllll}\n",
       " Accepted.Symbol & Synonym.Symbol & Scientific.Name & State.and.Province & Duration & ⋯ & Foliage.Color & Height..Mature..feet. & Drought.Tolerance & Shade.Tolerance & Bloom.Period\\\\\n",
       "\\hline\n",
       "\t ABAM                                                                                                                                  & NA                                                                                                                                    & Abies amabilis                                                                                                                        & USA (AK, CA, OR, WA), CAN (BC)                                                                                                        & Perennial                                                                                                                             & ⋯                                                                                                                                     & Gray-Green                                                                                                                            & 165                                                                                                                                   & Low                                                                                                                                   & Tolerant                                                                                                                              & Late Spring                                                                                                                          \\\\\n",
       "\t ABBA                                                                                                                                  & NA                                                                                                                                    & Abies balsamea                                                                                                                        & USA (CT, IA, IN, MA, MD, ME, MI, MN, NH, NJ, NY, OH, PA, RI, VA, VT, WI, WV), CAN (AB, LB, MB, NB, NF, NS, ON, PE, QC, SK), FRA (SPM) & Perennial                                                                                                                             & ⋯                                                                                                                                     & Green                                                                                                                                 &  60                                                                                                                                   & Low                                                                                                                                   & Tolerant                                                                                                                              & Mid Summer                                                                                                                           \\\\\n",
       "\t ABCO                                                                                                                                  & NA                                                                                                                                    & Abies concolor                                                                                                                        & USA (AZ, CO, ID, MA, ME, NM, NV, OR, UT, WY)                                                                                          & Perennial                                                                                                                             & ⋯                                                                                                                                     & Green                                                                                                                                 & 120                                                                                                                                   & Medium                                                                                                                                & Intermediate                                                                                                                          & Late Spring                                                                                                                          \\\\\n",
       "\t ABFR                                                                                                                                  & NA                                                                                                                                    & Abies fraseri                                                                                                                         & USA (GA, NC, TN, VA)                                                                                                                  & Perennial                                                                                                                             & ⋯                                                                                                                                     & Dark Green                                                                                                                            &  55                                                                                                                                   & Low                                                                                                                                   & Tolerant                                                                                                                              & Mid Spring                                                                                                                           \\\\\n",
       "\t ABGR                                                                                                                                  & NA                                                                                                                                    & Abies grandis                                                                                                                         & USA (CA, ID, MT, OR, WA), CAN (BC)                                                                                                    & Perennial                                                                                                                             & ⋯                                                                                                                                     & Green                                                                                                                                 & 150                                                                                                                                   & Medium                                                                                                                                & Tolerant                                                                                                                              & Late Spring                                                                                                                          \\\\\n",
       "\t ABLA                                                                                                                                  & NA                                                                                                                                    & Abies lasiocarpa                                                                                                                      & USA (AK, AZ, CA, CO, ID, MT, NM, NV, OR, UT, WA, WY), CAN (AB, BC, NT, YT)                                                            & Perennial                                                                                                                             & ⋯                                                                                                                                     & Green                                                                                                                                 &  90                                                                                                                                   & Low                                                                                                                                   & Tolerant                                                                                                                              & Late Spring                                                                                                                          \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Accepted.Symbol Synonym.Symbol Scientific.Name \n",
       "1 ABAM            NA             Abies amabilis  \n",
       "2 ABBA            NA             Abies balsamea  \n",
       "3 ABCO            NA             Abies concolor  \n",
       "4 ABFR            NA             Abies fraseri   \n",
       "5 ABGR            NA             Abies grandis   \n",
       "6 ABLA            NA             Abies lasiocarpa\n",
       "  State.and.Province                                                                                                                   \n",
       "1 USA (AK, CA, OR, WA), CAN (BC)                                                                                                       \n",
       "2 USA (CT, IA, IN, MA, MD, ME, MI, MN, NH, NJ, NY, OH, PA, RI, VA, VT, WI, WV), CAN (AB, LB, MB, NB, NF, NS, ON, PE, QC, SK), FRA (SPM)\n",
       "3 USA (AZ, CO, ID, MA, ME, NM, NV, OR, UT, WY)                                                                                         \n",
       "4 USA (GA, NC, TN, VA)                                                                                                                 \n",
       "5 USA (CA, ID, MT, OR, WA), CAN (BC)                                                                                                   \n",
       "6 USA (AK, AZ, CA, CO, ID, MT, NM, NV, OR, UT, WA, WY), CAN (AB, BC, NT, YT)                                                           \n",
       "  Duration  ⋯ Foliage.Color Height..Mature..feet. Drought.Tolerance\n",
       "1 Perennial ⋯ Gray-Green    165                   Low              \n",
       "2 Perennial ⋯ Green          60                   Low              \n",
       "3 Perennial ⋯ Green         120                   Medium           \n",
       "4 Perennial ⋯ Dark Green     55                   Low              \n",
       "5 Perennial ⋯ Green         150                   Medium           \n",
       "6 Perennial ⋯ Green          90                   Low              \n",
       "  Shade.Tolerance Bloom.Period\n",
       "1 Tolerant        Late Spring \n",
       "2 Tolerant        Mid Summer  \n",
       "3 Intermediate    Late Spring \n",
       "4 Tolerant        Mid Spring  \n",
       "5 Tolerant        Late Spring \n",
       "6 Tolerant        Late Spring "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plantsDF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "'data.frame'"
      ],
      "text/latex": [
       "'data.frame'"
      ],
      "text/markdown": [
       "'data.frame'"
      ],
      "text/plain": [
       "[1] \"data.frame\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class(plantsDF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plants = data.table(plantsDF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'data.table'</li>\n",
       "\t<li>'data.frame'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'data.table'\n",
       "\\item 'data.frame'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'data.table'\n",
       "2. 'data.frame'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"data.table\" \"data.frame\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class(plants)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'AcceptedSymbol'</li>\n",
       "\t<li>'SynonymSymbol'</li>\n",
       "\t<li>'ScientificName'</li>\n",
       "\t<li>'StateandProvince'</li>\n",
       "\t<li>'Duration'</li>\n",
       "\t<li>'GrowthHabit'</li>\n",
       "\t<li>'NativeStatus'</li>\n",
       "\t<li>'FlowerColor'</li>\n",
       "\t<li>'FlowerConspicuous'</li>\n",
       "\t<li>'FoliageColor'</li>\n",
       "\t<li>'HeightMaturefeet'</li>\n",
       "\t<li>'DroughtTolerance'</li>\n",
       "\t<li>'ShadeTolerance'</li>\n",
       "\t<li>'BloomPeriod'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'AcceptedSymbol'\n",
       "\\item 'SynonymSymbol'\n",
       "\\item 'ScientificName'\n",
       "\\item 'StateandProvince'\n",
       "\\item 'Duration'\n",
       "\\item 'GrowthHabit'\n",
       "\\item 'NativeStatus'\n",
       "\\item 'FlowerColor'\n",
       "\\item 'FlowerConspicuous'\n",
       "\\item 'FoliageColor'\n",
       "\\item 'HeightMaturefeet'\n",
       "\\item 'DroughtTolerance'\n",
       "\\item 'ShadeTolerance'\n",
       "\\item 'BloomPeriod'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'AcceptedSymbol'\n",
       "2. 'SynonymSymbol'\n",
       "3. 'ScientificName'\n",
       "4. 'StateandProvince'\n",
       "5. 'Duration'\n",
       "6. 'GrowthHabit'\n",
       "7. 'NativeStatus'\n",
       "8. 'FlowerColor'\n",
       "9. 'FlowerConspicuous'\n",
       "10. 'FoliageColor'\n",
       "11. 'HeightMaturefeet'\n",
       "12. 'DroughtTolerance'\n",
       "13. 'ShadeTolerance'\n",
       "14. 'BloomPeriod'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] \"AcceptedSymbol\"    \"SynonymSymbol\"     \"ScientificName\"   \n",
       " [4] \"StateandProvince\"  \"Duration\"          \"GrowthHabit\"      \n",
       " [7] \"NativeStatus\"      \"FlowerColor\"       \"FlowerConspicuous\"\n",
       "[10] \"FoliageColor\"      \"HeightMaturefeet\"  \"DroughtTolerance\" \n",
       "[13] \"ShadeTolerance\"    \"BloomPeriod\"      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "names(plants) = gsub('\\\\.', '', names(plants))\n",
    "names(plants)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rows\n",
    "\n",
    "Can subset rows using\n",
    "* row indices\n",
    "* logical statements\n",
    "* logical statements don't need to refer to the df name!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>AcceptedSymbol</th><th scope=col>SynonymSymbol</th><th scope=col>ScientificName</th><th scope=col>StateandProvince</th><th scope=col>Duration</th><th scope=col>⋯</th><th scope=col>FoliageColor</th><th scope=col>HeightMaturefeet</th><th scope=col>DroughtTolerance</th><th scope=col>ShadeTolerance</th><th scope=col>BloomPeriod</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>ABAM                          </td><td>NA                            </td><td>Abies amabilis                </td><td>USA (AK, CA, OR, WA), CAN (BC)</td><td>Perennial                     </td><td>⋯                             </td><td>Gray-Green                    </td><td>165                           </td><td>Low                           </td><td>Tolerant                      </td><td>Late Spring                   </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllllllllllll}\n",
       " AcceptedSymbol & SynonymSymbol & ScientificName & StateandProvince & Duration & ⋯ & FoliageColor & HeightMaturefeet & DroughtTolerance & ShadeTolerance & BloomPeriod\\\\\n",
       "\\hline\n",
       "\t ABAM                           & NA                             & Abies amabilis                 & USA (AK, CA, OR, WA), CAN (BC) & Perennial                      & ⋯                              & Gray-Green                     & 165                            & Low                            & Tolerant                       & Late Spring                   \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  AcceptedSymbol SynonymSymbol ScientificName StateandProvince              \n",
       "1 ABAM           NA            Abies amabilis USA (AK, CA, OR, WA), CAN (BC)\n",
       "  Duration  ⋯ FoliageColor HeightMaturefeet DroughtTolerance ShadeTolerance\n",
       "1 Perennial ⋯ Gray-Green   165              Low              Tolerant      \n",
       "  BloomPeriod\n",
       "1 Late Spring"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>AcceptedSymbol</th><th scope=col>SynonymSymbol</th><th scope=col>ScientificName</th><th scope=col>StateandProvince</th><th scope=col>Duration</th><th scope=col>⋯</th><th scope=col>FoliageColor</th><th scope=col>HeightMaturefeet</th><th scope=col>DroughtTolerance</th><th scope=col>ShadeTolerance</th><th scope=col>BloomPeriod</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>ABAM                          </td><td>NA                            </td><td>Abies amabilis                </td><td>USA (AK, CA, OR, WA), CAN (BC)</td><td>Perennial                     </td><td>⋯                             </td><td>Gray-Green                    </td><td>165                           </td><td>Low                           </td><td>Tolerant                      </td><td>Late Spring                   </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllllllllllll}\n",
       " AcceptedSymbol & SynonymSymbol & ScientificName & StateandProvince & Duration & ⋯ & FoliageColor & HeightMaturefeet & DroughtTolerance & ShadeTolerance & BloomPeriod\\\\\n",
       "\\hline\n",
       "\t ABAM                           & NA                             & Abies amabilis                 & USA (AK, CA, OR, WA), CAN (BC) & Perennial                      & ⋯                              & Gray-Green                     & 165                            & Low                            & Tolerant                       & Late Spring                   \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  AcceptedSymbol SynonymSymbol ScientificName StateandProvince              \n",
       "1 ABAM           NA            Abies amabilis USA (AK, CA, OR, WA), CAN (BC)\n",
       "  Duration  ⋯ FoliageColor HeightMaturefeet DroughtTolerance ShadeTolerance\n",
       "1 Perennial ⋯ Gray-Green   165              Low              Tolerant      \n",
       "  BloomPeriod\n",
       "1 Late Spring"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# First row:\n",
    "plants[1, ]\n",
    "# or\n",
    "plants[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get all the plants with white-gray foliage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>AcceptedSymbol</th><th scope=col>SynonymSymbol</th><th scope=col>ScientificName</th><th scope=col>StateandProvince</th><th scope=col>Duration</th><th scope=col>⋯</th><th scope=col>FoliageColor</th><th scope=col>HeightMaturefeet</th><th scope=col>DroughtTolerance</th><th scope=col>ShadeTolerance</th><th scope=col>BloomPeriod</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>ARLU                                                                                                                                                                                                                                  </td><td>NA                                                                                                                                                                                                                                    </td><td>Artemisia ludoviciana                                                                                                                                                                                                                 </td><td>USA (AR, AZ, CA, CO, CT, DE, FL, GA, IA, ID, IL, IN, KS, KY, LA, MA, MD, ME, MI, MN, MO, MS, MT, NC, ND, NE, NH, NJ, NM, NV, NY, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VA, VT, WA, WI, WY), CAN (AB, BC, MB, NB, NT, ON, PE, QC, SK)</td><td>Perennial                                                                                                                                                                                                                             </td><td>⋯                                                                                                                                                                                                                                     </td><td>White-Gray                                                                                                                                                                                                                            </td><td> 3.0                                                                                                                                                                                                                                  </td><td>High                                                                                                                                                                                                                                  </td><td>Intolerant                                                                                                                                                                                                                            </td><td>Early Spring                                                                                                                                                                                                                          </td></tr>\n",
       "\t<tr><td>ARRI2                                                                                                                                                                                                                                 </td><td>NA                                                                                                                                                                                                                                    </td><td>Artemisia rigida                                                                                                                                                                                                                      </td><td>USA (ID, MT, OR, WA)                                                                                                                                                                                                                  </td><td>Perennial                                                                                                                                                                                                                             </td><td>⋯                                                                                                                                                                                                                                     </td><td>White-Gray                                                                                                                                                                                                                            </td><td> 2.0                                                                                                                                                                                                                                  </td><td>High                                                                                                                                                                                                                                  </td><td>Intolerant                                                                                                                                                                                                                            </td><td>Late Summer                                                                                                                                                                                                                           </td></tr>\n",
       "\t<tr><td>CORA                                                                                                                                                                                                                                  </td><td>NA                                                                                                                                                                                                                                    </td><td>Coleogyne ramosissima                                                                                                                                                                                                                 </td><td>USA (AZ, CA, CO, NV, UT)                                                                                                                                                                                                              </td><td>Perennial                                                                                                                                                                                                                             </td><td>⋯                                                                                                                                                                                                                                     </td><td>White-Gray                                                                                                                                                                                                                            </td><td> 6.0                                                                                                                                                                                                                                  </td><td>High                                                                                                                                                                                                                                  </td><td>Intolerant                                                                                                                                                                                                                            </td><td>Mid Spring                                                                                                                                                                                                                            </td></tr>\n",
       "\t<tr><td>COFL3                                                                                                                                                                                                                                 </td><td>NA                                                                                                                                                                                                                                    </td><td>Corydalis flavula                                                                                                                                                                                                                     </td><td>USA (AL, AR, CT, DC, DE, FL, GA, IA, IL, IN, KS, KY, LA, MD, MI, MO, MS, NC, NE, NJ, NY, OH, OK, PA, SC, TN, VA, WV), CAN (ON)                                                                                                        </td><td>Annual                                                                                                                                                                                                                                </td><td>⋯                                                                                                                                                                                                                                     </td><td>White-Gray                                                                                                                                                                                                                            </td><td> 1.5                                                                                                                                                                                                                                  </td><td>Low                                                                                                                                                                                                                                   </td><td>Intermediate                                                                                                                                                                                                                          </td><td>Mid Spring                                                                                                                                                                                                                            </td></tr>\n",
       "\t<tr><td>CRMO4                                                                                                                                                                                                                                 </td><td>NA                                                                                                                                                                                                                                    </td><td>Crepis modocensis                                                                                                                                                                                                                     </td><td>USA (CA, CO, ID, MT, NV, OR, UT, WA, WY), CAN (BC)                                                                                                                                                                                    </td><td>Perennial                                                                                                                                                                                                                             </td><td>⋯                                                                                                                                                                                                                                     </td><td>White-Gray                                                                                                                                                                                                                            </td><td> 1.4                                                                                                                                                                                                                                  </td><td>Medium                                                                                                                                                                                                                                </td><td>Intolerant                                                                                                                                                                                                                            </td><td>Late Spring                                                                                                                                                                                                                           </td></tr>\n",
       "\t<tr><td>ELCO                                                                                                                                                                                                                                  </td><td>NA                                                                                                                                                                                                                                    </td><td>Elaeagnus commutata                                                                                                                                                                                                                   </td><td>USA (AK, CO, ID, KY, MD, MN, MT, ND, NY, RI, SD, TX, UT, WA, WY), CAN (AB, BC, MB, NT, NU, ON, QC, SK, YT)                                                                                                                            </td><td>Perennial                                                                                                                                                                                                                             </td><td>⋯                                                                                                                                                                                                                                     </td><td>White-Gray                                                                                                                                                                                                                            </td><td>12.0                                                                                                                                                                                                                                  </td><td>High                                                                                                                                                                                                                                  </td><td>Intolerant                                                                                                                                                                                                                            </td><td>Early Summer                                                                                                                                                                                                                          </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllllllllllll}\n",
       " AcceptedSymbol & SynonymSymbol & ScientificName & StateandProvince & Duration & ⋯ & FoliageColor & HeightMaturefeet & DroughtTolerance & ShadeTolerance & BloomPeriod\\\\\n",
       "\\hline\n",
       "\t ARLU                                                                                                                                                                                                                                   & NA                                                                                                                                                                                                                                     & Artemisia ludoviciana                                                                                                                                                                                                                  & USA (AR, AZ, CA, CO, CT, DE, FL, GA, IA, ID, IL, IN, KS, KY, LA, MA, MD, ME, MI, MN, MO, MS, MT, NC, ND, NE, NH, NJ, NM, NV, NY, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VA, VT, WA, WI, WY), CAN (AB, BC, MB, NB, NT, ON, PE, QC, SK) & Perennial                                                                                                                                                                                                                              & ⋯                                                                                                                                                                                                                                      & White-Gray                                                                                                                                                                                                                             &  3.0                                                                                                                                                                                                                                   & High                                                                                                                                                                                                                                   & Intolerant                                                                                                                                                                                                                             & Early Spring                                                                                                                                                                                                                          \\\\\n",
       "\t ARRI2                                                                                                                                                                                                                                  & NA                                                                                                                                                                                                                                     & Artemisia rigida                                                                                                                                                                                                                       & USA (ID, MT, OR, WA)                                                                                                                                                                                                                   & Perennial                                                                                                                                                                                                                              & ⋯                                                                                                                                                                                                                                      & White-Gray                                                                                                                                                                                                                             &  2.0                                                                                                                                                                                                                                   & High                                                                                                                                                                                                                                   & Intolerant                                                                                                                                                                                                                             & Late Summer                                                                                                                                                                                                                           \\\\\n",
       "\t CORA                                                                                                                                                                                                                                   & NA                                                                                                                                                                                                                                     & Coleogyne ramosissima                                                                                                                                                                                                                  & USA (AZ, CA, CO, NV, UT)                                                                                                                                                                                                               & Perennial                                                                                                                                                                                                                              & ⋯                                                                                                                                                                                                                                      & White-Gray                                                                                                                                                                                                                             &  6.0                                                                                                                                                                                                                                   & High                                                                                                                                                                                                                                   & Intolerant                                                                                                                                                                                                                             & Mid Spring                                                                                                                                                                                                                            \\\\\n",
       "\t COFL3                                                                                                                                                                                                                                  & NA                                                                                                                                                                                                                                     & Corydalis flavula                                                                                                                                                                                                                      & USA (AL, AR, CT, DC, DE, FL, GA, IA, IL, IN, KS, KY, LA, MD, MI, MO, MS, NC, NE, NJ, NY, OH, OK, PA, SC, TN, VA, WV), CAN (ON)                                                                                                         & Annual                                                                                                                                                                                                                                 & ⋯                                                                                                                                                                                                                                      & White-Gray                                                                                                                                                                                                                             &  1.5                                                                                                                                                                                                                                   & Low                                                                                                                                                                                                                                    & Intermediate                                                                                                                                                                                                                           & Mid Spring                                                                                                                                                                                                                            \\\\\n",
       "\t CRMO4                                                                                                                                                                                                                                  & NA                                                                                                                                                                                                                                     & Crepis modocensis                                                                                                                                                                                                                      & USA (CA, CO, ID, MT, NV, OR, UT, WA, WY), CAN (BC)                                                                                                                                                                                     & Perennial                                                                                                                                                                                                                              & ⋯                                                                                                                                                                                                                                      & White-Gray                                                                                                                                                                                                                             &  1.4                                                                                                                                                                                                                                   & Medium                                                                                                                                                                                                                                 & Intolerant                                                                                                                                                                                                                             & Late Spring                                                                                                                                                                                                                           \\\\\n",
       "\t ELCO                                                                                                                                                                                                                                   & NA                                                                                                                                                                                                                                     & Elaeagnus commutata                                                                                                                                                                                                                    & USA (AK, CO, ID, KY, MD, MN, MT, ND, NY, RI, SD, TX, UT, WA, WY), CAN (AB, BC, MB, NT, NU, ON, QC, SK, YT)                                                                                                                             & Perennial                                                                                                                                                                                                                              & ⋯                                                                                                                                                                                                                                      & White-Gray                                                                                                                                                                                                                             & 12.0                                                                                                                                                                                                                                   & High                                                                                                                                                                                                                                   & Intolerant                                                                                                                                                                                                                             & Early Summer                                                                                                                                                                                                                          \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  AcceptedSymbol SynonymSymbol ScientificName       \n",
       "1 ARLU           NA            Artemisia ludoviciana\n",
       "2 ARRI2          NA            Artemisia rigida     \n",
       "3 CORA           NA            Coleogyne ramosissima\n",
       "4 COFL3          NA            Corydalis flavula    \n",
       "5 CRMO4          NA            Crepis modocensis    \n",
       "6 ELCO           NA            Elaeagnus commutata  \n",
       "  StateandProvince                                                                                                                                                                                                                      \n",
       "1 USA (AR, AZ, CA, CO, CT, DE, FL, GA, IA, ID, IL, IN, KS, KY, LA, MA, MD, ME, MI, MN, MO, MS, MT, NC, ND, NE, NH, NJ, NM, NV, NY, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VA, VT, WA, WI, WY), CAN (AB, BC, MB, NB, NT, ON, PE, QC, SK)\n",
       "2 USA (ID, MT, OR, WA)                                                                                                                                                                                                                  \n",
       "3 USA (AZ, CA, CO, NV, UT)                                                                                                                                                                                                              \n",
       "4 USA (AL, AR, CT, DC, DE, FL, GA, IA, IL, IN, KS, KY, LA, MD, MI, MO, MS, NC, NE, NJ, NY, OH, OK, PA, SC, TN, VA, WV), CAN (ON)                                                                                                        \n",
       "5 USA (CA, CO, ID, MT, NV, OR, UT, WA, WY), CAN (BC)                                                                                                                                                                                    \n",
       "6 USA (AK, CO, ID, KY, MD, MN, MT, ND, NY, RI, SD, TX, UT, WA, WY), CAN (AB, BC, MB, NT, NU, ON, QC, SK, YT)                                                                                                                            \n",
       "  Duration  ⋯ FoliageColor HeightMaturefeet DroughtTolerance ShadeTolerance\n",
       "1 Perennial ⋯ White-Gray    3.0             High             Intolerant    \n",
       "2 Perennial ⋯ White-Gray    2.0             High             Intolerant    \n",
       "3 Perennial ⋯ White-Gray    6.0             High             Intolerant    \n",
       "4 Annual    ⋯ White-Gray    1.5             Low              Intermediate  \n",
       "5 Perennial ⋯ White-Gray    1.4             Medium           Intolerant    \n",
       "6 Perennial ⋯ White-Gray   12.0             High             Intolerant    \n",
       "  BloomPeriod \n",
       "1 Early Spring\n",
       "2 Late Summer \n",
       "3 Mid Spring  \n",
       "4 Mid Spring  \n",
       "5 Late Spring \n",
       "6 Early Summer"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plants[FoliageColor == 'White-Gray', ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Columns\n",
    "\n",
    "* no quotes needed on column names\n",
    "* wrap names in `.()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>BloomPeriod</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Mid Summer </td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Mid Spring </td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " BloomPeriod\\\\\n",
       "\\hline\n",
       "\t Late Spring\\\\\n",
       "\t Mid Summer \\\\\n",
       "\t Late Spring\\\\\n",
       "\t Mid Spring \\\\\n",
       "\t Late Spring\\\\\n",
       "\t Late Spring\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  BloomPeriod\n",
       "1 Late Spring\n",
       "2 Mid Summer \n",
       "3 Late Spring\n",
       "4 Mid Spring \n",
       "5 Late Spring\n",
       "6 Late Spring"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plants[ , .(BloomPeriod)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Can also omit the `.()`\n",
    "* Result will be a vector, not a data.table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'Late Spring'</li>\n",
       "\t<li>'Mid Summer'</li>\n",
       "\t<li>'Late Spring'</li>\n",
       "\t<li>'Mid Spring'</li>\n",
       "\t<li>'Late Spring'</li>\n",
       "\t<li>'Late Spring'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'Late Spring'\n",
       "\\item 'Mid Summer'\n",
       "\\item 'Late Spring'\n",
       "\\item 'Mid Spring'\n",
       "\\item 'Late Spring'\n",
       "\\item 'Late Spring'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'Late Spring'\n",
       "2. 'Mid Summer'\n",
       "3. 'Late Spring'\n",
       "4. 'Mid Spring'\n",
       "5. 'Late Spring'\n",
       "6. 'Late Spring'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"Late Spring\" \"Mid Summer\"  \"Late Spring\" \"Mid Spring\"  \"Late Spring\"\n",
       "[6] \"Late Spring\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plants[ , BloomPeriod])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select more than one column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>FlowerColor</th><th scope=col>Duration</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Yellow   </td><td>Perennial</td></tr>\n",
       "\t<tr><td>Yellow   </td><td>Perennial</td></tr>\n",
       "\t<tr><td>Red      </td><td>Perennial</td></tr>\n",
       "\t<tr><td>Purple   </td><td>Perennial</td></tr>\n",
       "\t<tr><td>NA       </td><td>Perennial</td></tr>\n",
       "\t<tr><td>NA       </td><td>Perennial</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " FlowerColor & Duration\\\\\n",
       "\\hline\n",
       "\t Yellow    & Perennial\\\\\n",
       "\t Yellow    & Perennial\\\\\n",
       "\t Red       & Perennial\\\\\n",
       "\t Purple    & Perennial\\\\\n",
       "\t NA        & Perennial\\\\\n",
       "\t NA        & Perennial\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  FlowerColor Duration \n",
       "1 Yellow      Perennial\n",
       "2 Yellow      Perennial\n",
       "3 Red         Perennial\n",
       "4 Purple      Perennial\n",
       "5 NA          Perennial\n",
       "6 NA          Perennial"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plants[ , .(FlowerColor, Duration)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about indexing by the column number?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "14"
      ],
      "text/latex": [
       "14"
      ],
      "text/markdown": [
       "14"
      ],
      "text/plain": [
       "[1] 14"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "which(names(plants) == 'BloomPeriod')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'Late Spring'</li>\n",
       "\t<li>'Mid Summer'</li>\n",
       "\t<li>'Late Spring'</li>\n",
       "\t<li>'Mid Spring'</li>\n",
       "\t<li>'Late Spring'</li>\n",
       "\t<li>'Late Spring'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'Late Spring'\n",
       "\\item 'Mid Summer'\n",
       "\\item 'Late Spring'\n",
       "\\item 'Mid Spring'\n",
       "\\item 'Late Spring'\n",
       "\\item 'Late Spring'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'Late Spring'\n",
       "2. 'Mid Summer'\n",
       "3. 'Late Spring'\n",
       "4. 'Mid Spring'\n",
       "5. 'Late Spring'\n",
       "6. 'Late Spring'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"Late Spring\" \"Mid Summer\"  \"Late Spring\" \"Mid Spring\"  \"Late Spring\"\n",
       "[6] \"Late Spring\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plantsDF[ , 14])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "14"
      ],
      "text/latex": [
       "14"
      ],
      "text/markdown": [
       "14"
      ],
      "text/plain": [
       "[1] 14"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plants[ , 14]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ఠ ͟ಠ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Must use `with` argument**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>BloomPeriod</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Mid Summer </td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Mid Spring </td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " BloomPeriod\\\\\n",
       "\\hline\n",
       "\t Late Spring\\\\\n",
       "\t Mid Summer \\\\\n",
       "\t Late Spring\\\\\n",
       "\t Mid Spring \\\\\n",
       "\t Late Spring\\\\\n",
       "\t Late Spring\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  BloomPeriod\n",
       "1 Late Spring\n",
       "2 Mid Summer \n",
       "3 Late Spring\n",
       "4 Mid Spring \n",
       "5 Late Spring\n",
       "6 Late Spring"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plants[ , 14, with = FALSE])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about indexing by the **string** column name?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "'BloomPeriod'"
      ],
      "text/latex": [
       "'BloomPeriod'"
      ],
      "text/markdown": [
       "'BloomPeriod'"
      ],
      "text/plain": [
       "[1] \"BloomPeriod\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plants[ , 'BloomPeriod'] # doesn't work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>V1</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>BloomPeriod</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " V1\\\\\n",
       "\\hline\n",
       "\t BloomPeriod\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  V1         \n",
       "1 BloomPeriod"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plants[ , .('BloomPeriod')] # doesn't work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, need to use `with`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>BloomPeriod</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Mid Summer </td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Mid Spring </td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "\t<tr><td>Late Spring</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " BloomPeriod\\\\\n",
       "\\hline\n",
       "\t Late Spring\\\\\n",
       "\t Mid Summer \\\\\n",
       "\t Late Spring\\\\\n",
       "\t Mid Spring \\\\\n",
       "\t Late Spring\\\\\n",
       "\t Late Spring\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  BloomPeriod\n",
       "1 Late Spring\n",
       "2 Mid Summer \n",
       "3 Late Spring\n",
       "4 Mid Spring \n",
       "5 Late Spring\n",
       "6 Late Spring"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plants[ , 'BloomPeriod', with = FALSE])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This may break your existing code!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing operations my return a data.table or a vector, depending on the way you index \n",
    "Investigate the object returned by different indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "DT = data.table(a=LETTERS[c(1,1:3)],b=4:7,key=\"a\")\n",
    "DF = data.frame(a=LETTERS[c(1,1:3)], b=4:7, stringsAsFactors = FALSE)\n",
    "DT[, a]\n",
    "DT[, \"a\", with = FALSE]\n",
    "\n",
    "\n",
    "# Using the column name with no quotes gives a vector.\n",
    "str(DT[, a])\n",
    "# chr [1:4] \"A\" \"A\" \"B\" \"C\"\n",
    "\n",
    "# Using the CHARACTER NAME returns a data frame (thats also a data.table).\n",
    "str(DT[, \"a\", with = FALSE])\n",
    "#Classes ‘data.table’ and 'data.frame':  4 obs. of  1 variable:\n",
    "# $ a: chr  \"A\" \"A\" \"B\" \"C\"\n",
    "# - attr(*, \"sorted\")= chr \"a\"\n",
    "# - attr(*, \".internal.selfref\")=<externalptr>\n",
    "\n",
    "str(DT[, a])\n",
    "str(DT[, 1, with = FALSE])\n",
    "str(DT[, \"a\", with = FALSE])\n",
    "str(DT[[\"a\"]])\n",
    "str(DT[[1]])\n",
    "DT[, 1]                  # NO!\n",
    "DT[, .(1)]               # NO!\n",
    "DT[, 1, with = FALSE]       # returns a data table\n",
    "DT[, .(a)]\n",
    "DT[ , list(a)]\n",
    "\n",
    "\n",
    "# data FRAME returns a vector for all these uses.\n",
    "str(DF[[\"a\"]])\n",
    "str(DF[[1]])\n",
    "str(DF[, 1])\n",
    "str(DF[, \"a\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to access help files about indexing\n",
    "\n",
    "* Documentation for indexing\n",
    "* Tricky because we use the `[` operator and not a function name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#help('[.data.frame')\n",
    "#help('[.data.table')\n",
    "?'[.data.table'\n",
    "# Note the quote"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Can call all sorts of functions within the brackets! \n",
    "* Inside or outside!\n",
    "* Kind of like using the `with` function\n",
    "\n",
    "# ✧٩(•́⌄•́๑)و ✧"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>165</li>\n",
       "\t<li>60</li>\n",
       "\t<li>120</li>\n",
       "\t<li>55</li>\n",
       "\t<li>150</li>\n",
       "\t<li>90</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 165\n",
       "\\item 60\n",
       "\\item 120\n",
       "\\item 55\n",
       "\\item 150\n",
       "\\item 90\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 165\n",
       "2. 60\n",
       "3. 120\n",
       "4. 55\n",
       "5. 150\n",
       "6. 90\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 165  60 120  55 150  90"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plants[ , head(HeightMaturefeet)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "               FlowerColor\n",
       "BloomPeriod     Blue Brown Green Orange Purple Red White Yellow\n",
       "  Early Spring     4     1     8      4     18   5    36     51\n",
       "  Early Summer     4     1    10      0     12   8    38     50\n",
       "  Fall             1     0     5      0      2   3     3      7\n",
       "  Indeterminate    2     0     8      0      5   4    13     18\n",
       "  Late Spring     11    14   105      4     53  25   110    140\n",
       "  Late Summer      5     4    27      0     20   4    22     45\n",
       "  Late Winter      0     2     0      0      0   0     3      6\n",
       "  Mid Spring      11     4    34      4     24  16    68    112\n",
       "  Mid Summer       4     3    26      1     21   2    24     44\n",
       "  Spring           7     2    37      1     11  13    42     43\n",
       "  Summer          10     9    32      1     17  10    30     34\n",
       "  Winter           0     0     0      0      0   1     3      2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plants[ , table(BloomPeriod, FlowerColor)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculations **BY** another variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the mean height by growth habit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>GrowthHabit</th><th scope=col>V1</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Tree                      </td><td>87.977143                 </td></tr>\n",
       "\t<tr><td>NA                        </td><td>19.958000                 </td></tr>\n",
       "\t<tr><td>Forb/herb                 </td><td> 2.741739                 </td></tr>\n",
       "\t<tr><td>Tree, Shrub, Vine         </td><td>18.000000                 </td></tr>\n",
       "\t<tr><td>Tree, Shrub               </td><td>28.176355                 </td></tr>\n",
       "\t<tr><td>Graminoid                 </td><td> 2.929167                 </td></tr>\n",
       "\t<tr><td>Shrub                     </td><td> 7.545324                 </td></tr>\n",
       "\t<tr><td>Subshrub, Forb/herb       </td><td> 2.962821                 </td></tr>\n",
       "\t<tr><td>Subshrub, Shrub, Forb/herb</td><td> 1.900000                 </td></tr>\n",
       "\t<tr><td>Forb/herb, Shrub, Subshrub</td><td> 1.200000                 </td></tr>\n",
       "\t<tr><td>Subshrub, Shrub           </td><td> 3.362500                 </td></tr>\n",
       "\t<tr><td>Forb/herb, Subshrub       </td><td> 3.077778                 </td></tr>\n",
       "\t<tr><td>Vine                      </td><td>26.666667                 </td></tr>\n",
       "\t<tr><td>Vine, Forb/herb           </td><td> 2.471429                 </td></tr>\n",
       "\t<tr><td>Shrub, Subshrub           </td><td> 4.688889                 </td></tr>\n",
       "\t<tr><td>Subshrub                  </td><td> 4.220000                 </td></tr>\n",
       "\t<tr><td>Subshrub, Shrub, Graminoid</td><td>19.000000                 </td></tr>\n",
       "\t<tr><td>Shrub, Tree               </td><td>21.000000                 </td></tr>\n",
       "\t<tr><td>Tree, Subshrub, Shrub     </td><td> 9.000000                 </td></tr>\n",
       "\t<tr><td>Shrub, Vine               </td><td> 1.500000                 </td></tr>\n",
       "\t<tr><td>Vine, Subshrub            </td><td> 9.666667                 </td></tr>\n",
       "\t<tr><td>Forb/herb, Vine           </td><td>      NaN                 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " GrowthHabit & V1\\\\\n",
       "\\hline\n",
       "\t Tree                       & 87.977143                 \\\\\n",
       "\t NA                         & 19.958000                 \\\\\n",
       "\t Forb/herb                  &  2.741739                 \\\\\n",
       "\t Tree, Shrub, Vine          & 18.000000                 \\\\\n",
       "\t Tree, Shrub                & 28.176355                 \\\\\n",
       "\t Graminoid                  &  2.929167                 \\\\\n",
       "\t Shrub                      &  7.545324                 \\\\\n",
       "\t Subshrub, Forb/herb        &  2.962821                 \\\\\n",
       "\t Subshrub, Shrub, Forb/herb &  1.900000                 \\\\\n",
       "\t Forb/herb, Shrub, Subshrub &  1.200000                 \\\\\n",
       "\t Subshrub, Shrub            &  3.362500                 \\\\\n",
       "\t Forb/herb, Subshrub        &  3.077778                 \\\\\n",
       "\t Vine                       & 26.666667                 \\\\\n",
       "\t Vine, Forb/herb            &  2.471429                 \\\\\n",
       "\t Shrub, Subshrub            &  4.688889                 \\\\\n",
       "\t Subshrub                   &  4.220000                 \\\\\n",
       "\t Subshrub, Shrub, Graminoid & 19.000000                 \\\\\n",
       "\t Shrub, Tree                & 21.000000                 \\\\\n",
       "\t Tree, Subshrub, Shrub      &  9.000000                 \\\\\n",
       "\t Shrub, Vine                &  1.500000                 \\\\\n",
       "\t Vine, Subshrub             &  9.666667                 \\\\\n",
       "\t Forb/herb, Vine            &       NaN                 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   GrowthHabit                V1       \n",
       "1  Tree                       87.977143\n",
       "2  NA                         19.958000\n",
       "3  Forb/herb                   2.741739\n",
       "4  Tree, Shrub, Vine          18.000000\n",
       "5  Tree, Shrub                28.176355\n",
       "6  Graminoid                   2.929167\n",
       "7  Shrub                       7.545324\n",
       "8  Subshrub, Forb/herb         2.962821\n",
       "9  Subshrub, Shrub, Forb/herb  1.900000\n",
       "10 Forb/herb, Shrub, Subshrub  1.200000\n",
       "11 Subshrub, Shrub             3.362500\n",
       "12 Forb/herb, Subshrub         3.077778\n",
       "13 Vine                       26.666667\n",
       "14 Vine, Forb/herb             2.471429\n",
       "15 Shrub, Subshrub             4.688889\n",
       "16 Subshrub                    4.220000\n",
       "17 Subshrub, Shrub, Graminoid 19.000000\n",
       "18 Shrub, Tree                21.000000\n",
       "19 Tree, Subshrub, Shrub       9.000000\n",
       "20 Shrub, Vine                 1.500000\n",
       "21 Vine, Subshrub              9.666667\n",
       "22 Forb/herb, Vine                  NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plants[ , mean(HeightMaturefeet, na.rm = TRUE), by = GrowthHabit]  #[ , 2, with = FALSE]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment/Creating new variables\n",
    "\n",
    "* Much faster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# DF$c <- 18:13 # results in a deep copy of the entire data frame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `data.table` allows you to add/update/delete columns by reference using no copies at all\n",
    "\n",
    "* Use the `:=` operator\n",
    "\n",
    "Let's make a new variable that is the plant height plus 1.\n",
    "\n",
    "*Normally, these in-place assignments do not produce any output. They are producing output here. It may be an issue with jupyter notebook?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "head(plants[ , oneFootTaller := HeightMaturefeet + 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>ScientificName</th><th scope=col>HeightMaturefeet</th><th scope=col>oneFootTaller</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Abies amabilis  </td><td>165             </td><td>166             </td></tr>\n",
       "\t<tr><td>Abies balsamea  </td><td> 60             </td><td> 61             </td></tr>\n",
       "\t<tr><td>Abies concolor  </td><td>120             </td><td>121             </td></tr>\n",
       "\t<tr><td>Abies fraseri   </td><td> 55             </td><td> 56             </td></tr>\n",
       "\t<tr><td>Abies grandis   </td><td>150             </td><td>151             </td></tr>\n",
       "\t<tr><td>Abies lasiocarpa</td><td> 90             </td><td> 91             </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " ScientificName & HeightMaturefeet & oneFootTaller\\\\\n",
       "\\hline\n",
       "\t Abies amabilis   & 165              & 166             \\\\\n",
       "\t Abies balsamea   &  60              &  61             \\\\\n",
       "\t Abies concolor   & 120              & 121             \\\\\n",
       "\t Abies fraseri    &  55              &  56             \\\\\n",
       "\t Abies grandis    & 150              & 151             \\\\\n",
       "\t Abies lasiocarpa &  90              &  91             \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  ScientificName   HeightMaturefeet oneFootTaller\n",
       "1 Abies amabilis   165              166          \n",
       "2 Abies balsamea    60               61          \n",
       "3 Abies concolor   120              121          \n",
       "4 Abies fraseri     55               56          \n",
       "5 Abies grandis    150              151          \n",
       "6 Abies lasiocarpa  90               91          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plants[ , .(ScientificName, HeightMaturefeet, oneFootTaller)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a new variable and initialize it to missing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plants[ , numberOfPetals := NA]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Can update a specific value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "'Schoenoplectus maritimus'"
      ],
      "text/latex": [
       "'Schoenoplectus maritimus'"
      ],
      "text/markdown": [
       "'Schoenoplectus maritimus'"
      ],
      "text/plain": [
       "[1] \"Schoenoplectus maritimus\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's update the petal count for one particular plant.\n",
    "# Let's first find the scientific name for the 246th plant:\n",
    "plants[246, ScientificName]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plants[ScientificName == \"Schoenoplectus maritimus\", numberOfPetals := 17]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>ScientificName</th><th scope=col>numberOfPetals</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Blechnum spicant          </td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Blepharoneuron tricholepis</td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Boehmeria cylindrica      </td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Bolboschoenus fluviatilis </td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Schoenoplectus fluviatilis</td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Bolboschoenus maritimus   </td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Schoenoplectus maritimus  </td><td>TRUE                      </td></tr>\n",
       "\t<tr><td>Bolboschoenus robustus    </td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Schoenoplectus robustus   </td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Boltonia asteroides       </td><td>  NA                      </td></tr>\n",
       "\t<tr><td>Borrichia frutescens      </td><td>  NA                      </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " ScientificName & numberOfPetals\\\\\n",
       "\\hline\n",
       "\t Blechnum spicant           &   NA                      \\\\\n",
       "\t Blepharoneuron tricholepis &   NA                      \\\\\n",
       "\t Boehmeria cylindrica       &   NA                      \\\\\n",
       "\t Bolboschoenus fluviatilis  &   NA                      \\\\\n",
       "\t Schoenoplectus fluviatilis &   NA                      \\\\\n",
       "\t Bolboschoenus maritimus    &   NA                      \\\\\n",
       "\t Schoenoplectus maritimus   & TRUE                      \\\\\n",
       "\t Bolboschoenus robustus     &   NA                      \\\\\n",
       "\t Schoenoplectus robustus    &   NA                      \\\\\n",
       "\t Boltonia asteroides        &   NA                      \\\\\n",
       "\t Borrichia frutescens       &   NA                      \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   ScientificName             numberOfPetals\n",
       "1  Blechnum spicant             NA          \n",
       "2  Blepharoneuron tricholepis   NA          \n",
       "3  Boehmeria cylindrica         NA          \n",
       "4  Bolboschoenus fluviatilis    NA          \n",
       "5  Schoenoplectus fluviatilis   NA          \n",
       "6  Bolboschoenus maritimus      NA          \n",
       "7  Schoenoplectus maritimus   TRUE          \n",
       "8  Bolboschoenus robustus       NA          \n",
       "9  Schoenoplectus robustus      NA          \n",
       "10 Boltonia asteroides          NA          \n",
       "11 Borrichia frutescens         NA          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plants[240:250, .(ScientificName, numberOfPetals)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the value of numberOfPetals for \"Schoenoplectus fluviatilis!\" It's not 17 like we assigned!\n",
    "\n",
    "The reason is that when whe created a new variable and set it to missing, it automatically made the new variable a *logical* type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "'logical'"
      ],
      "text/latex": [
       "'logical'"
      ],
      "text/markdown": [
       "'logical'"
      ],
      "text/plain": [
       "[1] \"logical\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " logi [1:1725] NA NA NA NA NA NA ...\n"
     ]
    }
   ],
   "source": [
    "typeof(plants$numberOfPetals)\n",
    "str(plants$numberOfPetals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want the new variable to contain integers, we can define it initially by explicitly setting the type with `as.vector(NA, mode = 'integer')`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Start by removing the previous version of numberOfPetals from the DT:\n",
    "plants[ , numberOfPetals := NULL]\n",
    "plants[ , numberOfPetals := as.vector(NA, mode = 'integer')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "'integer'"
      ],
      "text/latex": [
       "'integer'"
      ],
      "text/markdown": [
       "'integer'"
      ],
      "text/plain": [
       "[1] \"integer\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " int [1:1725] NA NA NA NA NA NA NA NA NA NA ...\n"
     ]
    }
   ],
   "source": [
    "typeof(plants$numberOfPetals)\n",
    "str(plants$numberOfPetals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## To see the documentationfor assignment, search for \":=\"\n",
    "?\":=\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plants[ , c('AcceptedSymbol', 'SynonymSymbol', 'NativeStatus') := NULL]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This changes the plants data frame *in place*."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Similar for data frames, but if you have several columns to remove \n",
    "# it's a little clumsier.\n",
    "# You could also do this with subset, but you'd need to state the columns\n",
    "# you need to RETAIN rather than the ones you want to drop.\n",
    "plants = plants[ , -which(names(plants) %in% c(\"AcceptedSymbol\", \"SynonymSymbol\"))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assignment BY another variable\n",
    "\n",
    "Let's add a new column with this mean height, according to the plant's growth habit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plants[ , heightClassMean := mean(HeightMaturefeet, na.rm = TRUE), by = GrowthHabit]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>ScientificName</th><th scope=col>HeightMaturefeet</th><th scope=col>GrowthHabit</th><th scope=col>heightClassMean</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Acalypha virginica                    </td><td>3.0                                   </td><td>Forb/herb                             </td><td>2.741739                              </td></tr>\n",
       "\t<tr><td>Achillea millefolium                  </td><td>3.0                                   </td><td>Forb/herb                             </td><td>2.741739                              </td></tr>\n",
       "\t<tr><td>Achillea millefolium var. occidentalis</td><td>3.0                                   </td><td>Forb/herb                             </td><td>2.741739                              </td></tr>\n",
       "\t<tr><td>Acmella oppositifolia var. repens     </td><td>0.7                                   </td><td>Forb/herb                             </td><td>2.741739                              </td></tr>\n",
       "\t<tr><td>Aconitum columbianum                  </td><td>5.0                                   </td><td>Forb/herb                             </td><td>2.741739                              </td></tr>\n",
       "\t<tr><td>Acorus americanus                     </td><td>3.9                                   </td><td>Forb/herb                             </td><td>2.741739                              </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " ScientificName & HeightMaturefeet & GrowthHabit & heightClassMean\\\\\n",
       "\\hline\n",
       "\t Acalypha virginica                     & 3.0                                    & Forb/herb                              & 2.741739                              \\\\\n",
       "\t Achillea millefolium                   & 3.0                                    & Forb/herb                              & 2.741739                              \\\\\n",
       "\t Achillea millefolium var. occidentalis & 3.0                                    & Forb/herb                              & 2.741739                              \\\\\n",
       "\t Acmella oppositifolia var. repens      & 0.7                                    & Forb/herb                              & 2.741739                              \\\\\n",
       "\t Aconitum columbianum                   & 5.0                                    & Forb/herb                              & 2.741739                              \\\\\n",
       "\t Acorus americanus                      & 3.9                                    & Forb/herb                              & 2.741739                              \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  ScientificName                         HeightMaturefeet GrowthHabit\n",
       "1 Acalypha virginica                     3.0              Forb/herb  \n",
       "2 Achillea millefolium                   3.0              Forb/herb  \n",
       "3 Achillea millefolium var. occidentalis 3.0              Forb/herb  \n",
       "4 Acmella oppositifolia var. repens      0.7              Forb/herb  \n",
       "5 Aconitum columbianum                   5.0              Forb/herb  \n",
       "6 Acorus americanus                      3.9              Forb/herb  \n",
       "  heightClassMean\n",
       "1 2.741739       \n",
       "2 2.741739       \n",
       "3 2.741739       \n",
       "4 2.741739       \n",
       "5 2.741739       \n",
       "6 2.741739       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(plants[ , .(ScientificName, HeightMaturefeet, GrowthHabit, heightClassMean)][order(c(GrowthHabit))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Another example data set\n",
    "\n",
    "* Longitudinal\n",
    "* Demonstrate some practical applications\n",
    "* Can download this data at https://www.r-project.org/nosvn/conferences/useR-2007/competition/ragged.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading objects:\n",
      "  ragged\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>id</th><th scope=col>futime</th><th scope=col>status</th><th scope=col>sex</th><th scope=col>agebl</th><th scope=col>⋯</th><th scope=col>alkphos</th><th scope=col>ALT</th><th scope=col>protime</th><th scope=col>platelets</th><th scope=col>stage</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>1     </td><td> 400  </td><td>dead  </td><td>female</td><td>21464 </td><td>⋯     </td><td>1718  </td><td>138.0 </td><td>190   </td><td>12.2  </td><td>4     </td></tr>\n",
       "\t<tr><td>1     </td><td> 400  </td><td>dead  </td><td>female</td><td>21464 </td><td>⋯     </td><td>1612  </td><td>  6.2 </td><td>183   </td><td>11.2  </td><td>4     </td></tr>\n",
       "\t<tr><td>2     </td><td>5169  </td><td>pretx </td><td>female</td><td>20617 </td><td>⋯     </td><td>7395  </td><td>113.5 </td><td>221   </td><td>10.6  </td><td>3     </td></tr>\n",
       "\t<tr><td>2     </td><td>5169  </td><td>pretx </td><td>female</td><td>20617 </td><td>⋯     </td><td>2107  </td><td>139.5 </td><td>188   </td><td>11.0  </td><td>3     </td></tr>\n",
       "\t<tr><td>2     </td><td>5169  </td><td>pretx </td><td>female</td><td>20617 </td><td>⋯     </td><td>1711  </td><td>144.2 </td><td>161   </td><td>11.6  </td><td>3     </td></tr>\n",
       "\t<tr><td>2     </td><td>5169  </td><td>pretx </td><td>female</td><td>20617 </td><td>⋯     </td><td>1365  </td><td>144.2 </td><td>122   </td><td>10.6  </td><td>3     </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllllllllllllllll}\n",
       " id & futime & status & sex & agebl & ⋯ & alkphos & ALT & protime & platelets & stage\\\\\n",
       "\\hline\n",
       "\t 1      &  400   & dead   & female & 21464  & ⋯      & 1718   & 138.0  & 190    & 12.2   & 4     \\\\\n",
       "\t 1      &  400   & dead   & female & 21464  & ⋯      & 1612   &   6.2  & 183    & 11.2   & 4     \\\\\n",
       "\t 2      & 5169   & pretx  & female & 20617  & ⋯      & 7395   & 113.5  & 221    & 10.6   & 3     \\\\\n",
       "\t 2      & 5169   & pretx  & female & 20617  & ⋯      & 2107   & 139.5  & 188    & 11.0   & 3     \\\\\n",
       "\t 2      & 5169   & pretx  & female & 20617  & ⋯      & 1711   & 144.2  & 161    & 11.6   & 3     \\\\\n",
       "\t 2      & 5169   & pretx  & female & 20617  & ⋯      & 1365   & 144.2  & 122    & 10.6   & 3     \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  id futime status sex    agebl ⋯ alkphos ALT   protime platelets stage\n",
       "1 1   400   dead   female 21464 ⋯ 1718    138.0 190     12.2      4    \n",
       "2 1   400   dead   female 21464 ⋯ 1612      6.2 183     11.2      4    \n",
       "3 2  5169   pretx  female 20617 ⋯ 7395    113.5 221     10.6      3    \n",
       "4 2  5169   pretx  female 20617 ⋯ 2107    139.5 188     11.0      3    \n",
       "5 2  5169   pretx  female 20617 ⋯ 1711    144.2 161     11.6      3    \n",
       "6 2  5169   pretx  female 20617 ⋯ 1365    144.2 122     10.6      3    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "load(file = 'exampleData/ragged.Rdata', verbose = TRUE)\n",
    "head(ragged[order(ragged$id), ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ragged = data.table(ragged)\n",
    "ragged[ , id := as.character(id)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'id'</li>\n",
       "\t<li>'visittime'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'id'\n",
       "\\item 'visittime'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'id'\n",
       "2. 'visittime'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"id\"        \"visittime\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "setkey(ragged, id, visittime)\n",
    "key(ragged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can query by id without stating 'id,' since id is set as a key. `visittime` is set as a secondary key. \n",
    "\n",
    "Get the row for id 99 and visittime 193."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>id</th><th scope=col>futime</th><th scope=col>status</th><th scope=col>sex</th><th scope=col>agebl</th><th scope=col>⋯</th><th scope=col>alkphos</th><th scope=col>ALT</th><th scope=col>protime</th><th scope=col>platelets</th><th scope=col>stage</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>99   </td><td>4489 </td><td>pretx</td><td>male </td><td>17703</td><td>⋯    </td><td>231  </td><td>21.7 </td><td>218  </td><td>10.1 </td><td>2    </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllllllllllllllll}\n",
       " id & futime & status & sex & agebl & ⋯ & alkphos & ALT & protime & platelets & stage\\\\\n",
       "\\hline\n",
       "\t 99    & 4489  & pretx & male  & 17703 & ⋯     & 231   & 21.7  & 218   & 10.1  & 2    \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  id futime status sex  agebl ⋯ alkphos ALT  protime platelets stage\n",
       "1 99 4489   pretx  male 17703 ⋯ 231     21.7 218     10.1      2    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ragged[.('99', 193)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate each person's follow up time by taking their highest visittime. Accomplish this by using max(visittime) with by = id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>id</th><th scope=col>V1</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>1   </td><td> 192</td></tr>\n",
       "\t<tr><td>10  </td><td>   0</td></tr>\n",
       "\t<tr><td>100 </td><td> 514</td></tr>\n",
       "\t<tr><td>101 </td><td>4049</td></tr>\n",
       "\t<tr><td>102 </td><td>3354</td></tr>\n",
       "\t<tr><td>103 </td><td>   0</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " id & V1\\\\\n",
       "\\hline\n",
       "\t 1    &  192\\\\\n",
       "\t 10   &    0\\\\\n",
       "\t 100  &  514\\\\\n",
       "\t 101  & 4049\\\\\n",
       "\t 102  & 3354\\\\\n",
       "\t 103  &    0\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  id  V1  \n",
       "1 1    192\n",
       "2 10     0\n",
       "3 100  514\n",
       "4 101 4049\n",
       "5 102 3354\n",
       "6 103    0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(ragged[ ,  max(visittime), by = id])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we need those values in our datasat. I would usually use merge(). \n",
    "\n",
    "The following code both calculates everyone's last visit time and merges the results back into the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# This is the coolest thing ever!!!\n",
    "ragged[ ,  lastVisitTime := max(visittime), by = id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>id</th><th scope=col>visittime</th><th scope=col>lastVisitTime</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>15  </td><td> 196</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td> 366</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td> 735</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td>1128</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td>1491</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td>1946</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td>2143</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td>2499</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td>2891</td><td>3537</td></tr>\n",
       "\t<tr><td>15  </td><td>3537</td><td>3537</td></tr>\n",
       "\t<tr><td>150 </td><td>   0</td><td>2577</td></tr>\n",
       "\t<tr><td>150 </td><td> 188</td><td>2577</td></tr>\n",
       "\t<tr><td>150 </td><td> 365</td><td>2577</td></tr>\n",
       "\t<tr><td>150 </td><td> 741</td><td>2577</td></tr>\n",
       "\t<tr><td>150 </td><td>1112</td><td>2577</td></tr>\n",
       "\t<tr><td>150 </td><td>1476</td><td>2577</td></tr>\n",
       "\t<tr><td>150 </td><td>1841</td><td>2577</td></tr>\n",
       "\t<tr><td>150 </td><td>2214</td><td>2577</td></tr>\n",
       "\t<tr><td>150 </td><td>2577</td><td>2577</td></tr>\n",
       "\t<tr><td>151 </td><td>   0</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td> 167</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td> 356</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td> 714</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td>1098</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td>1470</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td>1868</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td>2179</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td>2550</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td>2919</td><td>3275</td></tr>\n",
       "\t<tr><td>151 </td><td>3275</td><td>3275</td></tr>\n",
       "\t<tr><td>152 </td><td>   0</td><td> 832</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " id & visittime & lastVisitTime\\\\\n",
       "\\hline\n",
       "\t 15   &  196 & 3537\\\\\n",
       "\t 15   &  366 & 3537\\\\\n",
       "\t 15   &  735 & 3537\\\\\n",
       "\t 15   & 1128 & 3537\\\\\n",
       "\t 15   & 1491 & 3537\\\\\n",
       "\t 15   & 1946 & 3537\\\\\n",
       "\t 15   & 2143 & 3537\\\\\n",
       "\t 15   & 2499 & 3537\\\\\n",
       "\t 15   & 2891 & 3537\\\\\n",
       "\t 15   & 3537 & 3537\\\\\n",
       "\t 150  &    0 & 2577\\\\\n",
       "\t 150  &  188 & 2577\\\\\n",
       "\t 150  &  365 & 2577\\\\\n",
       "\t 150  &  741 & 2577\\\\\n",
       "\t 150  & 1112 & 2577\\\\\n",
       "\t 150  & 1476 & 2577\\\\\n",
       "\t 150  & 1841 & 2577\\\\\n",
       "\t 150  & 2214 & 2577\\\\\n",
       "\t 150  & 2577 & 2577\\\\\n",
       "\t 151  &    0 & 3275\\\\\n",
       "\t 151  &  167 & 3275\\\\\n",
       "\t 151  &  356 & 3275\\\\\n",
       "\t 151  &  714 & 3275\\\\\n",
       "\t 151  & 1098 & 3275\\\\\n",
       "\t 151  & 1470 & 3275\\\\\n",
       "\t 151  & 1868 & 3275\\\\\n",
       "\t 151  & 2179 & 3275\\\\\n",
       "\t 151  & 2550 & 3275\\\\\n",
       "\t 151  & 2919 & 3275\\\\\n",
       "\t 151  & 3275 & 3275\\\\\n",
       "\t 152  &    0 &  832\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   id  visittime lastVisitTime\n",
       "1  15   196      3537         \n",
       "2  15   366      3537         \n",
       "3  15   735      3537         \n",
       "4  15  1128      3537         \n",
       "5  15  1491      3537         \n",
       "6  15  1946      3537         \n",
       "7  15  2143      3537         \n",
       "8  15  2499      3537         \n",
       "9  15  2891      3537         \n",
       "10 15  3537      3537         \n",
       "11 150    0      2577         \n",
       "12 150  188      2577         \n",
       "13 150  365      2577         \n",
       "14 150  741      2577         \n",
       "15 150 1112      2577         \n",
       "16 150 1476      2577         \n",
       "17 150 1841      2577         \n",
       "18 150 2214      2577         \n",
       "19 150 2577      2577         \n",
       "20 151    0      3275         \n",
       "21 151  167      3275         \n",
       "22 151  356      3275         \n",
       "23 151  714      3275         \n",
       "24 151 1098      3275         \n",
       "25 151 1470      3275         \n",
       "26 151 1868      3275         \n",
       "27 151 2179      3275         \n",
       "28 151 2550      3275         \n",
       "29 151 2919      3275         \n",
       "30 151 3275      3275         \n",
       "31 152    0       832         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ragged[400:430 , .(id, visittime, lastVisitTime)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a new variable indicating the number of times each person occurs in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ragged[ , numberObservations := .N, by = id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>id</th><th scope=col>visittime</th><th scope=col>numberObservations</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>1   </td><td>   0</td><td> 2  </td></tr>\n",
       "\t<tr><td>1   </td><td> 192</td><td> 2  </td></tr>\n",
       "\t<tr><td>10  </td><td>   0</td><td> 1  </td></tr>\n",
       "\t<tr><td>100 </td><td>   0</td><td> 4  </td></tr>\n",
       "\t<tr><td>100 </td><td> 171</td><td> 4  </td></tr>\n",
       "\t<tr><td>100 </td><td> 358</td><td> 4  </td></tr>\n",
       "\t<tr><td>100 </td><td> 514</td><td> 4  </td></tr>\n",
       "\t<tr><td>101 </td><td>   0</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td> 178</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td> 368</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td> 746</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td>1097</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td>1467</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td>1844</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td>2189</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td>2596</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td>3313</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td>3692</td><td>12  </td></tr>\n",
       "\t<tr><td>101 </td><td>4049</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td>   0</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td> 267</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td> 483</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td> 826</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td>1182</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td>1421</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td>1751</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td>2093</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td>2444</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td>2808</td><td>12  </td></tr>\n",
       "\t<tr><td>102 </td><td>3109</td><td>12  </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " id & visittime & numberObservations\\\\\n",
       "\\hline\n",
       "\t 1    &    0 &  2  \\\\\n",
       "\t 1    &  192 &  2  \\\\\n",
       "\t 10   &    0 &  1  \\\\\n",
       "\t 100  &    0 &  4  \\\\\n",
       "\t 100  &  171 &  4  \\\\\n",
       "\t 100  &  358 &  4  \\\\\n",
       "\t 100  &  514 &  4  \\\\\n",
       "\t 101  &    0 & 12  \\\\\n",
       "\t 101  &  178 & 12  \\\\\n",
       "\t 101  &  368 & 12  \\\\\n",
       "\t 101  &  746 & 12  \\\\\n",
       "\t 101  & 1097 & 12  \\\\\n",
       "\t 101  & 1467 & 12  \\\\\n",
       "\t 101  & 1844 & 12  \\\\\n",
       "\t 101  & 2189 & 12  \\\\\n",
       "\t 101  & 2596 & 12  \\\\\n",
       "\t 101  & 3313 & 12  \\\\\n",
       "\t 101  & 3692 & 12  \\\\\n",
       "\t 101  & 4049 & 12  \\\\\n",
       "\t 102  &    0 & 12  \\\\\n",
       "\t 102  &  267 & 12  \\\\\n",
       "\t 102  &  483 & 12  \\\\\n",
       "\t 102  &  826 & 12  \\\\\n",
       "\t 102  & 1182 & 12  \\\\\n",
       "\t 102  & 1421 & 12  \\\\\n",
       "\t 102  & 1751 & 12  \\\\\n",
       "\t 102  & 2093 & 12  \\\\\n",
       "\t 102  & 2444 & 12  \\\\\n",
       "\t 102  & 2808 & 12  \\\\\n",
       "\t 102  & 3109 & 12  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   id  visittime numberObservations\n",
       "1  1      0       2                \n",
       "2  1    192       2                \n",
       "3  10     0       1                \n",
       "4  100    0       4                \n",
       "5  100  171       4                \n",
       "6  100  358       4                \n",
       "7  100  514       4                \n",
       "8  101    0      12                \n",
       "9  101  178      12                \n",
       "10 101  368      12                \n",
       "11 101  746      12                \n",
       "12 101 1097      12                \n",
       "13 101 1467      12                \n",
       "14 101 1844      12                \n",
       "15 101 2189      12                \n",
       "16 101 2596      12                \n",
       "17 101 3313      12                \n",
       "18 101 3692      12                \n",
       "19 101 4049      12                \n",
       "20 102    0      12                \n",
       "21 102  267      12                \n",
       "22 102  483      12                \n",
       "23 102  826      12                \n",
       "24 102 1182      12                \n",
       "25 102 1421      12                \n",
       "26 102 1751      12                \n",
       "27 102 2093      12                \n",
       "28 102 2444      12                \n",
       "29 102 2808      12                \n",
       "30 102 3109      12                "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(ragged[ , .(id, visittime, numberObservations)], 30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do calculations by two variables.\n",
    "\n",
    "Get the mean platelets for each patient within each level of stage, and add that as a new variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ragged[ , patientMeanPlateletByStage := mean(platelets, na.rm = TRUE), by = .(id, stage)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>id</th><th scope=col>platelets</th><th scope=col>stage</th><th scope=col>patientMeanPlateletByStage</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>15      </td><td>10.4    </td><td>3       </td><td>10.70000</td></tr>\n",
       "\t<tr><td>15      </td><td>10.4    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>15      </td><td>10.1    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>15      </td><td>10.0    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>15      </td><td> 9.8    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>15      </td><td>10.4    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>15      </td><td>10.7    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>15      </td><td>10.5    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>15      </td><td>10.7    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>15      </td><td>13.0    </td><td>4       </td><td>10.62222</td></tr>\n",
       "\t<tr><td>150     </td><td>10.0    </td><td>1       </td><td> 9.65000</td></tr>\n",
       "\t<tr><td>150     </td><td> 9.3    </td><td>1       </td><td> 9.65000</td></tr>\n",
       "\t<tr><td>150     </td><td>10.1    </td><td>2       </td><td>10.00000</td></tr>\n",
       "\t<tr><td>150     </td><td> 9.9    </td><td>2       </td><td>10.00000</td></tr>\n",
       "\t<tr><td>150     </td><td>10.0    </td><td>3       </td><td>10.00000</td></tr>\n",
       "\t<tr><td>150     </td><td> 9.9    </td><td>4       </td><td>10.90000</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " id & platelets & stage & patientMeanPlateletByStage\\\\\n",
       "\\hline\n",
       "\t 15       & 10.4     & 3        & 10.70000\\\\\n",
       "\t 15       & 10.4     & 4        & 10.62222\\\\\n",
       "\t 15       & 10.1     & 4        & 10.62222\\\\\n",
       "\t 15       & 10.0     & 4        & 10.62222\\\\\n",
       "\t 15       &  9.8     & 4        & 10.62222\\\\\n",
       "\t 15       & 10.4     & 4        & 10.62222\\\\\n",
       "\t 15       & 10.7     & 4        & 10.62222\\\\\n",
       "\t 15       & 10.5     & 4        & 10.62222\\\\\n",
       "\t 15       & 10.7     & 4        & 10.62222\\\\\n",
       "\t 15       & 13.0     & 4        & 10.62222\\\\\n",
       "\t 150      & 10.0     & 1        &  9.65000\\\\\n",
       "\t 150      &  9.3     & 1        &  9.65000\\\\\n",
       "\t 150      & 10.1     & 2        & 10.00000\\\\\n",
       "\t 150      &  9.9     & 2        & 10.00000\\\\\n",
       "\t 150      & 10.0     & 3        & 10.00000\\\\\n",
       "\t 150      &  9.9     & 4        & 10.90000\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   id  platelets stage patientMeanPlateletByStage\n",
       "1  15  10.4      3     10.70000                  \n",
       "2  15  10.4      4     10.62222                  \n",
       "3  15  10.1      4     10.62222                  \n",
       "4  15  10.0      4     10.62222                  \n",
       "5  15   9.8      4     10.62222                  \n",
       "6  15  10.4      4     10.62222                  \n",
       "7  15  10.7      4     10.62222                  \n",
       "8  15  10.5      4     10.62222                  \n",
       "9  15  10.7      4     10.62222                  \n",
       "10 15  13.0      4     10.62222                  \n",
       "11 150 10.0      1      9.65000                  \n",
       "12 150  9.3      1      9.65000                  \n",
       "13 150 10.1      2     10.00000                  \n",
       "14 150  9.9      2     10.00000                  \n",
       "15 150 10.0      3     10.00000                  \n",
       "16 150  9.9      4     10.90000                  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ragged[400:415, .(id, platelets, stage, patientMeanPlateletByStage)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a new variable which gives each patient's immediately previous platelet measure. \n",
    "\n",
    "NB: The data must be sorted by time for this to work. How do we know that these data are already sorted by visittime?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ragged[ , plateletsLag1 :=  shift(platelets), by = id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>id</th><th scope=col>visittime</th><th scope=col>platelets</th><th scope=col>plateletsLag1</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>186 </td><td> 782</td><td>11.8</td><td>11.7</td></tr>\n",
       "\t<tr><td>187 </td><td>   0</td><td>11.5</td><td>  NA</td></tr>\n",
       "\t<tr><td>187 </td><td> 217</td><td>11.5</td><td>11.5</td></tr>\n",
       "\t<tr><td>187 </td><td> 356</td><td>11.1</td><td>11.5</td></tr>\n",
       "\t<tr><td>187 </td><td> 729</td><td>20.8</td><td>11.1</td></tr>\n",
       "\t<tr><td>188 </td><td>   0</td><td>11.0</td><td>  NA</td></tr>\n",
       "\t<tr><td>188 </td><td> 380</td><td>11.0</td><td>11.0</td></tr>\n",
       "\t<tr><td>188 </td><td> 538</td><td>10.2</td><td>11.0</td></tr>\n",
       "\t<tr><td>188 </td><td> 930</td><td>10.8</td><td>10.2</td></tr>\n",
       "\t<tr><td>188 </td><td>1324</td><td>10.3</td><td>10.8</td></tr>\n",
       "\t<tr><td>188 </td><td>1644</td><td>11.6</td><td>10.3</td></tr>\n",
       "\t<tr><td>188 </td><td>2001</td><td>11.6</td><td>11.6</td></tr>\n",
       "\t<tr><td>188 </td><td>2351</td><td>11.8</td><td>11.6</td></tr>\n",
       "\t<tr><td>189 </td><td>   0</td><td> 9.9</td><td>  NA</td></tr>\n",
       "\t<tr><td>189 </td><td> 211</td><td>11.1</td><td> 9.9</td></tr>\n",
       "\t<tr><td>189 </td><td> 355</td><td> 9.7</td><td>11.1</td></tr>\n",
       "\t<tr><td>189 </td><td> 733</td><td> 9.9</td><td> 9.7</td></tr>\n",
       "\t<tr><td>189 </td><td>1098</td><td> 9.8</td><td> 9.9</td></tr>\n",
       "\t<tr><td>19  </td><td>   0</td><td>11.0</td><td>  NA</td></tr>\n",
       "\t<tr><td>19  </td><td> 194</td><td>11.2</td><td>11.0</td></tr>\n",
       "\t<tr><td>19  </td><td> 404</td><td>10.4</td><td>11.2</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " id & visittime & platelets & plateletsLag1\\\\\n",
       "\\hline\n",
       "\t 186  &  782 & 11.8 & 11.7\\\\\n",
       "\t 187  &    0 & 11.5 &   NA\\\\\n",
       "\t 187  &  217 & 11.5 & 11.5\\\\\n",
       "\t 187  &  356 & 11.1 & 11.5\\\\\n",
       "\t 187  &  729 & 20.8 & 11.1\\\\\n",
       "\t 188  &    0 & 11.0 &   NA\\\\\n",
       "\t 188  &  380 & 11.0 & 11.0\\\\\n",
       "\t 188  &  538 & 10.2 & 11.0\\\\\n",
       "\t 188  &  930 & 10.8 & 10.2\\\\\n",
       "\t 188  & 1324 & 10.3 & 10.8\\\\\n",
       "\t 188  & 1644 & 11.6 & 10.3\\\\\n",
       "\t 188  & 2001 & 11.6 & 11.6\\\\\n",
       "\t 188  & 2351 & 11.8 & 11.6\\\\\n",
       "\t 189  &    0 &  9.9 &   NA\\\\\n",
       "\t 189  &  211 & 11.1 &  9.9\\\\\n",
       "\t 189  &  355 &  9.7 & 11.1\\\\\n",
       "\t 189  &  733 &  9.9 &  9.7\\\\\n",
       "\t 189  & 1098 &  9.8 &  9.9\\\\\n",
       "\t 19   &    0 & 11.0 &   NA\\\\\n",
       "\t 19   &  194 & 11.2 & 11.0\\\\\n",
       "\t 19   &  404 & 10.4 & 11.2\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   id  visittime platelets plateletsLag1\n",
       "1  186  782      11.8      11.7         \n",
       "2  187    0      11.5        NA         \n",
       "3  187  217      11.5      11.5         \n",
       "4  187  356      11.1      11.5         \n",
       "5  187  729      20.8      11.1         \n",
       "6  188    0      11.0        NA         \n",
       "7  188  380      11.0      11.0         \n",
       "8  188  538      10.2      11.0         \n",
       "9  188  930      10.8      10.2         \n",
       "10 188 1324      10.3      10.8         \n",
       "11 188 1644      11.6      10.3         \n",
       "12 188 2001      11.6      11.6         \n",
       "13 188 2351      11.8      11.6         \n",
       "14 189    0       9.9        NA         \n",
       "15 189  211      11.1       9.9         \n",
       "16 189  355       9.7      11.1         \n",
       "17 189  733       9.9       9.7         \n",
       "18 189 1098       9.8       9.9         \n",
       "19 19     0      11.0        NA         \n",
       "20 19   194      11.2      11.0         \n",
       "21 19   404      10.4      11.2         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ragged[650:670 , .(id, visittime, platelets, plateletsLag1)]"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}