Dplyr: A grammar of data manipulation
R C++ Other
Latest commit 8430adc Feb 27, 2017 @zeehio zeehio committed with Combine character and factor show warn. Closes #2317 (#2478)
My previous PR did not fix this case.
Permalink
Failed to load latest commit information.
R Repair names in bind_cols Feb 27, 2017
data-raw Adds storms data set with documentation. (#2431) Feb 16, 2017
data Adds storms data set with documentation. (#2431) Feb 16, 2017
inst Combine character and factor show warn. Closes #2317 (#2478) Feb 27, 2017
man Document recode_factor more clearly Feb 27, 2017
revdep Prepare for release Jun 23, 2016
src SymbolString cleanup Feb 25, 2017
tests Combine character and factor show warn. Closes #2317 (#2478) Feb 27, 2017
vignettes Use TRUE, not T Feb 21, 2017
.Rbuildignore Add full license (as md) Feb 17, 2017
.gitattributes add .gitattributes Nov 25, 2015
.gitignore ignore Sep 29, 2016
.travis.yml Simplify hybrid evaluation (#2190) Dec 9, 2016
API add v0.5.0 API Jan 23, 2017
CMakeLists.txt Use String instead of Symbol (#2388) Feb 21, 2017
DESCRIPTION Combine strips unknown classes. Support integer64. (#2425) Feb 26, 2017
Doxyfile fine-tune Sep 29, 2016
ISSUE_TEMPLATE.md Add issue template Feb 14, 2017
LICENSE update copyright years Apr 14, 2015
LICENSE.md Add full license (as md) Feb 17, 2017
NAMESPACE Implement pull Feb 24, 2017
NEWS.md Combine character and factor show warn. Closes #2317 (#2478) Feb 27, 2017
README.Rmd Use travis svg icon Feb 14, 2017
README.md Use travis svg icon Feb 14, 2017
appveyor.yml don't install devtools from GitHub Aug 24, 2016
codecov.yml Supress codecov comments Jun 20, 2016
cran-comments.md Prepare for release Jun 23, 2016
dplyr.Rproj install tests Jun 21, 2016

README.md

dplyr

Build Status AppVeyor Build Status CRAN\_Status\_Badge Coverage Status

dplyr is the next iteration of plyr, focussed on tools for working with data frames (hence the d in the name). It has three main goals:

  • Identify the most important data manipulation tools needed for data analysis and make them easy to use from R.

  • Provide blazing fast performance for in-memory data by writing key pieces in C++.

  • Use the same interface to work with data no matter where it's stored, whether in a data frame, a data table or database.

You can install:

  • the latest released version from CRAN with

    install.packages("dplyr")
  • the latest development version from github with

    if (packageVersion("devtools") < 1.6) {
      install.packages("devtools")
    }
    devtools::install_github("hadley/lazyeval")
    devtools::install_github("hadley/dplyr")

You'll probably also want to install the data packages used in most examples: install.packages(c("nycflights13", "Lahman")).

If you encounter a clear bug, please file a minimal reproducible example on github. For questions and other discussion, please use the manipulatr mailing list.

Learning dplyr

To get started, read the notes below, then read the intro vignette: vignette("introduction", package = "dplyr"). To make the most of dplyr, I also recommend that you familiarise yourself with the principles of tidy data: this will help you get your data into a form that works well with dplyr, ggplot2 and R's many modelling functions.

If you need more help, I recommend the following (paid) resources:

  • dplyr on datacamp, by Garrett Grolemund. Learn the basics of dplyr at your own pace in this interactive online course.

  • Introduction to Data Science with R: How to Manipulate, Visualize, and Model Data with the R Language, by Garrett Grolemund. This O'Reilly video series will teach you the basics needed to be an effective analyst in R.

Key data structures

The key object in dplyr is a tbl, a representation of a tabular data structure. Currently dplyr supports:

You can create them as follows:

library(dplyr) # for functions
library(nycflights13) # for data
flights
#> # A tibble: 336,776 × 19
#>     year month   day dep_time sched_dep_time dep_delay arr_time
#>    <int> <int> <int>    <int>          <int>     <dbl>    <int>
#> 1   2013     1     1      517            515         2      830
#> 2   2013     1     1      533            529         4      850
#> 3   2013     1     1      542            540         2      923
#> 4   2013     1     1      544            545        -1     1004
#> 5   2013     1     1      554            600        -6      812
#> 6   2013     1     1      554            558        -4      740
#> 7   2013     1     1      555            600        -5      913
#> 8   2013     1     1      557            600        -3      709
#> 9   2013     1     1      557            600        -3      838
#> 10  2013     1     1      558            600        -2      753
#> # ... with 336,766 more rows, and 12 more variables: sched_arr_time <int>,
#> #   arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
#> #   origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
#> #   minute <dbl>, time_hour <dttm>

# Caches data in local SQLite db
flights_db1 <- tbl(nycflights13_sqlite(), "flights")

# Caches data in local postgres db
flights_db2 <- tbl(nycflights13_postgres(host = "localhost"), "flights")

Each tbl also comes in a grouped variant which allows you to easily perform operations "by group":

carriers_df  <- flights %>% group_by(carrier)
carriers_db1 <- flights_db1 %>% group_by(carrier)
carriers_db2 <- flights_db2 %>% group_by(carrier)

Single table verbs

dplyr implements the following verbs useful for data manipulation:

  • select(): focus on a subset of variables
  • filter(): focus on a subset of rows
  • mutate(): add new columns
  • summarise(): reduce each group to a smaller number of summary statistics
  • arrange(): re-order the rows

They all work as similarly as possible across the range of data sources. The main difference is performance:

system.time(carriers_df %>% summarise(delay = mean(arr_delay)))
#>    user  system elapsed 
#>   0.048   0.001   0.051
system.time(carriers_db1 %>% summarise(delay = mean(arr_delay)) %>% collect())
#>    user  system elapsed 
#>   0.231   0.201   0.750
system.time(carriers_db2 %>% summarise(delay = mean(arr_delay)) %>% collect())
#>    user  system elapsed 
#>   0.005   0.001   0.133

Data frame methods are much much faster than the plyr equivalent. The database methods are slower, but can work with data that don't fit in memory.

system.time(plyr::ddply(flights, "carrier", plyr::summarise,
  delay = mean(arr_delay, na.rm = TRUE)))
#>    user  system elapsed 
#>   0.119   0.041   0.163

do()

As well as the specialised operations described above, dplyr also provides the generic do() function which applies any R function to each group of the data.

Let's take the batting database from the built-in Lahman database. We'll group it by year, and then fit a model to explore the relationship between their number of at bats and runs:

by_year <- lahman_df() %>% 
  tbl("Batting") %>%
  group_by(yearID)
by_year %>% 
  do(mod = lm(R ~ AB, data = .))
#> Source: local data frame [145 x 2]
#> Groups: <by row>
#> 
#> # A tibble: 145 × 2
#>    yearID      mod
#> *   <int>   <list>
#> 1    1871 <S3: lm>
#> 2    1872 <S3: lm>
#> 3    1873 <S3: lm>
#> 4    1874 <S3: lm>
#> 5    1875 <S3: lm>
#> 6    1876 <S3: lm>
#> 7    1877 <S3: lm>
#> 8    1878 <S3: lm>
#> 9    1879 <S3: lm>
#> 10   1880 <S3: lm>
#> # ... with 135 more rows

Note that if you are fitting lots of linear models, it's a good idea to use biglm because it creates model objects that are considerably smaller:

by_year %>% 
  do(mod = lm(R ~ AB, data = .)) %>%
  object.size() %>%
  print(unit = "MB")
#> 23.1 Mb

by_year %>% 
  do(mod = biglm::biglm(R ~ AB, data = .)) %>%
  object.size() %>%
  print(unit = "MB")
#> 0.8 Mb

Multiple table verbs

As well as verbs that work on a single tbl, there are also a set of useful verbs that work with two tbls at a time: joins and set operations.

dplyr implements the four most useful joins from SQL:

  • inner_join(x, y): matching x + y
  • left_join(x, y): all x + matching y
  • semi_join(x, y): all x with match in y
  • anti_join(x, y): all x without match in y

And provides methods for:

  • intersect(x, y): all rows in both x and y
  • union(x, y): rows in either x or y
  • setdiff(x, y): rows in x, but not y

Plyr compatibility

You'll need to be a little careful if you load both plyr and dplyr at the same time. I'd recommend loading plyr first, then dplyr, so that the faster dplyr functions come first in the search path. By and large, any function provided by both dplyr and plyr works in a similar way, although dplyr functions tend to be faster and more general.

Related approaches