Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
352 lines (268 sloc) 13.9 KB
---
title: "Vroom Benchmarks"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Vroom Benchmarks}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>"
)
library(ggplot2)
library(forcats)
library(dplyr)
library(tidyr)
pretty_sec <- function(x) {
x[!is.na(x)] <- prettyunits::pretty_sec(x[!is.na(x)])
x
}
read_benchmark <- function(file, desc) {
vroom::vroom(file, col_types = c("cccdddd")) %>%
filter(type == "real", op != "setup") %>%
mutate(
method = case_when(
package == "data.table" ~ "data.table",
package == "readr" ~ "dplyr",
package == "read.delim" ~ "base",
package == "write.delim" ~ "base",
grepl("^vroom", package) ~ sub(".*_", "", package)
),
altrep = case_when(
grepl("^vroom [(]full altrep[)]", package) ~ "full",
grepl("^vroom [(]no altrep[)]", package) ~ "none",
grepl("^vroom_", package) ~ "normal",
TRUE ~ NA_character_
),
package = case_when(
grepl("^vroom", package) ~ "vroom",
TRUE ~ package
),
label = fct_reorder(
glue::glue("{package}{altrep}\n{method}",
altrep = ifelse(is.na(altrep), "", glue::glue("(altrep: {altrep})"))
),
time,
sum),
op = factor(op, desc)
)
}
generate_subtitle <- function(data) {
rows <- scales::comma(data$rows[[1]])
cols <- scales::comma(data$cols[[1]])
size <- fs::fs_bytes(data$size[[1]])
glue::glue("{rows} x {cols} - {size}B")
}
plot_benchmark <- function(data, title) {
subtitle <- generate_subtitle(data)
data %>%
filter(package != "read.delim") %>%
ggplot() +
geom_bar(aes(x = label, y = time, fill = op, group = label), stat = "identity") +
scale_fill_brewer(type = "qual", palette = "Set2") +
scale_y_continuous(labels = scales::number_format(suffix = "s")) +
theme(legend.position = "bottom") +
coord_flip() +
labs(title = title, subtitle = subtitle, x = NULL, y = NULL, fill = NULL)
}
make_table <- function(data) {
data %>%
select(-label, -type, -size, -rows, -cols) %>%
spread(op, time) %>%
mutate(
total = read + print + head + tail + sample + filter + aggregate,
) %>%
rename(manip = method) %>%
arrange(desc(total)) %>%
mutate_if(is.numeric, pretty_sec) %>%
knitr::kable(digits = 2, align = "r", format = "html")
}
desc <- c("setup", "read", "print", "head", "tail", "sample", "filter", "aggregate")
```
vroom is a new approach to reading delimited and fixed width data into R.
It stems from the observation that when parsing files reading data from disk
and finding the delimiters is generally not the main bottle neck. Instead
(re)-allocating memory and parsing the values into R data types (particularly
for characters) takes the bulk of the time.
Therefore you can obtain very rapid input by first performing a fast indexing
step and then using the Altrep framework available in R versions 3.5+ to
access the values in a lazy / delayed fashion.
## How it works
The initial reading of the file simply records the locations of each individual
record, the actual values are not read into R. Altrep vectors are
created for each column in the data which hold a pointer to the index and the
memory mapped file. When these vectors are indexed the value is read from the
memory mapping.
This means initial reading is extremely fast, in the real world dataset below
it is ~ 1/4 the time of the multi-threaded `data.table::fread()`. Sampling
operations are likewise extremely fast, as only the data actually included in
the sample is read. This means things like the tibble print method, calling
`head()`, `tail()` `x[sample(), ]` etc. have very low overhead. Filtering also
can be fast, only the columns included in the filter selection have to be fully
read and only the data in the filtered rows needs to be read from the remaining
columns. Grouped aggregations likewise only need to read the grouping
variables and the variables aggregated.
Once a particular vector is fully materialized the speed for all subsequent
operations should be identical to a normal R vector.
This approach potentially also allows you to work with data that is larger than
memory. As long as you are careful to avoid materializing the entire dataset at
once it can be efficiently queried and subset.
# Reading delimited files
The following benchmarks all measure reading delimited files of various sizes
and data types. Because vroom delays reading the benchmarks also do some
manipulation of the data afterwards to try and provide a more realistic
performance comparison.
Because the `read.delim` results are so much slower than the others they are
excluded from the plots, but are retained in the tables.
## Taxi Trip Dataset
This real world dataset is from Freedom of Information Law (FOIL)
Taxi Trip Data from the NYC Taxi and Limousine Commission 2013, originally
posted at <http://chriswhong.com/open-data/foil_nyc_taxi/>. It is also hosted
on [archive.org](https://archive.org/details/nycTaxiTripData2013).
The first table trip_fare_1.csv is 1.55G in size.
#> Observations: 14,776,615
#> Variables: 11
#> $ medallion <chr> "89D227B655E5C82AECF13C3F540D4CF4", "0BD7C8F5B...
#> $ hack_license <chr> "BA96DE419E711691B9445D6A6307C170", "9FD8F69F0...
#> $ vendor_id <chr> "CMT", "CMT", "CMT", "CMT", "CMT", "CMT", "CMT...
#> $ pickup_datetime <chr> "2013-01-01 15:11:48", "2013-01-06 00:18:35", ...
#> $ payment_type <chr> "CSH", "CSH", "CSH", "CSH", "CSH", "CSH", "CSH...
#> $ fare_amount <dbl> 6.5, 6.0, 5.5, 5.0, 9.5, 9.5, 6.0, 34.0, 5.5, ...
#> $ surcharge <dbl> 0.0, 0.5, 1.0, 0.5, 0.5, 0.0, 0.0, 0.0, 1.0, 0...
#> $ mta_tax <dbl> 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0...
#> $ tip_amount <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
#> $ tolls_amount <dbl> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 4.8, 0.0, 0...
#> $ total_amount <dbl> 7.0, 7.0, 7.0, 6.0, 10.5, 10.0, 6.5, 39.3, 7.0...
### Taxi Benchmarks
The code used to run the taxi benchmarks is in
[bench/taxi-benchmark.R](https://github.com/r-lib/vroom/blob/master/inst/bench/taxi-benchmark.R).
The benchmarks labeled `vroom_base` uses `vroom` with base functions for
manipulation. `vroom_dplyr` uses `vroom` to read the file and dplyr functions to
manipulate. `data.table` uses `fread()` to read the file and `data.table`
functions to manipulate and `readr` uses `readr` to read the file and `dplyr` to
manipulate. By default vroom only uses Altrep for character vectors, these are
labeled `vroom(altrep: normal)`. The benchmarks labeled `vroom(altrep: full)`
instead use Altrep vectors for all supported types and `vroom(altrep: none)`
disable Altrep entirely.
The following operations are performed.
- The data is read
- `print()` - _N.B. read.delim uses `print(head(x, 10))` because printing the whole dataset takes > 10 minutes_
- `head()`
- `tail()`
- Sampling 100 random rows
- Filtering for "UNK" payment, this is 6434 rows (0.0435% of total).
- Aggregation of mean fare amount per payment type.
```{r, fig.height = 8, fig.width=10, warning = FALSE, echo = FALSE, message = FALSE}
taxi <- read_benchmark(system.file("bench", "taxi-times.tsv", package = "vroom"), desc)
plot_benchmark(taxi, "Time to analyze taxi trip data")
make_table(taxi)
```
(*N.B. Rcpp used in the dplyr implementation
fully materializes all the Altrep numeric vectors when using `filter()` or `sample_n()`,
which is why the first of these cases have additional overhead when using full Altrep.*).
## All numeric data
The code used to run the all numeric benchmarks is in
[bench/all_numeric-benchmark.R](https://github.com/r-lib/vroom/blob/master/inst/bench/all_numeric-benchmark.R).
All numeric data is really a worst case scenario for vroom. The index takes
about as much memory as the parsed data. Also because parsing doubles can be
done quickly in parallel and text representations of doubles are only ~25
characters at most there isn't a great deal of savings for delayed parsing.
For these reasons (and because the data.table implementation is very fast)
vroom is a bit slower than fread for pure numeric data.
However the vroom is multi-threaded and therefore is quicker than readr and
read.delim.
```{r, fig.height = 8, fig.width=10, warning = FALSE, message = FALSE, echo = FALSE}
all_num <- read_benchmark(system.file("bench", "all_numeric-times.tsv", package = "vroom"), desc)
plot_benchmark(all_num, "Time to analyze all numeric data")
make_table(all_num)
```
### Wide all numeric data
The code used to run the wide all numeric benchmarks is in
[bench/all_numeric-benchmark.R](https://github.com/r-lib/vroom/blob/master/inst/bench/all_numeric_wide-benchmark.R).
```{r, fig.height = 8, fig.width=10, warning = FALSE, message = FALSE, echo = FALSE}
all_num_wide <- read_benchmark(system.file("bench", "all_numeric_wide-times.tsv", package = "vroom"), desc)
plot_benchmark(all_num_wide, "Time to analyze wide all numeric data")
make_table(all_num_wide)
```
## All character data
The code used to run the all character benchmarks is in
[bench/all_character-benchmark.R](https://github.com/r-lib/vroom/blob/master/inst/bench/all_character-benchmark.R).
All character data is a best case scenario for vroom when using Altrep, as it takes full advantage of the lazy reading.
```{r, fig.height = 8, fig.width=10, warning = FALSE, message = FALSE, echo = FALSE}
all_chr <- read_benchmark(system.file("bench", "all_character-times.tsv", package = "vroom"), desc)
plot_benchmark(all_chr, "Time to analyze all character data")
make_table(all_chr)
```
### Wide all character data
The code used to run the wide all character benchmarks is in
[bench/all_character-benchmark.R](https://github.com/r-lib/vroom/blob/master/inst/bench/all_character_wide-benchmark.R).
```{r, fig.height = 8, fig.width=10, warning = FALSE, message = FALSE, echo = FALSE}
all_chr_wide <- read_benchmark(system.file("bench", "all_character_wide-times.tsv", package = "vroom"), desc)
plot_benchmark(all_chr_wide, "Time to analyze wide all character data")
make_table(all_chr_wide)
```
# Reading multiple delimited files
The code used to run the taxi multiple file benchmarks is at
[bench/taxi_multiple-benchmark.R](https://github.com/r-lib/vroom/blob/master/inst/bench/taxi_multiple-benchmark.R).
```{r, echo = FALSE, message = FALSE}
mult <- read_benchmark(system.file("bench", "taxi_multiple-times.tsv", package = "vroom"), desc)
```
The benchmark reads all 12 files in the taxi trip fare data, totaling `r scales::comma(mult$rows[[1]])` rows
and `r mult$cols[[1]]` columns for a total file size of `r format(fs::fs_bytes(mult$size[[1]]))`.
```{r, fig.height = 8, fig.width=10, warning = FALSE, message = FALSE, echo = FALSE}
plot_benchmark(mult, "Time to analyze multiple file data")
make_table(mult)
```
# Reading fixed width files
## United States Census 5-Percent Public Use Microdata Sample files
```{r, echo = FALSE, message = FALSE}
fwf <- read_benchmark(system.file("bench", "fwf-times.tsv", package = "vroom"), desc)
```
This fixed width dataset contains individual records of the characteristics of a 5 percent sample of people and housing units from the year 2000 and is freely available at <https://www2.census.gov/census_2000/datasets/PUMS/FivePercent/California/all_California.zip>. The data is split into files by state, and the state of California was used in this benchmark.
The data totals `r scales::comma(fwf$rows[[1]])` rows
and `r fwf$cols[[1]]` columns with a total file size of `r format(fs::fs_bytes(fwf$size[[1]]))`.
## Census data benchmarks
The code used to run the census data benchmarks is at
[bench/fwf-benchmark.R](https://github.com/r-lib/vroom/blob/master/inst/bench/fwf-benchmark.R).
```{r, fig.height = 8, fig.width=10, warning = FALSE, message = FALSE, echo = FALSE}
plot_benchmark(fwf, "Time to analyze fixed width data")
make_table(fwf)
```
# Writing delimited files
The code used to run the taxi writing benchmarks is at
[bench/taxi_writing-benchmark.R](https://github.com/r-lib/vroom/blob/master/inst/bench/taxi_writing-benchmark.R).
The benchmarks write out the taxi trip dataset in three different ways.
- An uncompressed file
- A gzip compressed file using `gzfile()` _(readr and vroom do this automatically for files ending in `.gz`)_
- A multithreaded gzip compressed file using a `pipe()` connection to [pigz](https://zlib.net/pigz/)
*Note the current CRAN version of data.table (1.12.2) does not support writing directly to gzip compressed files, though the developement version (1.12.3) does *
```{r, fig.height = 8, fig.width=10, warning = FALSE, message = FALSE, echo = FALSE}
desc_w <- rev(c("xz", "gzip", "multithreaded gzip", "zstandard", "uncompressed"))
taxi_writing <- read_benchmark(system.file("bench", "taxi_writing-times.tsv", package = "vroom"), desc_w) %>%
mutate(package = factor(package, c("write.delim", "readr", "data.table", "vroom")))
subtitle <- generate_subtitle(taxi_writing)
taxi_writing %>%
ggplot(aes(x = op, y = time, fill = package)) +
geom_bar(stat = "identity", position = position_dodge2(reverse = TRUE, padding = .05)) +
scale_fill_brewer(type = "qual", palette = "Set2") +
scale_y_continuous(labels = scales::number_format(suffix = "s")) +
theme(legend.position = "bottom") +
coord_flip() +
labs(title = "Writing taxi trip data", subtitle = subtitle, x = NULL, y = NULL, fill = NULL)
taxi_writing %>%
select(-size, -rows, -cols, -type, -method, -altrep, -label) %>%
mutate_if(is.numeric, pretty_sec) %>%
spread(package, time) %>%
arrange(desc(op)) %>%
rename(method = op) %>%
knitr::kable(digits = 2, align = "r", format = "html")
```
## Session and package information
```{r, echo = FALSE, warning = FALSE, message = FALSE}
si <- vroom::vroom(system.file("bench", "sessioninfo.tsv", package = "vroom"))
class(si) <- c("packages_info", "data.frame")
select(as.data.frame(si), package, version = ondiskversion, date, source) %>%
knitr::kable()
```
You can’t perform that action at this time.