flights.df %>% sample_frac(0.01) %>% collect %>% head
#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
-#> 1: 2013 1 10 1614 1605 9 1926 1944
-#> 2: 2013 5 13 1136 1136 0 1225 1234
-#> 3: 2013 5 16 1428 1420 8 1535 1538
-#> 4: 2013 12 29 1536 1520 16 1817 1750
-#> 5: 2013 12 21 849 820 29 1315 1345
-#> 6: 2013 12 25 1624 1559 25 1846 1825
+#> 1: 2013 5 24 1452 1455 -3 1840 1820
+#> 2: 2013 1 10 939 905 34 1259 1235
+#> 3: 2013 2 4 1911 1915 -4 2117 2111
+#> 4: 2013 5 12 1634 1545 49 1940 1916
+#> 5: 2013 1 16 2315 2250 25 40 7
+#> 6: 2013 12 28 913 900 13 1215 1227
#> arr_delay carrier flight tailnum origin dest air_time distance hour minute
-#> 1: -18 DL 1508 N952DL JFK RSW 163 1074 16 5
-#> 2: -9 EV 3830 N13955 EWR PVD 35 160 11 36
-#> 3: -3 EV 4284 N11536 EWR ROC 45 246 14 20
-#> 4: 27 MQ 3553 N520MQ LGA XNA 181 1147 15 20
-#> 5: -30 DL 454 N682DA JFK STT 188 1623 8 20
-#> 6: 21 EV 5567 N870AS LGA CAE 99 617 15 59
+#> 1: 20 AA 1769 N372AA JFK MIA 165 1089 14 55
+#> 2: 24 VX 407 N622VA JFK LAX 349 2475 9 5
+#> 3: 6 9E 3525 N903XJ JFK ORD 127 740 19 15
+#> 4: 24 DL 1773 N3739P JFK SLC 262 1990 15 45
+#> 5: 33 B6 30 N187JB JFK ROC 59 264 22 50
+#> 6: -12 DL 422 N713TW JFK LAX 327 2475 9 0
#> time_hour
-#> 1: 2013-01-10 21:00:00
-#> 2: 2013-05-13 15:00:00
-#> 3: 2013-05-16 18:00:00
-#> 4: 2013-12-29 20:00:00
-#> 5: 2013-12-21 13:00:00
-#> 6: 2013-12-25 20:00:00
+#> 1: 2013-05-24 18:00:00
+#> 2: 2013-01-10 14:00:00
+#> 3: 2013-02-05 00:00:00
+#> 4: 2013-05-12 19:00:00
+#> 5: 2013-01-17 03:00:00
+#> 6: 2013-12-28 14:00:00
We note that there is some time needed for disk.frame to start up all the workers. Next we try to convert the largest CSV file to disk.frame format. The file to be converted is about 2.2GB in size
time_to_convert_disk.frame = system.time(df1 <- csv_to_disk.frame("c:/data/Performance_2004Q3.txt", header = FALSE))[3]
time_to_convert_disk.frame
#> elapsed
-#> 28.3Now that we have converted it, we want to a count by the first column. To achieve this we use a “two-stage” aggregation strategy. Note that use keep="V1" to bring only the column V1 into RAM. This avoids the reading of other unnecessary columns and should speed-up the analysis significantly
time_to_agg_disk.frame = system.time(summ <- df1[,.N, V1, keep = "V1"][, .(N = sum(N)), V1])
@@ -204,7 +204,7 @@ disk.frametime_to_agg_disk.frame
#> user system elapsed
-#> 0.13 0.03 8.48
We can inspect the result as well.
summ
@@ -228,7 +228,7 @@ disk.framesummarise(N = n()) %>%
collect)
#> user system elapsed
-#> 1.53 0.17 10.25
However, the dplyr syntax tends to be slightly slower than using data.table syntax. This may be improved as much of the overhead is due to inefficient use of NSE.
ZJ D (2022). disk.frame: Larger-than-RAM Disk-Based Data Manipulation Framework. -R package version 0.6.0, https://diskframe.com. +R package version 0.7, https://diskframe.com.
@Manual{,
title = {disk.frame: Larger-than-RAM Disk-Based Data Manipulation Framework},
author = {Dai ZJ},
year = {2022},
- note = {R package version 0.6.0},
+ note = {R package version 0.7},
url = {https://diskframe.com},
}
diff --git a/docs/index.html b/docs/index.html
index 7ea2fc4e..f86e68c5 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -37,7 +37,7 @@
@@ -228,6 +228,7 @@
# where is the disk.frame stored
attr(flights.df, "path")
-#> [1] "C:\\Users\\RTX2080\\AppData\\Local\\Temp\\RtmpQH7obF\\file42d452c32907.df"
+#> [1] "C:\\Users\\RTX2080\\AppData\\Local\\Temp\\RtmpQJt0m9\\file393027da33b.df"
A number of data.frame functions are implemented for disk.frame
# get first few rows
head(flights.df, 1)
-#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
-#> 1: 2013 1 1 517 515 2 830 819
-#> arr_delay carrier flight tailnum origin dest air_time distance hour minute
-#> 1: 11 UA 1545 N14228 EWR IAH 227 1400 5 15
+#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay
+#> 1: 2013 1 1 517 515 2 830 819 11
+#> carrier flight tailnum origin dest air_time distance hour minute
+#> 1: UA 1545 N14228 EWR IAH 227 1400 5 15
#> time_hour
#> 1: 2013-01-01 05:00:00
# get last few rows
tail(flights.df, 1)
-#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
-#> 1: 2013 9 30 NA 840 NA NA 1020
-#> arr_delay carrier flight tailnum origin dest air_time distance hour minute
-#> 1: NA MQ 3531 N839MQ LGA RDU NA 431 8 40
+#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay
+#> 1: 2013 9 30 NA 840 NA NA 1020 NA
+#> carrier flight tailnum origin dest air_time distance hour minute
+#> 1: MQ 3531 N839MQ LGA RDU NA 431 8 40
#> time_hour
#> 1: 2013-09-30 08:00:00
diff --git a/docs/news/index.html b/docs/news/index.html
index 48d1c26d..551c9495 100644
--- a/docs/news/index.html
+++ b/docs/news/index.html
@@ -17,7 +17,7 @@
NEWS.md
+
+ path = file.path(tempdir(),"cars")
as.disk.frame(cars, outdir=path, overwrite = TRUE, nchunks = 2)
-#> path: "C:\Users\RTX2080\AppData\Local\Temp\RtmpyknGIm/cars"
+#> path: "C:\Users\RTX2080\AppData\Local\Temp\RtmpUX5lOH/cars"
#> nchunks: 2
#> nrow (at source): 50
#> ncol (at source): 2
diff --git a/docs/reference/dplyr_verbs.html b/docs/reference/dplyr_verbs.html
index 143e9c7e..ff377e41 100644
--- a/docs/reference/dplyr_verbs.html
+++ b/docs/reference/dplyr_verbs.html
@@ -18,7 +18,7 @@
R/util.r
+ find_globals_recursively.RdFind globals in an expression by searching through the chain
+find_globals_recursively(code, envir)An expression to search for globals
The environment from which to begin the search
get_chunk(...)
# S3 method for disk.frame
-get_chunk(df, n, keep = NULL, full.names = FALSE, ...)the columns to keep
whether n is the full path to the chunks or just a relative path file name. Ignored if n is numeric
for internal use only. It's a data.frame used to help with filtering by partitions
Get the partitioning structure of a folder
+get_partition_paths(df)a disk.frame whose paths will be used to determine if it's +folder-partitioned disk.frame
evalparseglue()
Helper function to evalparse some `glue::glue` string
Find globals in an expression by searching through the chain
get_chunk_ids()
Get the chunk IDs and files names
Get the partitioning structure of a folder
overwrite_check()
Check if the outdir exists or not
Filter the dataset based on folder partitions
show_ceremony() ceremony_text() show_boilerplate() insert_ceremony()
Show the code to setup disk.frame
Turn a string of the form /partion1=val/partion2=val2 into data.frame
R/partition-filter.r
+ partition_filter.RdFilter the dataset based on folder partitions
+partition_filter(x, ...)a disk.frame
filtering conditions for filtering the disk.frame at (folder) partition level
R/get_partition.r
+ split_string_into_df.RdTurn a string of the form /partion1=val/partion2=val2 into data.frame
+split_string_into_df(path_strs)The paths in string form to break into partition format
overwrite output directory
the columns to shard by
the columns to (folder) partition by
compression ratio for fst files
splitting of chunks: "hash" for hash function or "sort" for semi-sorted chunks
for the "sort" shardby function, a dataframe with the split values.
for the "sort" shardby function, the variables to sort descending.
passed to cmap.disk.frame