-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ARROW-8376: [R] Add experimental interface to ScanTask/RecordBatch it…
…erators As an alternative to calling `ToTable()` to bring everything into memory, it would be nice to expose the stream of batches so that you could aggregate (or really do whatever) on each chunk. That gives access to the full dataset, which otherwise you can't handle unless it's small. On the NYC taxi dataset (10.5 years, 125 parquet files), ```r tab <- ds %>% select(passenger_count) %>% map_batches(~count(., passenger_count)) %>% group_by(passenger_count) %>% summarize(n = sum(n)) ``` gives me the tabulation of `passenger_count` in about 200s (no parallelization). And you can see all sorts of weird features in the data: ``` > as.data.frame(tab) passenger_count n 1 -127 7 2 -123 1 3 -122 1 4 -119 1 5 -115 1 6 -101 1 7 -98 1 8 -96 1 9 -93 1 10 -92 1 11 -91 1 12 -79 1 13 -64 2 14 -63 1 15 -48 1508 16 -45 1 17 -43 4 18 -33 1 19 -31 1 20 -9 1 21 -7 1 22 -6 3 23 -2 1 24 -1 10 25 0 5809809 26 1 1078624900 27 2 227454966 28 3 67096194 29 4 32443710 30 5 9906444 31 6 37241244 32 7 1753 33 8 1437 34 9 1304 35 10 17 36 13 1 37 15 2 38 17 1 39 19 1 40 25 1 41 33 2 42 34 1 43 36 1 44 37 1 45 38 1 46 47 1 47 49 26 48 53 1 49 58 2 50 61 1 51 65 3 52 66 1 53 69 1 54 70 1 55 84 1 56 96 1 57 97 1 58 113 1 59 125 1 ``` Closes #6365 from nealrichardson/r-map-reduce Authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: François Saint-Jacques <fsaintjacques@gmail.com>
- Loading branch information
1 parent
b7044a1
commit 9662dd6
Showing
11 changed files
with
261 additions
and
32 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.