[R] Implementing tidyr interface

[asfimport]

I think it would be reasonable to implement an interface to the tidyr package. The implementation would allow to lazily process ArrowTables before put it back into the memory. However, currently you need to collect the table first before applying tidyr methods. The following code chunk shows an example routine:

library(magrittr)
arrow_table <- arrow::read_feather("table.feather", as_data_frame = FALSE) 
nested_df <-
   arrow_table %>%
   dplyr::select(ID, 4:7, Value) %>%
   dplyr::filter(Value >= 5) %>%
   dplyr::group_by(ID) %>%
   dplyr::collect() %>%
   tidyr::nest()

The main focus might be the following three methods:

• tidyr::[un]nest(),

• tidyr::pivot_[longer|wider](), and

• tidyr::seperate().

  I suppose the last two can be fairly quickly implemented, but tidyr::nest() and tidyr::unnest() cannot be implement before conversion to List will be accessible.

Reporter: Dominic Dennenmoser

_{Note: This issue was originally created as ARROW-8813. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
If you wanted to explore this, one challenge I see is that pivot_longer and pivot_wider aren't generics, so you can't just make arrow methods for them.

[asfimport]

Dominic Dennenmoser:
Thanks for refering to that. I've just looked for issues or pull-requests mention anything in that direction. Fortunately, a generic version of pivot_[longer|wider]() will be available in the upcoming version of tidyr, and is already implemented into the development version (#800).

[asfimport]

Nigel McKernan:
The issue [~domiden] references was committed into tidyr  1.1.0 back in May of 2020, as you can see here, more than 2 years ago.

 

Would it be possible now to incorporate some tidyr methods that have been converted to generics into {}arrow{}?

EDIT: As well, the nest() generic is now lazily-evaluated, making it easier to do remote operations, as of the tidyr 1.2.0 release earlier this year.

[eitsupi]

Related to #34265