Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ListCallback to complement SideEffect and DataFrame callbacks #568

Closed
mmuurr opened this issue Dec 29, 2016 · 2 comments
Closed

ListCallback to complement SideEffect and DataFrame callbacks #568

mmuurr opened this issue Dec 29, 2016 · 2 comments
Labels
feature a feature request or enhancement read 📖

Comments

@mmuurr
Copy link

mmuurr commented Dec 29, 2016

Related to (PR) #520.

There are times where it's useful for the chunked callback to return a list (that is combined into a single list at the end).
An example is when processing a large JSONL file with purrr functions: (1) read file chunk, (2) use purrr to transform, filter, reduce, etc. as needed, and (3) combine into a final list, call that final object my_data.
My specific use-case is to filter the data to discard irrelevant data such that the final object can fit into memory for interactive work/exploration.

When downstream functions expect my_data to be a data frame, then DataFrameCallback works just fine.

But if there already exist some downstream functions written to deal with my_data as a list (e.g. prepared purrr-enabled pipelines), the DataFrameCallback must be written to return a single list-column data frame, then extract that single list-column, and finally pass the now-extracted list to the downstream functions.

It seems silly to return a single-column data frame, just to then extract that single column later, when we know we'd like have a list in-hand at the end of the chunked reading/processing phase.

Minor suggestion, but certainly not a blocker/bug/etc. (since the work-around is described above).
In all other aspects, readr's great!

@hadley hadley added the feature a feature request or enhancement label Jan 25, 2017
@hadley
Copy link
Member

hadley commented Jan 25, 2017

This seems like a good idea

@jimhester
Copy link
Collaborator

Closed by #601

@lock lock bot locked and limited conversation to collaborators Sep 24, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature a feature request or enhancement read 📖
Projects
None yet
Development

No branches or pull requests

3 participants