New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ListCallback to complement SideEffect and DataFrame callbacks #568

Closed
mmuurr opened this Issue Dec 29, 2016 · 2 comments

Comments

3 participants
@mmuurr

mmuurr commented Dec 29, 2016

Related to (PR) #520.

There are times where it's useful for the chunked callback to return a list (that is combined into a single list at the end).
An example is when processing a large JSONL file with purrr functions: (1) read file chunk, (2) use purrr to transform, filter, reduce, etc. as needed, and (3) combine into a final list, call that final object my_data.
My specific use-case is to filter the data to discard irrelevant data such that the final object can fit into memory for interactive work/exploration.

When downstream functions expect my_data to be a data frame, then DataFrameCallback works just fine.

But if there already exist some downstream functions written to deal with my_data as a list (e.g. prepared purrr-enabled pipelines), the DataFrameCallback must be written to return a single list-column data frame, then extract that single list-column, and finally pass the now-extracted list to the downstream functions.

It seems silly to return a single-column data frame, just to then extract that single column later, when we know we'd like have a list in-hand at the end of the chunked reading/processing phase.

Minor suggestion, but certainly not a blocker/bug/etc. (since the work-around is described above).
In all other aspects, readr's great!

@hadley hadley added the feature label Jan 25, 2017

@hadley

This comment has been minimized.

Member

hadley commented Jan 25, 2017

This seems like a good idea

@hadley hadley added the read label Jan 25, 2017

@jimhester jimhester added this to DONE in jimhester Feb 7, 2017

@jimhester

This comment has been minimized.

Member

jimhester commented Feb 10, 2017

Closed by #601

@jimhester jimhester closed this Feb 10, 2017

@lock lock bot locked and limited conversation to collaborators Sep 24, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.