Suggestion - enable option to save tables as parquet rather than csv

[iainmwallace]

Hi,

Based on this recent blog, https://ursalabs.org/blog/2019-10-columnar-perf/ it would be beneficial from a performance perspective to enable an option to store tables in parquet format instead of csv

arrow::write_parquet(iris,"iris.parquet")
arrow::read_parquet("iris.parquet")

Thanks

Iain

[javierluraschi]

This PR should ease the performance issue a little bit:

#74

It basically allows you to pin something with I() to avoid additional files form being crated.

The suggestion to use parquet is terrific, we actually took a look at using arrow or feather a couple months ago. However, we need a way to preview feather and arrow datasets in the browser which, at the time we tried this out, had restrictions on the supported data types javascript could handle.

That said, this is something worth considering in the future and also allowing users to customize how data frames are persisted in RStudio Connect. We'll follow up with another PR to customize how support files are exported in pins.

[javierluraschi]

Yeah, this is a must do at some point; however, I think it needs to be opt-it. The challenge is that CSVs are universal while Parquet is not... one might still want to open a pin in Excel or Google Sheets and I'm not sure when that will be supported, if ever.

I think the approach I would suggest here would be something like:

board_register(..., format = "parquet")

Which would opt-in to store everything as parquet and loose interoperability with CSVs, which some users would really appreciate since some are creating pins with pin(I(data), board = "") to force R to use RDS serialization and avoid the overhead of using CSVs. Users in this situation would be better served, in some cases where performance is not the top-priority, to save in Parquet instead of RDS.

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.