Minimise peak memory usage when doing copy_to from one database to another

[mungojam]

The copy_to now supports a src_sql source as well as destination, but currently does this via a tibble. This makes it performant only for datasets that fit into available RAM.

It would be good if some sort of streaming or batching were implemented so that the peak RAM usage is much smaller. This is so that we can use dplyr to do an initial large extract from a large remote table into a local SQLite database (for example).

Perhaps this fits better as a DBI feature but it's nice to be able to do a custom select on the source table using dplyr.

library(dplyr, warn.conflicts = FALSE)

src1 <- src_sqlite("fileDb1.db", create = TRUE)
src2 <- src_sqlite("fileDb2.db", create = TRUE)

# Create from dataframe
iris_1 <- copy_to(src1, iris, "iris1")

# Create from tbl in another data, currently goes via in-memory tibble containing full dataset
db2 <- copy_to(src2, iris, 'data2', temporary = FALSE)

[ghost]

This issue was moved by hadley to tidyverse/dbplyr/issues/96.

[lock]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/