spread causes system to run out of memory

[jtlowell]

How long/wide can a data frame be -- going from gathered to spread form?

I have a 200,000 row data frame I'm spreading to create 200,000 columns and I'm running out of memory.

Have you tested to see limits on the operations with various machines?

[hadley]

I have not. It might be possible to replace the vectorised R code with optimised C++ code that would need less memory.

[JamesOwers]

Just a quick note that I'm having memory issues with spread(..., drop=FALSE). If I use spread(..., drop=TRUE) then everything works out fine, the process takes just a few seconds, and the result is of size 0.2Mb.

My input dataset is 0.4MB, has 6000 rows, and 11 variables. This is the result of a filter on a dataset which is of size 200Mb. When running with spread(..., drop=FALSE), the rsession memory expands to over 20Gb.

Unfortunately I can't provide the exact dataset, but if there is anything I can provide to help, I'll be happy to do so.

[hadley]

How many unique values are there in the variables that you are spreading? It is easy to create very very large data frames with spread.

[JamesOwers]

There are some numeric variables with a few thousand unique values, but isn't spread just going to make a variable for each key? Also, by virtue of spread(..., drop=TRUE) working fine, the only variables remaining to spread only have one value: NA.