read_csv coerces all column values to NA if 100 first observations are missing

[vincentarelbundock]

If the first 100 observations of a variable are missing, read_csv overwrites all column values with NAs (unless the column is boolean). Ideally, column type should be determined using the first 100 non-missing values of each column.

library(readr)
x = data.frame(matrix(rnorm(1000), ncol=5))
x$X1[1:100] = NA
write.csv(x, file='test.csv', row.names=FALSE)
y = read_csv('test.csv')
y$X1
problems(x)

[vincentarelbundock]

FWIW, this actually happens a lot with the type of data I tend to work with. It is also not practical to manually specify column types in my particular application, since hundreds of columns raised a warning.

[hadley]

It's not possible to look for the first 100 non-missing values, because that doesn't take a bounded amount of time to run - it might have to scan the whole file.

[vincentarelbundock]

I had not thought of that. I guess I'll just keep using read.csv for now. Thanks for your work.

[hadley]

Oh hmmmm, I think the reason that this is so painful is that I have a bug in my logic somewhere - if the first 100 values are all missing, it should guess that the column is character, since that ensures you don't lose info

[vincentarelbundock]

Makes sense. Also, you probably don't want to see a proliferation of arguments, but since the 100 number is arbitrary, it might be useful to allow users to specify how many rows the function checks. For example, I'd be willing to waste a few cpu cycles to check 1000 lines and get good type inference.

[artemklevtsov]

Seems duplicated #124. Also readxl have the same bug.

[hadley]

The major annoyingness of this behaviour should be fixed - now all contents will be loaded without errors into a character vector. I'll continue to explore better heuristics for guessing column type.