Don't error on empty files -- warn & skip

[MichaelChirico]

I'm trying to read a file that was output from parquet to csv(s) from Spark.

In its infinite wisdom, Spark created some empty files. So this work flow failed:

read_f = list.files('path/to/csvs', pattern = 'csv$', full.names = TRUE)
DT = rbindlist(lapply(read_f, fread))

It's kind of a pain to have to single out empty files (basically add the line read_f = read_f[file.info(read_f)$size > 0]) when the vast majority of the time this operation works as intended (since it's rare for spark to output empty files) -- is there any reason fread can't just warn for such a file and skip?

[jangorecki]

+1, also please link issue created (found) in spark jira or whatever they use so we can have link in documentation related to that change.

[MichaelChirico]

Not sure I follow -- are you suggesting I file an issue with them? This didn't come from any particular issue.

As I understand it, the root cause here is I'm reading from a parquet directory that has a bunch of empty constituent files. So when I coalesce there's still a partition with no actual data in it. Not sure how to overcome this, so I'm stuck with empty files for now. (that is to say, I don't think this is a bug on Spark's part, per se)

[jangorecki]

even if it is not a bug but just limitation in spark, it is good to have reference to it.