Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
data.table::fread CSV logic fails with complex field #2051
I'm trying to load a 1.6GB csv file with data.table::fread. The process fails at some point complaining about a specific line:
I have checked that the offending line is ok (with csvfix and also https://csvlint.io/). I have included this line in the following example file (which contains the header, a non-failing line and then the failing line):
As you can see, it has some non-trivial quoting and escaping
Do you think the fread csv logic could be extended to be able to deal with things like this?
Thanks a lot, best regards!
Hi, there is ongoing development related to fread and quoting, it is currently planned for next release. You can read more in https://github.com/Rdatatable/data.table/wiki/Convenience-features-of-fread#10-automatic-quote-escape-method-detection-including-no-escape Though I'm not sure if the change is going to cover your use case.