Dealing with horrible files

[Ironholds]

Some humans - some terrible, terrible humans - leave tabs in content-insertable fields. Tabs that are not consistently escaped. Would it be possible to add an option to read_delim that, if set to FALSE, continues current behaviour around one row unexpectedly having an extra field and, if set to TRUE, mashes the extra field into the last "expected" field and issues a warning noting the row number?

[hadley]

I think so. Could you outline a few examples? (maybe using commas so it's a bit easier to see)

[Ironholds]

Totally!

h1 h2 h3
1, 6, A RUN-ON SENTENCE (sometimes called a "fused sentence")
1, 9, has at least two parts, either one of which can stand by itself
5, 4, (in other words, two independent clauses)

We've got an unquoted text field, which shouldn't be a thing that exists but is, because humans are terrible. As a result, any comma is (potentially) a delimiter; it's impossible for a machine to tell.

In R-core, the response is to create an additional field at the end and then (confusingly) warn you about incomplete lines. In readr, the response is to say "more columns than column names, abort!". It would be nice to have a way of solving for this by saying "I don't care if you've got a comma in you, field, I've been told there are six fields and you're no.7 so I'm going to issue a warning and then fields[6] += fields[7]. In you go now."

[hadley]

Hmmm, this is the same basic behaviour as tidyr::separate() so maybe have the same option? i.e. extra = c("error", "drop", "merge")

[hadley]

Maybe warn (= warning + drop) would be the best default

[Ironholds]

Yep; sounds perfect!

[hadley]

Related to #189

[hadley]

I think this is ok now - readr will just expand the columns as needed, and you can do the cleanup afterwards. (i.e. it's like extra = "warn" with no option to change the behaviour). The code is already getting pretty complicated, so I don't want to mess with it at the moment.