You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
read_delim() fails when sample of parsing problems contains non-ASCII characters #1136
When a non-ASCII file is read using read_delim() or its friends and there are parsing failures and the text used by warn_problems() contains non-ASCII characters, the following error is returned:
## Error in nchar(x) : invalid multibyte string, element 3
This is quite hard to debug because the error makes it seem like the encoding was set incorrectly or the file has problems. (In my case it only occurred with 2 out of about 6000 CSV files read using map_dfr() even though all contained non-ASCII text.)
where nchar() throws an error if one of the problems "sampled" for display contains a non-UTF-8 string. The data is in fact read correctly inside read_delimited(); the call only fails once warn_problems() is called.
Once I supplied the column specification and the non-ASCII text no longer appeared in the list of reported problems, the error no longer occurred (see reprex).
Possibly related to #1111 where the same error is thrown by parse_number(); this may have a similar root cause though I haven't investigated specifically.
Reprex:
tf<- tempfile()
download.file("https://vdp.cuzk.cz/vymenny_format/csv/20200930_OB_547999_ADR.csv.zip", destfile=tf)
read_csv2(tf, locale= locale(encoding="Windows-1250"))
## Error in nchar(x) : invalid multibyte string, element 3
Inspecting this in the debugger, the probs_f variable used inside warn_problems() does indeed contain non-ASCII text.
The file referenced in the reprex above was correctly read using read.csv2 also using the "WINDOWS-1250" encoding. Likewise it can be read in python using the pandas CSV reader with the same encoding.
Activity
hidekoji commentedon Nov 16, 2020
I have a similar issue with Shift JIS encoding.
Created on 2020-11-16 by the reprex package (v0.3.0)
This works if I set FALSE for the
col_namesargument.Created on 2020-11-16 by the reprex package (v0.3.0)