Skip to content

Commit

Permalink
fread respects encoding on header cols as well, closes #1680.
Browse files Browse the repository at this point in the history
  • Loading branch information
arunsrinivasan committed Apr 27, 2016
1 parent d6f7959 commit f91bba1
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 1 deletion.
2 changes: 2 additions & 0 deletions NEWS.md
Expand Up @@ -191,6 +191,8 @@

54. `chmatch()` handles `nomatch = integer(0)` properly, [#1672](https://github.com/Rdatatable/data.table/issues/1672).

55. `fread()` did not respect encoding on header column. Now fixed, [#1680](https://github.com/Rdatatable/data.table/issues/1680). Thanks @nachti.

#### NOTES

1. Updated error message on invalid joins to reflect the new `on=` syntax, [#1368](https://github.com/Rdatatable/data.table/issues/1368). Thanks @MichaelChirico.
Expand Down
6 changes: 6 additions & 0 deletions inst/tests/tests.Rraw
Expand Up @@ -8824,6 +8824,12 @@ x = data.table(Date = as.Date(c("2015-12-29", "2015-12-29", "2015-12-29", "2015-
y = CJ(Date = as.Date(c("2015-12-31", "2016-01-31", "2016-02-29", "2016-03-31")), ID = unique(x$ID))
test(1669, x[y, on=c("ID", "Date"), roll=TRUE, which=TRUE], 1:16)

# 1680 fix, fread header encoding issue
x = "Stra\xdfe"
Encoding(x) = "latin1"
nm = names(fread("1680-fread-header-encoding.csv", encoding="Latin-1"))
test(1670, nm[2], x)

##########################

# TODO: Tests involving GForce functions needs to be run with optimisation level 1 and 2, so that both functions are tested all the time.
Expand Down
2 changes: 1 addition & 1 deletion src/fread.c
Expand Up @@ -929,7 +929,7 @@ SEXP readfile(SEXP input, SEXP separg, SEXP nrowsarg, SEXP headerarg, SEXP nastr
// Also Fiedl() it takes care of leading spaces. Easier to understand the logic.
skip_spaces(); Field();
if (fieldLen) {
SET_STRING_ELT(names, i, mkCharLen(fieldStart, fieldLen));
SET_STRING_ELT(names, i, mkCharLenCE(fieldStart, fieldLen, ienc)); // #1680 fix, respect encoding on header col
} else {
sprintf(buff,"V%d",i+1);
SET_STRING_ELT(names, i, mkChar(buff));
Expand Down

0 comments on commit f91bba1

Please sign in to comment.