Strings with and without encoding are not matched when joining

[krlmlr]

encoded <- function(x) {
  Encoding(x) <- "UTF-8"
  x
}

unencoded <- function(x) {
  Encoding(x) <- "unknown"
  x
}

dplyr::left_join(tibble::data_frame(a = encoded("ä")),
                 tibble::data_frame(a = unencoded("ä"), b = 1))
##       a     b
##   <chr> <dbl>
## 1     ä    NA

Expected: b == 1 in the result.

This can be mitigated by using r_match() instead of match() in the JoinStringStringVisitor, but I wonder if dplyr should warn instead.

[hadley]

Maybe tibble should force encoding to utf-8? Dplyr would still need to warn but that would mitigate some of the hassle

[krlmlr]

In my case, data came from a CSV file read using read.csv(); but readr already takes care of the encoding. If this done by tibble, perhaps readr doesn't have to do it anymore.

match() is Rcpp::match(), and Rcpp seems to respect the declared encoding (RcppCore/Rcpp#189, RcppCore/Rcpp#466). To me, r_match() looks like a safe, if perhaps slower, alternative. Or we fix upstream.

[hadley]

Would be good to fix upstream

[krlmlr]

Doesn't look like an upstream fix will become available soon. We should just make sure that column data is always UTF-8.

[krlmlr]

Related: #1950, column names with non-native encoding.