Re-encode character columns and column names to UTF-8

[krlmlr]

tidyverse/dplyr#1885 (comment)

During construction and coercion. If possible, also during assignment.

[hadley]

Also need to provide a function to alert people to this problem, and to fix it themselves.

Might be worth considering detecting the problem on print and warning the user. OTOH that's a rather expensive check because you have spider the complete list (including all attributes). We should at least have a way for the user to call this themselves.

[hadley]

We could definitely have a warning if the column names are not all UTF-8 - that's one of the most common causes of problems.

[krlmlr]

Not sure, see tidyverse/dplyr#1950.

[hadley]

I think the most important thing is to check the encoding of the column names. I think this means:

• always encode as UTF-8 on creation
• provide a helper method to re-encode if necessary
• on print, warn about non-UTF8 names

[krlmlr]

I'd like to release utf8 before we can proceed here, because there we have efficient methods for checking the encoding of a character value.

[krlmlr]

Apart from this and #151, I think the package is ready for another CRAN release.

To recap: Column names should be in the native encoding (with the ugly <U+xxxx> formatting if necessary), because all language objects are natively encoded anyway. Column data should be in UTF-8.

How about making sure that column headers are in native encoding first, and postponing the encoding of the column data until later?

[hadley]

Yeah, that sounds good. But I think we can leave to the next release? I need to think through the tradeoffs.

[krlmlr]

#150 -> #151, but releasing now is probably safer anyway.

[krlmlr]

A comment from @JanSchulz (#104 (comment)):

regarding windows <U+....> printing: r-lib/evaluate#66 This problem seems to happen quite deep in R. We had quite a lot of fun in IRkernel (or better in repr) with that because we use sink (or better evaluate does) to get the output of a computation: https://github.com/IRkernel/repr/blob/master/R/repr_matrix_df.r#L16-L27

[krlmlr]

A way to find a locale with latin-1 encoding:

system2("locale", "-a", stdout = TRUE) %>% grep("8859", ., value = TRUE)

Works on my Ubuntu and OS X machines. This can then be used for LC_CTYPE:

withr::with_locale(
  c(LC_CTYPE = system2("locale", "-a", stdout = TRUE) %>% grep(8859, ., value = TRUE) %>% .[[1]]),
  Encoding(enc2native("ä")))
## [1] "latin1"

[hadley]

Another useful locale from Winston: https://gist.github.com/wch/3a629cfe575846a14207

[krlmlr]

Fixing encoding here (i.e. encoding as native) should be responsibility of tidy_names() too. Need to double-check if this should be done for all tibbles.

[krlmlr]

We now have:

library(magrittr)

loc_info <- c(LC_CTYPE = system2("locale", "-a", stdout = TRUE) %>% grep(8859, ., value = TRUE) %>% .[[1]])
loc_info
#>         LC_CTYPE 
#> "en_US.iso88591"

alien <- "\u5e78\u798f"
alien
#> [1] "幸福"

withr::with_locale(
  loc_info,
  {
    print(alien)
    print(make.names(alien))
    print(tibble:::make_syntactic(alien))
  }
)
#> [1] "<U+5E78><U+798F>"
#> [1] "X.U.5E78..U.798F."
#> [1] ".U.5E78..U.798F."

^{Created on 2018-10-27 by the reprex package (v0.2.1.9000)}

This means that "universal" names won't have any problems here, and I'm not sure if the transliteration should be part of a less restrictive strategy.

The only remaining question here is if we should attempt to make these names look nicer, e.g. ".." in the example above. CC @jennybc.

[krlmlr]

Name repair is in vctrs now and seems to do well with different encodings.

[github-actions]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue and link to this old issue if necessary.