Encoding problems on Windows caused by character -> symbol -> character roundtrip

[krlmlr]

Some of the encoding problems (e.g., grouping) seem to be caused by converting character to symbol and back. On Linux (UTF-8 locale):

> "ä" %>% Encoding
[1] "UTF-8"
> "ä" %>% as.name %>% as.character %>% Encoding
[1] "UTF-8"
> "ä" %>% iconv(to = "latin1") %>% as.name %>% as.character %>% Encoding
[1] "unknown"
> "ä" %>% iconv(to = "latin1") %>% as.name %>% as.character %>% enc2utf8
[1] "<e4>"

On Windows (latin-1 locale):

> "ä" %>% Encoding
[1] "latin1"
> "ä" %>% as.name %>% as.character %>% Encoding
[1] "latin1"
> "ä" %>% iconv(to = "UTF-8") %>% as.name %>% as.character %>% Encoding
[1] "unknown"
> "ä" %>% iconv(to = "UTF-8") %>% as.name %>% as.character %>% enc2utf8
[1] "ä"

On Windows, one test fails because of that.

So, currently we should suggest using ASCII (or at least native-encoded) column names, and not fiddle with the encoding, in particular not set it to UTF-8 on Windows.

A lot of internal dplyr logic seems to be based on the symbol type, I don't see a quick solution here.

[kevinushey]

IIUC, the over-arching issue is that the as.name => as.character roundtrip loses the encoding, and so we no longer know how to properly translate from the original encoding to the desired encoding (hence why enc2utf8 doesn't produce the desired result)?

[krlmlr]

Yes -- if it's not the native encoding, the roundtrip loses it, and there's no way to recover.

[hadley]

Maybe dplyr could have it's own symbol coercion methods that always assumed UTF-8?

[krlmlr]

We could do that, but I think the safest option is not to use symbols to represent column names -- a simple S3 class should do the same job.

[hadley]

I think symbols are the correct way to represent column names for a number of reasons.

[krlmlr]

We could settle for a combination:

> structure(as.name("col_name"), class = "encoded_symbol")
col_name
attr(,"class")
[1] "encoded_symbol"

With a suitable as.character.encoded_symbol() function.

[krlmlr]

Windows users are confined to their native encoding for column names anyway if they want to use them in expressions. These are always in the native encoding, the following doesn't work on Windows:

~成交日期
## Error: unexpected input in "~\"

For characters that can be represented in the native encoding, . %>% as.name %>% as.character works as expected and reliably maintains the declared encoding. I think we should use (and expect) the native encoding whenever we interact with language objects, and UTF-8 otherwise.

I have submitted a bug report to R's bugzilla concerning the behavior of as.name() for strings in non-native encoding.

[krlmlr]

A comment in the R source leaves little hope for an upstream fix, but we'll see.

[krlmlr]

Related: #1885, joining strings with different encodings.

[krlmlr]

But the following works on Windows:

~"成交日期"

CC @hadley

[krlmlr]

> data_frame(a = 1) %>% setNames("成交日期")
# A tibble: 1 × 1
  `<U+6210><U+4EA4><U+65E5><U+671F>`
                               <dbl>
1                                  1