mutate_all() and UTF-8 names

[krlmlr]

library(magrittr)
suppressPackageStartupMessages(library(dplyr))

data_frame(a = 1) %>%
  setNames(enc2native("\u4e2d")) %>%
  mutate_all(funs(as.character))
#> # A tibble: 1 x 2
#>   `<U+4E2D>`  `中`
#>        <dbl> <chr>
#> 1          1     1

data_frame(a = 1) %>%
  setNames("\u4e2d") %>%
  mutate_all(funs(as.character))
#> Warning: Mangling the following names: <U+4E2D> -> <U+4E2D>. Use
#> enc2native() to avoid the warning.
#> Error in mutate_impl(.data, dots): Evaluation error: variable '<U+4E2D>' not found.

(I did manually change the output for the first example to include the Chinese character.)

In the first example, mutate_all() gets a data frame with Unicode-escaped column names at input. Do we want to repair encoding in column names for each dplyr verb?

In the second example, the column name is properly UTF-8 encoded, but r-lib/rlang@ff87439 seems to get confused.

CC @lionel-.

[krlmlr]

Locale-independent reprex:

suppressPackageStartupMessages(library(dplyr))
withr::with_locale(
  c(LC_CTYPE = "C"),
  data_frame(a = 1) %>%
    setNames("\u4e2d") %>%
    mutate_all(funs(as.character))
)
#> Warning: Mangling the following names: <U+4E2D> -> <U+4E2D>. Use
#> enc2native() to avoid the warning.
#> Error in mutate_impl(.data, dots): Evaluation error: variable '中' not found.

[krlmlr]

Seems this will be easier to fix with objectables.

[krlmlr]

Also need to bump rlang dependency and reenable test (#3049).

[krlmlr]

The error looks different with current rlang, will revisit.

[krlmlr]

Postponing until after #2311, we always need to create a data mask if we're not on a UTF-8 system.

Even simpler reprex:

library(tidyverse)
withr::with_locale(
  c(LC_CTYPE = "C"),
  {
    varname <- "\u4e2d"
    data_frame(!!varname := 1) %>%
      mutate(!!paste0("new-", varname) := as.character(!!sym(varname)))
  }
)
#> Warning: Mangling the following names: <U+4E2D> -> <U+4E2D>. Use
#> enc2native() to avoid the warning.
#> Error in mutate_impl(.data, dots): Evaluation error: variable '中' not found.

Created on 2018-03-13 by the reprex package (v0.2.0).

The problem here is that the data frame has UTF-8 names, we need a proper data mask with mangled names (i.e., <U+4E2D> instead of 中). Currently we don't build the data mask for ungrouped operations, but we need to if we have foreign column names that cannot be translated.

[romainfrancois]

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
data_frame(a = 1) %>%
  setNames(enc2native("\u4e2d")) %>%
  mutate_all(funs(as.character))
#> # A tibble: 1 x 1
#>   中   
#>   <chr>
#> 1 1

data_frame(a = 1) %>%
  setNames("\u4e2d") %>%
  mutate_all(funs(as.character))
#> # A tibble: 1 x 1
#>   中   
#>   <chr>
#> 1 1

Created on 2018-03-26 by the reprex package (v0.2.0).

[romainfrancois]

🤔 ok the second example still fails for me:

suppressPackageStartupMessages(library(dplyr))
withr::with_locale(
  c(LC_CTYPE = "C"),
  data_frame(a = 1) %>%
    setNames("\u4e2d") %>%
    mutate_all(funs(as.character))
)
#> Warning: Mangling the following names: <U+4E2D> -> <U+4E2D>. Use
#> enc2native() to avoid the warning.
#> Error in mutate_impl(.data, dots): Evaluation error: variable '中' not found.

Created on 2018-03-26 by the reprex package (v0.2.0).