diffobj doesn't ignore encodings

[hadley]

x <- c("fa\xE7ile", "fa\ue7ile")
Encoding(x) <- c("latin1", "UTF-8")
y <- rev(x)

x == y
#> [1] TRUE TRUE
identical(x, y)
#> [1] TRUE

diffobj::ses(x, y)
#> [1] "1d0" "2a2"

^{Created on 2020-03-29 by the reprex package (v0.3.0)}

[brodieG]

Thanks for reporting. Comparisons are done the string pool memory addresses, so anything that gets a new address will be considered different. We could translate to a common encoding, although that has a cost. Since diffobj claims to do diffs on the display representation of objects, at a minimum this should be documented.

[hadley]

I think you should consider always re-encode to UTF-8 (or some other matching encoding) since it is very rare for R to distinguish between the same string in different encodings (ie see identical() result above)

[brodieG]

Certainly that identical does the re-encoding is a strong argument in favor of doing the same. I'll look into it next time I update the package; if you have a pressing need for this change let me know (I guess worst case you can re-encode yourself first in the meantime).

[brodieG]

I don't think we can avoid enc2utf8 b/c there is no cheap way to distinguish between "unknown" and ASCII encoding, and if we have strings with both "latin1" and ASCII in a non-latin1/UTF-8 locale we are must assume there could be some non-ASCII in the "unknown". However cost is minimal, seemingly:

> x <- rawToChar((as.raw(160:255)), multiple=TRUE)
> xx <- do.call(paste0, expand.grid(x, x, x))
> Encoding(xx) <- "latin1"
> zz <- paste0(xx, rev(xx), xx, rev(xx))
> length(zz)
[1] 884736
> system.time(ww <- enc2utf8(zz))
   user  system elapsed 
  0.009   0.000   0.008 
>