Functions not handling accented chars properly

[ChrisMuir]

Testing this on a Mac and a PC and getting different results.

library(refinr)
vect <- c("César Moreira Nuñez", "cesar moreira nunez")

On the PC:

key_collision_merge(vect)
#> "César Moreira Nuñez" "César Moreira Nuñez" # This is the correct output
n_gram_merge(vect)
#> "César Moreira Nuñez" "cesar moreira nunez"

On the Mac:

key_collision_merge(vect)
#> "César Moreira Nuñez" "cesar moreira nunez"
n_gram_merge(vect)
#> "César Moreira Nuñez" "cesar moreira nunez"

The expected output for all four functions above is c("César Moreira Nuñez", "César Moreira Nuñez").

This issue is possibly related to issue #58 from the rOpenSci pkg tokenizers (and the reprex above was stolen from that issue).

Both the Mac and PC are running R v3.4.4, and here's the local and encoding setting for each:

PC:

Sys.getlocale()
#> "LC_COLLATE=English_United States.1252;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252"

getOption("encoding")
#> "native.enc"

Mac:

Sys.getlocale()
#> en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

getOption("encoding")
#> native.enc

[ChrisMuir]

On the PC, the issue with function n_gram_merge() seems to be fixed simply by rearranging some of the operations within function get_fingerprint_ngram() and changing when the input strings are run through function iconv():

Current:

...<snip code>...
vect <- strsplit(vect, "", fixed = TRUE)
vect <- cpp_get_char_ngrams(vect, numgram = numgram)
vect <- iconv(vect, to = "ASCII//TRANSLIT")

Updated:

...<snip code>...
vect <- iconv(vect, to = "ASCII//TRANSLIT")
vect <- strsplit(vect, "", fixed = TRUE)
vect <- cpp_get_char_ngrams(vect, numgram = numgram)

[ChrisMuir]

The purpose of using iconv() within both get_fingerprint_ functions is to strip all char accent marks. The issue on the Mac is that iconv() (with to = "ASCII//TRANSLIT) is not achieving this:

vect <- c("César Moreira Nuñez", "cesar moreira nunez")
iconv(vect, to = "ASCII//TRANSLIT")
#> "C'esar Moreira Nu~nez" "cesar moreira nunez"

I could use stringi::stri_trans_general() in place of iconv(), but would rather not depend on stringi just for one function.
Here's an example, from the Mac:

stringi::stri_trans_general(vect, "Latin-ASCII")
#> "Cesar Moreira Nunez" "cesar moreira nunez"

[ChrisMuir]

Fixed in commit 3c0625b.