korean encoding issue

[mrchypark]

#9

new start.

[jwijffels]

I don't know of a clear solution if you really want to incorporate the code your provided inside the function. But the docs in udpipe_annotate are pretty clear on the input needed. It says that x should be a character vector in UTF-8 encoding
If you do not have x in UTF-8 encoding then you need to make sure x is in UTF-8 encoding. This can be done with iconv
As in iconv(x, from = "CP949", to = "UTF-8")
Where the list of encodings is specified in iconvlist for example

Encoding("Je n'aime pas ça")
[1] "latin1"
Encoding(iconv("Je n'aime pas ça", from = "latin1", to = "UTF-8"))
[1] "UTF-8"

But the default encoding if you type in text in R depends on your locale, mine is as follows.

Sys.getlocale()
[1] "LC_COLLATE=Dutch_Netherlands.1252;LC_CTYPE=Dutch_Netherlands.1252;LC_MONETARY=Dutch_Netherlands.1252;LC_NUMERIC=C;LC_TIME=Dutch_Netherlands.1252"
localeToCharset()
[1] "ISO8859-1"

So for my case I would need to do

library(udpipe)
ud_model <- udpipe_download_model("french")
ud_model <- udpipe_load_model(ud_model$file_model)
x <- udpipe_annotate(ud_model, x = iconv("Je n'aime pas ça", from = "latin1", to = "UTF-8"))
as.data.frame(x)

Testing inside udpipe_annotate if x is in UTF-8 has the following complexities shown below:

• ASCII code has always unknown encoding - so unknown can be an indication of ascii which is fine
• You are never sure that text data on your machine is in the default locale of your machine as shown below

As a result, I'm reluctant to do any fixes inside the udpipe_annotate function. The user just needs to make sure his input is in UTF-8

> ## ASCII is always Encoding unknown
> Sys.setlocale("LC_ALL", locale = "Dutch_Netherlands.1252")
[1] "LC_COLLATE=Dutch_Netherlands.1252;LC_CTYPE=Dutch_Netherlands.1252;LC_MONETARY=Dutch_Netherlands.1252;LC_NUMERIC=C;LC_TIME=Dutch_Netherlands.1252"
> x <- "I drink milk in the morning"
> Encoding(x)
[1] "unknown"
> Encoding(iconv(x, to = "UTF-8"))
[1] "unknown"
> Sys.setlocale("LC_ALL", locale = "Dutch_Netherlands.1252")
[1] "LC_COLLATE=Dutch_Netherlands.1252;LC_CTYPE=Dutch_Netherlands.1252;LC_MONETARY=Dutch_Netherlands.1252;LC_NUMERIC=C;LC_TIME=Dutch_Netherlands.1252"
> x <- "안녕하세요. 저는 박찬엽입니다. 한글의 인코딩 문제를 재현하려고 합니다."
> Encoding(x)
[1] "UTF-8"
> out <- iconv(x, from = "UTF-8", to = "CP949")
> iconv(out, from = "CP949", to = "UTF-8")
[1] "안녕하세요. 저는 박찬엽입니다. 한글의 인코딩 문제를 재현하려고 합니다."
> result <- iconv(out, to = "UTF-8")
> Sys.setlocale("LC_ALL", locale = "Korean_Korea.949")
[1] "LC_COLLATE=Korean_Korea.949;LC_CTYPE=Korean_Korea.949;LC_MONETARY=Korean_Korea.949;LC_NUMERIC=C;LC_TIME=Korean_Korea.949"
> result
[1] "¾È³çÇϼ¼¿ä. Àú´Â ¹ÚÂù¿±ÀÔ´Ï´Ù. ÇѱÛÀÇ ÀÎÄÚµù ¹®Á¦¸¦ ÀçÇöÇÏ·Á°í ÇÕ´Ï´Ù."

[dselivanov]

Absolutely agree with @jwijffels

[mrchypark]

I agree with The user just needs to make sure his input is in UTF-8.

how about add warning message if Encoding(x)!= "UTF-8"?

[dselivanov]

As mentioned above ascii will have unknown encoding which is != Utf-8. 19 янв. 2018 г. 13:39 пользователь "Chan-Yub Park" <notifications@github.com> написал:

…

I agree with The user just needs to make sure his input is in UTF-8. how about add warning message if Encoding(x)!= "UTF-8"? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE4u3WNMmpuyh5ZweYTwAN0o5N1jt294ks5tMGLSgaJpZM4RjxYj> .

[dselivanov]

I mean it would be nice to have such check/warning but it seems it will be tricky to implement it.

[mrchypark]

Ok, just my opinion. Thank you guys for support and discuss. Is it ok to close issue?

[jwijffels]

Completely agree that it would be nice to have such a check/warning but due to the 2 elements I just enumerated, I don't know of any valid way on how to implement this.

[jwijffels]

Closing this. Feel free to re-open if needed.