dfm error with special characters

[jhfowler]

i am trying to turn a list of tweets into a dfm and it occasionally creates an error on special characters. here's an example:

library(quanteda)
x <- dfm(corpus("あﾞづいﾞ"))
Error in Matrix::sparseMatrix(i = index, p = cumsum(c(1, lengths(x))) - :
NA's in (i,j) are not allowed

[jhfowler]

On further inspection, it seems like the error is always caused by Hiragana. Strings that fail:
"づいﾞ"
"゛んﾞ"
"たーﾟ"

[kbenoit]

OK, definitely one for @koheiw. It's an issue with the hashing function:

> tokens(c("づいﾞ", "゛んﾞ", "たーﾟ"))
tokens from 3 documents.
Component 1 :
[1] "づ"  NA    "いﾞ"

Component 2 :
[1] "゛"  NA    "んﾞ"

Component 3 :
[1] "た"  NA    "ーﾟ"

> tokens(c("づいﾞ", "゛んﾞ", "たーﾟ"), hash = FALSE)
tokenizedTexts from 3 documents.
Component 1 :
[1] "づ"  ""    "いﾞ"

Component 2 :
[1] "゛"  ""    "んﾞ"

Component 3 :
[1] "た"  ""    "ーﾟ"

[jhfowler]

Thanks! BTW I am really loving the package. This is the first snag I've hit. j

…

On Feb 16, 2017, at 2:22 PM, Kenneth Benoit ***@***.***> wrote: OK, definitely one for @koheiw. It's an issue with the hashing function: > tokens(c("づいﾞ", "゛んﾞ", "たーﾟ")) tokens from 3 documents. Component 1 : [1] "づ" NA "いﾞ" Component 2 : [1] "゛" NA "んﾞ" Component 3 : [1] "た" NA "ーﾟ" > tokens(c("づいﾞ", "゛んﾞ", "たーﾟ"), hash = FALSE) tokenizedTexts from 3 documents. Component 1 : [1] "づ" "" "いﾞ" Component 2 : [1] "゛" "" "んﾞ" Component 3 : [1] "た" "" "ーﾟ" — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[kbenoit]

Thanks Jim! Testimonials always welcome at #461. Ken On 16 Feb 2017, at 23:42, jhfowler <notifications@github.com<mailto:notifications@github.com>> wrote: Thanks! BTW I am really loving the package. This is the first snag I've hit. j

On Feb 16, 2017, at 2:22 PM, Kenneth Benoit ***@***.******@***.***>> wrote: OK, definitely one for @koheiw. It's an issue with the hashing function: > tokens(c("づいﾞ", "゛んﾞ", "たーﾟ")) tokens from 3 documents. Component 1 : [1] "づ" NA "いﾞ" Component 2 : [1] "゛" NA "んﾞ" Component 3 : [1] "た" NA "ーﾟ" > tokens(c("づいﾞ", "゛んﾞ", "たーﾟ"), hash = FALSE) tokenizedTexts from 3 documents. Component 1 : [1] "づ" "" "いﾞ" Component 2 : [1] "゛" "" "んﾞ" Component 3 : [1] "た" "" "ーﾟ" — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub<#554 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACFMZg-FwdZKUpgMe6rhN8zRv7BlVoA8ks5rdNDbgaJpZM4MCHB_>.