tokens(x) stumbles on interior punctuation

[kbenoit]

Punctuation in the middle of a word should be preserved, but it is not.

txt <- "holy s%$t, quanteda is a f***ing great"

tokens(txt, remove_punct = FALSE)
## tokens from 1 document.
## text1 :
##  [1] "holy"     "s"        "%"        "$"        "t"        ","        "quanteda"
##  [8] "is"       "a"        "f"        "*"        "*"        "*"        "ing"     
## [15] "great"   
tokens(txt, remove_punct = TRUE)
## tokens from 1 document.
## text1 :
## [1] "holy"     "s"        "t"        "quanteda" "is"       "a"        "f"       
## [8] "ing"      "great" 

Changing the boundary method helps but still cannot distinguish the end of word punctuation:

txt <- "holy s%$t, quanteda, is a f***ing great"
tokens(txt, what = "fasterword", remove_punct = TRUE)
## tokens from 1 document.
## text1 :
## [1] "holy"      "s%$t,"     "quanteda," "is"        "a"         "f***ing"  
## [7] "great"   

True for Twitter characters too. The first is correct, but when we try to remove Twitter characters, the problem arises:

txt <- "I'm @kenbenoit on Twitter, #rstats don't u B a h@ter!"
tokens(txt, remove_twitter = FALSE, remove_punct = TRUE)
## tokens from 1 document.
## text1 :
## [1] "I'm"        "@kenbenoit" "on"         "Twitter"    "#rstats"    "don't"     
## [7] "u"          "B"          "a"          "h@ter"

tokens(txt, remove_twitter = TRUE, remove_punct = TRUE)
tokens from 1 document.
text1 :
## [1] "I'm"       "kenbenoit" "on"        "Twitter"   "rstats"    "don't"     "u"        
## [8] "B"         "a"         "h"         "ter"

[koheiw]

We could use stringi's RBBI to preserve internal punctuations as mentioned in #896

[kbenoit]

Excellent idea, RBBI is in the current CRAN version of stringi now (v1.2.4).