removeHyphens = FALSE not working in tokenise for some multiple intra-word hyphens

[kbenoit]

> tokenize("one-of-a-kind", removeHyphens = FALSE, removePunct = TRUE)
tokenizedText object from 1 document.
Component 1 :
[1] "one-of-a" "kind"    

> tokenize("one-of-another-kind", removeHyphens = FALSE, removePunct = TRUE)
tokenizedText object from 1 document.
Component 1 :
[1] "one-of-another-kind"

[tpaskhalis]

My impression is that it has something to do with the stringi package. Specifically, the way it handles single-character words. I find this behaviour rather perplexing:

> stringi::stri_replace_all_regex("a-a-a-a", "(\\w)[\\p{Pd}](\\w)", "$1_hy_$2")
[1] "a_hy_a-a_hy_a"
> stringi::stri_replace_all_regex("a-aa-aa-a", "(\\w)[\\p{Pd}](\\w)", "$1_hy_$2")
[1] "a_hy_aa_hy_aa_hy_a"
> stringi::stri_replace_all_regex("a-aa-a-aa", "(\\w)[\\p{Pd}](\\w)", "$1_hy_$2")
[1] "a_hy_aa_hy_a-aa"
> stringi::stri_replace_all_regex("aa-aa-aa-aa", "(\\w)[\\p{Pd}](\\w)", "$1_hy_$2")
[1] "aa_hy_aa_hy_aa_hy_aa"

[kbenoit]

See http://stackoverflow.com/questions/35072153/how-to-substitute-intra-word-hyphens-when-hyphens-surround-a-single-inner-charac?noredirect=1#comment57867440_35072153

Changing the \w to \b seems to work.