tokens(x, what = "fasterword") bug? not splitting on \n \t \r

[kbenoit]

Because of how we split for what = "faster word", we are running into this problem:

> tokens("one\ntwo\tthree", what = "fasterword")
tokens from 1 document.
text1 :
[1] "one\ntwo\tthree"

Those should be split into three tokens.

This behaviour seems to come from stringi:

> stringi::stri_split_regex("one\ntwo\tthree", "\\p{Z}+")
[[1]]
[1] "one\ntwo\tthree"

> stringi::stri_split_regex("one\ntwo\tthree", "\\p{WHITE_SPACE}+")
[[1]]
[1] "one"   "two"   "three"

Because the Z unicode category should match for the \p and \n, I filed an issue for this at gagolews/stringi#327.