tokens(x, remove_separators = TRUE) does not remove all separators

[kbenoit]

Discovered in #784.

For instance:

txt <- "space tab\t newline\n non-breakingspace\u00A0 em-space\u2003 variationselector16 \uFE0F end"

(toks <- tokens(txt, remove_separators = TRUE))
# tokens from 1 document.
# Component 1 :
#  [1] "space"               "tab"                 "newline"             "non"                
#  [5] "-"                   "breakingspace"       "em"                  "-"                  
#  [9] "space"               "variationselector16" " ️"                   "end"                

tokens_wordstem(toks)
#  Error in char_wordstem(types(x), language = language) : 
#   whitespace detected: you can only stem tokenized texts 

This is solvable by removing all \\p{Z}:

tokens_remove(toks, "\\p{Z}", valuetype = "regex")
# tokens from 1 document.
# Component 1 :
#  [1] "space"               "tab"                 "newline"             "non"                
#  [5] "-"                   "breakingspace"       "em"                  "-"                  
#  [9] "space"               "variationselector16" "end"