error with tokens() in v4.0

[davidemorselli]

I get the following error when tokenizing

my command is token <- tokens(threads$x, what="word",verbose = T, remove_separators = F)

threads$x is character vector in a data.table

the error is

Creating a tokens from a character object...
...starting tokenization
...tokenizing 1 of 1 blocks
...preserving hyphens
...preserving elisions
...preserving social media tags (#, @)
Error in stri_split_boundaries(x, type = rule) :
Illegal argument. (U_ILLEGAL_ARGUMENT_ERROR)

[koheiw]

If you are using Linux, it is likely that that your ICU library is very old. Try to install stringi with a newer library:

https://stringi.gagolewski.com/install.html#customising-the-build-process