Error when using the "phrasetotoken" function with collocations (bi- AND trigrams)

[atrolle]

Hi,

I'm using tm and quanteda packages conjointly.
But I'm facing an error message when using the "phrasetotoken" function with collocations (bi- and trigrams). Here is my code :

Text <- readLines("Text.txt", encoding = "latin1")

CollocsText <- collocations(Text, size = 2:3, method = "lr")
CollocsText <- removeFeatures(CollocsText, stopwords("french"))


vsText <- VectorSource(Text)
corpusText <- Corpus(vsText)
corpusText <- tm_map(corpusText, content_transformer(tolower))
corpusText <- tm_map(corpusText, removeWords, stopwords("french"))
corpusText <- tm_map(corpusText, removePunctuation)

TokensText <- phrasetotoken(corpus(corpusText),CollocsText)

When executing the last command, the error message is as follows :
Error in stringi::stri_replace_all_regex(object, re.search, re.replace,  : 
  Syntax error in regexp pattern. (U_REGEX_RULE_SYNTAX)

I don't understand the reason why this error message appears. The "phrasetotoken" function works fine with collocations of bi-grams only (i.e. size = 2).

Thank you in advance for your help,
Kind regards,
Arnaud.

[kbenoit]

Can you upload the file Test.txt to this issue so that I can test this?

Note that there is no need to us tm for this - you could have gotten the same from quanteda using:

corpusText <- corpus(textfile("Test.txt", encoding = "latin1"))
corpusTextPhrases <- phrasetotoken(corpusText, CollocsText)

and then only lowercased, removed stop words, punctuation, etc when performing a relevant operation such as creating a document-feature matrix.

[atrolle]

Ok, thank you so much for your help.
Here is the file :
Test.txt

KInd regards,
Arnaud.

[kbenoit]

Thanks, I've tested it. Turns out that your text produces collocations where one of the terms is a "+" as a token. This threw an error in phrasetotoken() because it was not escaped. Solutions:

1. Get v0.9.7-22 just pushed, where I caught the exception.
2. Use this in your code:

CollocsText <- removeFeatures(CollocsText, c(stopwords("french"), "+"))

[atrolle]

Thank you again. I had not noticed this "+".

KInd regards,
Arnaud.