textstat_collocations error when tokens used on vector with spaces

[trinker]

As of quanteda .99 there is an unexpected error that occurs when you pass a vector with an empty element (i.e., "" ), tokenize, and then run textstat_collocations. If you pass the vector into textstat_collocations directly it runs fine but if you run tokens first you get the error below:

txt <- c('I like good ice cream.', 'Me too!  I like good ice cream.', '')

textstat_collocations(txt, size = 2, min_count = 2, tolower = TRUE)

##   collocation count length   lambda        z
## 1   ice cream     2      2 4.317488 2.027787
## 2   like good     2      2 4.317488 2.027787
## 3      i like     2      2 4.317488 2.027787
## 4    good ice     2      2 4.317488 2.027787

tkns <- quanteda::tokens(tolower(txt), remove_punct = TRUE, remove_symbols = TRUE)
textstat_collocations(tkns, size = 2, min_count = 2)

 ## Error in `[[<-`(`*tmp*`, allempties, value = character(0)) : 
# #  attempt to select less than one element in integerOneIndex 

[koheiw]

@kbenoit , it seems that tokens_segment_by_punctuation is causing the problem. I can fix this using tokens_segment, but I cannot find that function in the master. Where it has gone?

[kbenoit]

Yes that's right, and I know what the problem is and how to fix it, just no time today. Will fix this pm.

[koheiw]

OK, I will leave it with you. Let me know if you want me to change tokens_segment.