textstat_lexdiv not behaving correctly

[kbenoit]

> mytxt <- "one one two one one two one"
> mydfm <- dfm(mytxt)
> textstat_lexdiv(mydfm)
$text1
[1] 0.2857143

$<NA>
[1] 0.3562072

$<NA>
[1] 0.7559289

$<NA>
[1] 0.5345225

$<NA>
[1] 1.312686

$<NA>
[1] 7.133254

$<NA>
[1] 0.8728096

$<NA>
[1] 0.3221615

$<NA>
[1] 0.7418044

[imrem]

There seems to be a problem when using textstat_lexdiv for multiple texts at once. I made an example textcorpus, the first document (1000bla.pdf) is just the word "bla" repeated 1000 times.

> exampletexts <- readtext("http://www.michaelimre.bplaced.net/exampletexts.zip")
> exampletextcorpus <- corpus(exampletexts)
> exampletextDf <- dfm(exampletextcorpus)

If I only calculate the results (in this example the Type-Token Ratio) for the first text, it works just fine (1/1000=0.001)

> textstat_lexdiv(exampletextDf[1], "TTR")

However, if I calculate it for all the texts at once, it doesn't work correctly anymore. The text "100bla.pdf" is seen as the most diverse, even thought it's just the word "bla" repeated 100 times. The value is above 1, which should not even be possible. Generally, when using multiple texts, shorter text seem to get higher scores.

> textstat_lexdiv(exampletextDf, "TTR")

[kbenoit]

Thanks @imrem - we are going to look into this asap.

[HaiyanLW]

@kbenoit The error found by @imrem is due to that ntype() returns the sum of types of tokens across all files. Do you think whether we should keep the functionality of ntype() and just fix textstat_lexdiv() or update ntype()?