The question has arisen in #1969 of how to handle NA documents. What about empty string documents? Right now, textstat_readability() returns nothing if the document was "". This is not true for other textstat returns. We should make them consistent.
library("quanteda")
## Package version: 2.1.0
txt <- c(d1 = "The cat in the hat", d2 = "", d3 = "Once upon a time.")
corp <- corpus(txt)
toks <- tokens(corp)
dfmat <- dfm(toks)
textstat_readability(txt)
## document Flesch
## 1 d1 117.160
## 2 d3 97.025
textstat_readability(corp)
## document Flesch
## 1 d1 117.160
## 2 d3 97.025
textstat_dist(dfmat) %>%
as.data.frame()
## document1 document2 euclidean
## 1 d3 d1 3.464102
## 2 d1 d3 3.464102
textstat_entropy(dfmat)
## document entropy
## 1 d1 1.921928
## 2 d2 0.000000
## 3 d3 2.321928
textstat_lexdiv(toks)
## document TTR
## 1 d1 0.8
## 2 d2 NaN
## 3 d3 1.0
textstat_lexdiv(dfmat)
## document TTR
## 1 d1 0.8
## 2 d2 NaN
## 3 d3 1.0
Suggestion
Return NA for textstat_readability() if the document is zero.
The question has arisen in #1969 of how to handle
NAdocuments. What about empty string documents? Right now,textstat_readability()returns nothing if the document was"". This is not true for othertextstatreturns. We should make them consistent.Suggestion
Return NA for
textstat_readability()if the document is zero.