Consider uniformity of textstat functions

[kbenoit]

As we have added more textstat functions, and as we redesign the textstat_simil() and textstat_dist(), we've introduced some inconsistencies in the output object types. It's time to think this through and consider some adjustments.

Differences:

• most functions return a data.frame that has the margin name as the id column (e.g. document or feature), but others such as textstat_entropy return a named numeric vector.
• some functions change the id column depending on the value of the margin function.
• some functions return multiple statistic names (e.g. textstat_readability()) while others (e.g. textstat_keyness()) return only one, even though it has a different column name.
• the new textstat_simil() returns just similarity as the statistic column name, regardless of the value of the measure. Other functions return the name of the measure/method as the statistic.
• the name of the measure is measure for all functions except the simil and dist functions, where it is method.
• In most of the older ones, we classed them as: , textstat, data.frame. But we do not have any methods that are common to a textstat class.

function	output class name	object base	margin?	id column(s)	stat column
textstat_simil	textstat_simil	data.frame	Y	feature/document[1,2]	similarity, regardless of method
textstat_dist	dist	dist	Y	n/a
textstat_entropy	n/a	named numeric	Y	n/a, but names are features or documents	n/a
textstat_collocations	collocations, textstat	data.frame	N	collocation	lambda, z
textstat_frequency	frequency, textstat	data.frame	N	feature	frequency, rank
textstat_keyness	keyness, textstat	data.frame	N	feature	(single) name of measure
textstat_lexdiv	lexdiv, textstat	data.frame	N	document	(multiple) name of measure
textstat_readability	readability, textstat	data.frame	N	document	(multiple) name of measure

Proposals:

1. All textstat functions should return a data.frame.
2. The id column should be one of: feature, document, collocation, with the pairwise simil and dist functions returning feature1, feature2 or document1, document2.
3. We rename method in textstat_simil/dist() to measure, so that all functions use this argument name.
4. We return the statistic as the name of the measure, when multiple options exist.
5. We allow multiple columns to be returned, by passing multiple values to measure.
6. We class the returned data.frames as c("textstat_name", "textstat", "data.frame"). This allows us to class the return from textstat_dist for instance without colliding with the class name of the stats::dist() return. This is also consistent with what we return from the textmodel functions.

[kbenoit]

• textstat_simil and textstat_dist are now Matrix objects, since v1.5, so:
• outstanding question should we change the format of textstat_entropy()? I think we should.

[kbenoit]

Update for changes with v2:

function	output class name	object base	margin?	id column(s)	stat column
textstat_simil	textstat_simil (Matrix)	Y	feature/document[1,2]	method name
textstat_dist	textstat_dist (Matrix)	Y	feature/document[1,2]	method name
textstat_entropy	n/a	named numeric	Y	n/a, but names are features or documents	n/a
textstat_collocations	collocations, textstat	data.frame	N	collocation	lambda, z
textstat_frequency	frequency, textstat	data.frame	N	feature	frequency, rank
textstat_keyness	keyness, textstat	data.frame	N	feature	(single) name of measure
textstat_lexdiv	lexdiv, textstat	data.frame	N	document	(multiple) name of measure
textstat_readability	readability, textstat	data.frame	N	document	(multiple) name of measure

Suggestion

We keep everything as it is, except textstat_entropy() whose output we convert to a data.frame, like this:

library("quanteda")

tstat_entropy <- function(x, margin = c("documents", "features"), base = 2) {
    margin <- match.arg(margin)
    ent <- textstat_entropy(x, margin = margin, base = base)
    entdf <- data.frame(names(ent), entropy = ent, stringsAsFactors = FALSE, 
                        row.names = NULL)
    names(entdf)[1] <- stringi::stri_trim_right(margin, "[^s]")
    entdf
}

tstat_entropy(data_dfm_lbgexample)
#>   document  entropy
#> 1       R1 3.386943
#> 2       R2 3.386943
#> 3       R3 3.386943
#> 4       R4 3.386943
#> 5       R5 3.386943
#> 6       V1 3.386943
tstat_entropy(data_dfm_lbgexample, margin = "features") %>%
    head(10)
#>    feature   entropy
#> 1        A 0.0000000
#> 2        B 0.0000000
#> 3        C 0.0000000
#> 4        D 0.0000000
#> 5        E 0.0000000
#> 6        F 0.1686609
#> 7        G 0.1708952
#> 8        H 0.4371120
#> 9        I 0.6476138
#> 10       J 1.0338027