dfm_group() produces unexpected results for multiple groups

[kbenoit]

For multiple grouping variables,

• docvars are wrong, compared to a single grouping variable, which both names the new aggregated document with the group label, and adds the group as a docvar;
• it includes all pairwise combinations even when some of these are empty. Should this be automatic, or should there be an option with groups to remove these? (for instance fill = TRUE by default to keep, or FALSE to remove empty combinations)

library("quanteda")
#> Package version: 1.9.9004
#> Parallel computing: 2 of 4 threads used.
#> See https://quanteda.io for tutorials and examples.
#> 
#> Attaching package: 'quanteda'
#> The following object is masked from 'package:utils':
#> 
#>     View

corp <- corpus(c("one two three", "three four five", "five six"),
               docvars = data.frame(num = c(1, 1, 2), alpha = letters[1:3],
                                    grp = c("X", "X", "Y")))

# correct
dfmat1a <- dfm(corp, groups = "grp")
dfmat1a
#> Document-feature matrix of: 2 documents, 6 features (41.7% sparse).
#> 2 x 6 sparse Matrix of class "dfm"
#>     features
#> docs one two three four five six
#>    X   1   1     2    1    1   0
#>    Y   0   0     0    0    1   1
docvars(dfmat1a)
#>   num grp
#> 1   1   X
#> 2   2   Y

# correct
dfmat1b <- dfm(corp, groups = "num")
dfmat1b
#> Document-feature matrix of: 2 documents, 6 features (41.7% sparse).
#> 2 x 6 sparse Matrix of class "dfm"
#>     features
#> docs one two three four five six
#>    1   1   1     2    1    1   0
#>    2   0   0     0    0    1   1
docvars(dfmat1b)
#>   num grp
#> 1   1   X
#> 2   2   Y

# not what we intended?
dfmat2 <- dfm(corp, groups = c("grp", "num"))
dfmat2
#> Document-feature matrix of: 4 documents, 6 features (70.8% sparse).
#> 4 x 6 sparse Matrix of class "dfm"
#>      features
#> docs  one two three four five six
#>   X.1   1   1     2    1    1   0
#>   Y.1   0   0     0    0    0   0
#>   X.2   0   0     0    0    0   0
#>   Y.2   0   0     0    0    1   1
docvars(dfmat2)
#>   num  grp
#> 1   1    X
#> 2  NA <NA>
#> 3  NA <NA>
#> 4   2    Y

^{Created on 2019-12-17 by the reprex package (v0.3.0)}

[koheiw]

Seems like tokens_group() works like dfm_group(fill= TRUE) I will change this.

require(quanteda)
#> Loading required package: quanteda
#> Package version: 1.9.9004
#> Parallel computing: 2 of 12 threads used.
#> See https://quanteda.io for tutorials and examples.
#> 
#> Attaching package: 'quanteda'
#> The following object is masked from 'package:utils':
#> 
#>     View
corp <- corpus(c("one two three", "three four five", "five six"),
               docvars = data.frame(num = c(1, 1, 2), alpha = letters[1:3],
                                    grp = c("X", "X", "Y")))
toks <- tokens(corp)

dfm(toks, groups = c("grp", "num"))
#> Document-feature matrix of: 4 documents, 6 features (70.8% sparse).
#> 4 x 6 sparse Matrix of class "dfm"
#>      features
#> docs  one two three four five six
#>   X.1   1   1     2    1    1   0
#>   Y.1   0   0     0    0    0   0
#>   X.2   0   0     0    0    0   0
#>   Y.2   0   0     0    0    1   1

dfmt <- dfm(corp)
dfm_group(dfmt, c("grp", "num"))
#> Document-feature matrix of: 2 documents, 6 features (41.7% sparse).
#> 2 x 6 sparse Matrix of class "dfm"
#>      features
#> docs  one two three four five six
#>   X.1   1   1     2    1    1   0
#>   Y.2   0   0     0    0    1   1
dfm_group(dfmt, c("grp", "num"), fill = TRUE)
#> Document-feature matrix of: 4 documents, 6 features (70.8% sparse).
#> 4 x 6 sparse Matrix of class "dfm"
#>      features
#> docs  one two three four five six
#>   X.1   1   1     2    1    1   0
#>   Y.1   0   0     0    0    0   0
#>   X.2   0   0     0    0    0   0
#>   Y.2   0   0     0    0    1   1

quanteda:::tokens_group(toks, c("grp", "num"))
#> tokens from 4 documents.
#> X.1 :
#> [1] "one"   "two"   "three" "three" "four"  "five" 
#> 
#> Y.1 :
#> character(0)
#> 
#> X.2 :
#> character(0)
#> 
#> Y.2 :
#> [1] "five" "six"

^{Created on 2019-12-17 by the reprex package (v0.3.0)}