dfm_compress() drops _document from @docvars slot

[kbenoit]

Describe the bug

dfm.dfm() drops the _document column from the internal @docvars data.frame.

Yet one more reason to get rid of reliance on row.names and shift to a universal document key for all docvars!

Note: A similar problem does not happen for tokens.tokens().

Reproducible code

Please paste minimal code that reproduces the bug. If possible, please upload the data file as .rds.

dfmorig <- dfm(data_corpus_irishbudget2010)
dfmorig@docvars
#                           year debate number      foren     name party                 _document
# Lenihan, Brian (FF)       2010 BUDGET     01      Brian  Lenihan    FF       Lenihan, Brian (FF)
# Bruton, Richard (FG)      2010 BUDGET     02    Richard   Bruton    FG      Bruton, Richard (FG)
# Burton, Joan (LAB)        2010 BUDGET     03       Joan   Burton   LAB        Burton, Joan (LAB)
# Morgan, Arthur (SF)       2010 BUDGET     04     Arthur   Morgan    SF       Morgan, Arthur (SF)
# Cowen, Brian (FF)         2010 BUDGET     05      Brian    Cowen    FF         Cowen, Brian (FF)
# Kenny, Enda (FG)          2010 BUDGET     06       Enda    Kenny    FG          Kenny, Enda (FG)
# ODonnell, Kieran (FG)     2010 BUDGET     07     Kieran ODonnell    FG     ODonnell, Kieran (FG)
# Gilmore, Eamon (LAB)      2010 BUDGET     08      Eamon  Gilmore   LAB      Gilmore, Eamon (LAB)
# Higgins, Michael (LAB)    2010 BUDGET     09    Michael  Higgins   LAB    Higgins, Michael (LAB)
# Quinn, Ruairi (LAB)       2010 BUDGET     10     Ruairi    Quinn   LAB       Quinn, Ruairi (LAB)
# Gormley, John (Green)     2010 BUDGET     11       John  Gormley Green     Gormley, John (Green)
# Ryan, Eamon (Green)       2010 BUDGET     12      Eamon     Ryan Green       Ryan, Eamon (Green)
# Cuffe, Ciaran (Green)     2010 BUDGET     13     Ciaran    Cuffe Green     Cuffe, Ciaran (Green)
# OCaolain, Caoimhghin (SF) 2010 BUDGET     14 Caoimhghin OCaolain    SF OCaolain, Caoimhghin (SF)

dfm(dfmorig)@docvars
#                           year debate number      foren     name party
# Lenihan, Brian (FF)       2010 BUDGET     01      Brian  Lenihan    FF
# Bruton, Richard (FG)      2010 BUDGET     02    Richard   Bruton    FG
# Burton, Joan (LAB)        2010 BUDGET     03       Joan   Burton   LAB
# Morgan, Arthur (SF)       2010 BUDGET     04     Arthur   Morgan    SF
# Cowen, Brian (FF)         2010 BUDGET     05      Brian    Cowen    FF
# Kenny, Enda (FG)          2010 BUDGET     06       Enda    Kenny    FG
# ODonnell, Kieran (FG)     2010 BUDGET     07     Kieran ODonnell    FG
# Gilmore, Eamon (LAB)      2010 BUDGET     08      Eamon  Gilmore   LAB
# Higgins, Michael (LAB)    2010 BUDGET     09    Michael  Higgins   LAB
# Quinn, Ruairi (LAB)       2010 BUDGET     10     Ruairi    Quinn   LAB
# Gormley, John (Green)     2010 BUDGET     11       John  Gormley Green
# Ryan, Eamon (Green)       2010 BUDGET     12      Eamon     Ryan Green
# Cuffe, Ciaran (Green)     2010 BUDGET     13     Ciaran    Cuffe Green
# OCaolain, Caoimhghin (SF) 2010 BUDGET     14 Caoimhghin OCaolain    SF

Expected behaviour

The second example above should also have a _document column.

Additional info

When this is fixed, it will resolve the two tests that are currently failing in PR #1504.

• https://github.com/quanteda/quanteda/blob/revise-documentation/tests/testthat/test-dfm.R#L224
• https://github.com/quanteda/quanteda/blob/revise-documentation/tests/testthat/test-dfm.R#L230

[kbenoit]

After much hunting, I discovered that the bug comes from docvars getting dropped in the internal group_dfm() function, called by dfm_compress(), which is called by dfm_tolower().

library("quanteda")
## Package version: 1.3.16
## Parallel computing: 2 of 12 threads used.
## See https://quanteda.io for tutorials and examples.
## 
## Attaching package: 'quanteda'
## The following object is masked from 'package:utils':
## 
##     View
corp <- corpus(c(
  d1 = "A A A b c D D",
  d2 = "b b b b D D D"
),
docvars = data.frame(bool = c(TRUE, FALSE))
)
thedfm <- dfm(corp)
# no _document
thedfm@docvars
##     bool
## d1  TRUE
## d2 FALSE

# this adds the _document
docnames(thedfm) <- docnames(thedfm)
thedfm@docvars
##     bool _document
## d1  TRUE        d1
## d2 FALSE        d2

dfm(thedfm, tolower = FALSE)@docvars
##     bool _document
## d1  TRUE        d1
## d2 FALSE        d2
# this kills it
dfm(thedfm, tolower = TRUE)@docvars
##     bool
## d1  TRUE
## d2 FALSE

# because dfm_tolower kills it
dfm_tolower(thedfm)@docvars
##     bool
## d1  TRUE
## d2 FALSE
# because it calls dfm_compress which handles docvars wrongly
dfm_compress(thedfm, margin = "features")@docvars
##     bool
## d1  TRUE
## d2 FALSE
dfm_compress(thedfm, margin = "documents")@docvars
## data frame with 0 columns and 2 rows