Can't apply or change dfm weights with dfm_weight after grouping with dfm_group

[juba]

Description and reproducible code

Suppose I've got the following dfm, weighted with a boolean scheme :

m <- matrix(
  c(3,1,5,
    0,0,1,
    4,0,0,
    1,0,2),
  ncol = 3, byrow = TRUE,
)
dfm <- as.dfm(m)
dfm <- dfm_weight(dfm, scheme = "boolean")

Later on I use dfm_group on it :

grouped_dfm <- dfm_group(dfm, groups = c(1, 1, 2, 2))

The resulting dfm is, coherently, not boolean anymore. But if I want to use dfm_weight to convert it back to boolean, I can't because the grouped dfm is still considered weighted :

> dfm_weight(grouped_dfm, scheme = "boolean")
Error in dfm_weight.dfm(grouped_dfm, scheme = "boolean") : 
  this dfm has already been term weighted as: boolean

And I can't either "unweight" it with dfm_weight(scheme = "count") as it generates the same error.

The workaround I use for now is to directly modify grouped_dfm@x.

Expected behavior

I would expect the result of dfm_group not to be weighted, at least for weights that are not preserved by the grouping operation.

System information

R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: KDE neon User Edition 5.14

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3

locale:
 [1] LC_CTYPE=fr_FR.UTF-8       LC_NUMERIC=C               LC_TIME=fr_FR.UTF-8       
 [4] LC_COLLATE=fr_FR.UTF-8     LC_MONETARY=fr_FR.UTF-8    LC_MESSAGES=fr_FR.UTF-8   
 [7] LC_PAPER=fr_FR.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] testthat_2.0.1  bindrcpp_0.2.2  rainette_0.0.1  stringi_1.2.4   RSQLite_2.1.1  
 [6] DBI_1.0.0       quanteda_1.3.14 forcats_0.3.0   stringr_1.3.1   dplyr_0.7.8    
[11] purrr_0.2.5     readr_1.3.0     tidyr_0.8.2     tibble_1.4.2    ggplot2_3.1.0  
[16] tidyverse_1.2.1

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0          lubridate_1.7.4     lattice_0.20-38     prettyunits_1.0.2  
 [5] assertthat_0.2.0    packrat_0.5.0       digest_0.6.18       RSpectra_0.13-1    
 [9] R6_2.3.0            cellranger_1.1.0    plyr_1.8.4          backports_1.1.2    
[13] httr_1.3.1          pillar_1.3.1        rlang_0.3.0.1       progress_1.2.0.9000
[17] lazyeval_0.2.1      readxl_1.1.0        rstudioapi_0.8      data.table_1.11.8  
[21] blob_1.1.1          Matrix_1.2-15       bit_1.1-14          munsell_0.5.0      
[25] broom_0.5.1         compiler_3.5.2      spacyr_0.9.91       modelr_0.1.2       
[29] pkgconfig_2.0.2     tidyselect_0.2.5    crayon_1.3.4        withr_2.1.2        
[33] SnowballC_0.5.1     grid_3.5.2          nlme_3.1-137        jsonlite_1.6       
[37] gtable_0.2.0        magrittr_1.5        scales_1.0.0        RcppParallel_4.4.1 
[41] cli_1.0.1           xml2_1.2.0          stopwords_0.9.0     generics_0.0.2     
[45] fastmatch_1.1-0     tools_3.5.2         bit64_0.9-7         glue_1.3.0         
[49] hms_0.4.2           parallel_3.5.2      colorspace_1.3-2    rvest_0.3.2        
[53] memoise_1.1.0       bindr_0.1.1         haven_2.0.0        

Thanks a lot !

[kbenoit]

Currently, grouping a dfm sums its features by group. (We might broaden this to other aggregation functions in the future, but are not planning on implementing this any time soon.) In your case, it would be better to group first and then weight by Boolean. Then any count in a group will be (correctly) converted to 1.

[juba]

Yes, I think I understand this. The problem just occured because in my workflow, I start from a "count" dfm, use dfm_weight to convert it to boolean, use dfm_group later on, and then would like to use dfm_weight again to convert the grouped dfm back to boolean.

I understand this is a very specific use case that may not justify any code modification, and I've found a workaround. Maybe it could be possible to add an option to manually "unweight" a previously weighted dfm ?

Anyway, thanks for your quick answer and for all your work on quanteda, a really powerful and user-friendly package.

[kbenoit]

Closed with #1547 (comment)