Describe the bug
unexpected behavior in fcm. Specifically: @meta$object$margin counts are the same whether the texts are lower-cased or not. Even more specifically, it looks like fcm lowercases prior to doing counts (but not prior to printing the matrix), which is a problem if the original text contains upper-case tokens (to be preserved) and one needs to feed the fcm object to another function.
Reproducible code
require(quanteda)
txt3 <- c("The quick brown fox jumped over the lazy dog.",
"The dog jumped and ate the fox.")
toks3 <- tokens(char_tolower(txt3), remove_punct = TRUE)
f3 <- fcm(toks3, context = "window", window = 3)
print(f3)
print(f3@meta$object$margin)
toks3.1 <- tokens(txt3, remove_punct = TRUE)
f3.1 <- fcm(toks3.1, context = "window", window = 3)
print(f3.1)
print(f3.1@meta$object$margin)
Expected behavior
identical(f3, f3.1)
[1] FALSE
which is as expected (because one of the matrices is bigger than the other). But
> identical(f3@meta$object$margin , f3.1@meta$object$margin)
[1] TRUE
is unexpected, because the texts are different insofar as the latter is not lowercased (note e.g. the absence of The in the second object).
System information
R version 4.2.0 (2022-04-22 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19043)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.utf8 LC_CTYPE=English_United States.utf8
[3] LC_MONETARY=English_United States.utf8 LC_NUMERIC=C
[5] LC_TIME=English_United States.utf8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] conText_1.1.0 text2vec_0.6.1 stringr_1.4.0 readtext_0.81 quanteda_3.2.1
loaded via a namespace (and not attached):
[1] Rcpp_1.0.8.3 plyr_1.8.7 pillar_1.7.0
[4] compiler_4.2.0 tools_4.2.0 stopwords_2.3
[7] digest_0.6.29 lifecycle_1.0.1 tibble_3.1.7
[10] lattice_0.20-45 pkgconfig_2.0.3 rlang_1.0.2
[13] Matrix_1.4-1 fastmatch_1.1-3 cli_3.3.0
[16] mlapi_0.1.1 parallel_4.2.0 RhpcBLASctl_0.21-247.1
[19] httr_1.4.3 dplyr_1.0.9 generics_0.1.2
[22] vctrs_0.4.1 tidyselect_1.1.2 grid_4.2.0
[25] glue_1.6.2 data.table_1.14.2 R6_2.5.1
[28] fansi_1.0.3 reshape2_1.4.4 tidyr_1.2.0
[31] purrr_0.3.4 lgr_0.4.3 magrittr_2.0.3
[34] ellipsis_0.3.2 float_0.3-0 rsparse_0.5.0
[37] utf8_1.2.2 stringi_1.7.6 RcppParallel_5.1.5
[40] crayon_1.5.1
Additional info
Not sure if the quanteda team is aware of this or not -- is there some other obvious way to pull the counts from the fcm object? If not, would be great if default behavior was to not lowercase before doing counts.
Describe the bug
unexpected behavior in
fcm. Specifically:@meta$object$margincounts are the same whether the texts are lower-cased or not. Even more specifically, it looks likefcmlowercases prior to doing counts (but not prior to printing the matrix), which is a problem if the original text contains upper-case tokens (to be preserved) and one needs to feed thefcmobject to another function.Reproducible code
Expected behavior
which is as expected (because one of the matrices is bigger than the other). But
is unexpected, because the texts are different insofar as the latter is not lowercased (note e.g. the absence of
Thein the second object).System information
Additional info
Not sure if the
quantedateam is aware of this or not -- is there some other obvious way to pull the counts from thefcmobject? If not, would be great if default behavior was to not lowercase before doing counts.