dfm_remove behaves like dfm_keep if pattern is dfm

[theresagessler]

Hi,
I a trying to check the dfference between two dfms when selecting features based on one. It seems that if the pattern is a dfm (rather than e.g. a character vector), dfm_remove acts as dfm_keep. First occured with quanteda 1.0 but it still behaves the same after updating
And on this occasion, thanks for all the great work on the package, I was really happy about the new structure form 1.0 upwards!

This is a short reproduction of the problem:

test <- c("this is a test text")
test2 <- c("this is another one")
test_dfm <- dfm(test)
test2_dfm <- dfm(test2)
keep <- dfm_keep(test_dfm, test2_dfm)
remove <- dfm_remove(test_dfm, test2_dfm)
featnames(keep)
## [1] "this"    "is"      "another" "one"    
featnames(remove)
## [1] "this"    "is"      "another" "one"    
diff <- setdiff(featnames(test_dfm), featnames(test2_dfm))
diff_dfm <- dfm_keep(test_dfm, diff)
featnames(diff_dfm)
## [1] "a"    "test" "text"

[koheiw]

Hi @theresagessler, it is a bug (undefined behavior). If pattern is a DFM, tokens_select() makes the features of x identical to pattern's, but tokens_remove() is doing the same as you have discovered.

@kbenoit shall we just disallow pattern to be a DFM for dfm_remove()?

[theresagessler]

Hej @koheiw - that's true, I hadactually used dfm_select for aligning train and test sets to each other. Afterwards, I wanted to see which frequent words I lose when aligning a larger test dfm to the smaller train df - so using dfm_remove with a dfm as pattern was actually my intention.
Of course, when needed, one can do it with setdiff, but then I'd suggest adapting the description of the help file for dfm_remove respectively dfm_select:
"This function selects or removes features from a dfm or fcm, based on feature name matches with pattern. The most common usages are to eliminate features from a dfm already constructed, such as stopwords, or to select only terms of interest from a dictionary."
When reading this and seeing that pattern can be a dfm, I intuitively assumed I could use it to check the overlap and difference.

[koheiw]

I see you point. We will change it to make it more consistent with the documentation. Meanwhile, please use dfm_remove(x, featnames(m)).