Wishlist: remove common feature with dfm_trim()

[cschwem2er]

I'd like to suggest adding parameter to dfm_trim() for removing features that are very common, e.g. similar to stopwords which were not excluded during dfm construction. This could work similar to Gensims filter_extreme function which accepts a parameter no_above and for =0.5 removes terms occuring in more than 50 % of all documents.

[kbenoit]

The threshold parameters take fractions, for instance document frequency fractions. See https://kbenoit.github.io/quanteda/reference/dfm_trim.html. Is that what you mean?

[cschwem2er]

But from what I understand it only removes terms below the fraction right? I'm looking for the opposite, removing terms appearing in e.g. more than 60 percent of all documents.

[koheiw]

One vote for the function. max_docfreq argument would be useful.

[kbenoit]

OK, agreed, we will add max_docfreq and max_count soon. @methodds you're welcome to fork, mod the function, and issue a PR if you wish, otherwise it's on the to do list (short- to medium-term). Probably short-term since it's pretty straightforward to implement.

[kbenoit]

Note: I have started this modification in a branch, soon to be completed. It gives me a chance to tidy up a few other behaviours from dfm_trim.

[cschwem2er]

Thanks :)

[kbenoit]

Done - @methodds pls let us know if this works as expected.

[cschwem2er]

It works perfectly, thanks for the quick integration.