inconsistent output between topfeatures and docfreq

[BobMuenchen]

I find the use of topfeatures and docfreq needlessly confusing. Why not have freq_words, (or freq_tokens) and freq_docs? Both should have an “n” argument to show just the top few. Here’s an example that demonstrates how confusing the two sets of output is. topfeatures sorts by frequency by default, while docfreq sorts alphabetically. Since Appearance came out #1 in both, I assumed they were sorted the same way and that Humor had gone from #2 to #7. (This is the WordStat seeking dictionary applied to the seeking.csv).

> topfeatures(seeking_dfm_topics) # Counting terms.
   APPEARANCE         HUMOR     EDUCATION     NIGHTLIFE          ARTS COMMUNICATION       OUTDOOR 
           82            43            35            34            31            27            20 
    SEXUALITY          WORK        SPORTS 
           17            12            10 
> docfreq(seeking_dfm_topics)  # Counting docs.
   APPEARANCE          ARTS COMMUNICATION     EDUCATION        FAMILY       FINANCE         HUMOR 
           43            21            18            20             4             5            30 
    NIGHTLIFE       OUTDOOR     SEXUALITY  SPIRITUALITY        SPORTS          WORK 
           23            14            14             5             6            12

[kbenoit]

The two commands are very different. docfreq returns the document frequency of features, defined as the number of documents in which a feature occurs (see this SO question). It does not take an n argument but rather returns a named vector equal in length to the column dimension of the dfm.

topfeatures returns the most frequently occurring features by the sum of their values in the dfm - normally, counts but would be a weight if the dfm has been weighted - and this is sorted in descending order. Only the top n are returned, reflecting that this function is really just a convenience function for getting a quick look at the most commonly occurring features in a dfm, in one command.

If you want results that can be compared like for like, consider using colSums on th dfm:

mydfm <- dfm(data_char_ukimmig2010)
identical(names(docfreq(mydfm)), names(colSums(mydfm)))
# [1] TRUE
plot(colSums(mydfm), xlab = "Total term count",
     docfreq(mydfm), ylab = "Document frequency")

[BobMuenchen]

I understand how the two commands work, and I'm not trying to imply that the docfreq function is not useful, just that it seems to me that a quick look at the top frequencies at both the term and document level would be of interest. Also a detailed vector-based look at both levels would be of interest.

[kbenoit]

What sort of output would you want, and how would the new function look?

Perhaps:

topfeatures(x, count = c("termfreq", "docfreq"))

where including both count types produces a data.frame or a list?

[BobMuenchen]

That would be good, perhaps with descending=TRUE by default, sorting on whichever one was specified first. I don't know what you think of the tibble type of data frame that dplyr has made popular, but what's nice about it is that it'll print the first 10 rows for most data frames. But if the whole data frame is less than 20 rows, it'll print them all. That negates the need for an n=10 type of argument. Both of those values are adjustable through options. I love those defaults since if I'm using a dictionary, I rarely have more than 20 topics.

…

On Fri, Jan 13, 2017 at 8:57 AM, Kenneth Benoit ***@***.***> wrote: What sort of output would you want, and how would the new function look? Perhaps: topfeatures(x, count = c("termfreq", "docfreq")) where including *both* count types produces a data.frame or a list? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#408 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC_53ATb9-zDUwitiOospy1AnRgOJ6INks5rR4LHgaJpZM4LasDc> .

-- ================================================== Bob Muenchen (pronounced MINchen) Accredited Professional Statistician™ Manager, Research Computing Support, U of TN Voice: (865) 974-5230 Email: muenchen@utk.edu Twitter: @BobMuenchen Web Site: http://r4stats.com ==================================================