Feature request: function to get doc-feature-frequency triples

@koheiw

We're trying to use document metadata to make prettier word clouds.
For this I need to merge the document metadata with the per-document word frequencies.

I currently extract them by casting to a dgTMatrix and accessing the underlying triples, but @koheiw said it might be good to build this into quanteda.

Current code / prototype:

m = as(d, "dgTMatrix")
doc_ids = quanteda::docnames(d)
docvars(d)
terms = factor(m@j, labels = colnames(d))
docnames = factor(m@i, labels = rownames(d))
words = data.frame(docnames=docnames, term=terms, freq=m@x)

Use case: word cloud over time (https://gist.github.com/vanatteveldt/8abdcdeef3c19f641e5f565c5836c971)

We already have as.data.frame.dfm(), which returns dense matrix, so how about as.list.dfm() to generate a triplet? In this case,

as.list.dfm <- function(x) {
   x <- as(x, "dgTMatrix")
   featname <- factor(x@j, labels = colnames(x))
   docname <- factor(x@i, labels = rownames(x))
   list(docnames = docname, term = featname, freq = x@x)
}

words <- data.frame(as.list(d))

I'm sure sure I like the list semantics as the three elements are guaranteed to be of primitive type of the same length, i.e. the list is almost always of a rectangular form. So a df makes much more sense than a list.

I think it is mostly whether you want the df to be wide (=dense matrix) or long (=triplet list). I think either as.data.frame(m, format="long") or a specific function with either long or triples in the name.

Alternatively, I think the tidytext format is essentially the same as the triplet form. So we could also add format='tidy' to the convert function?

But of course I'll happily use whatever name you decide to give the function :)

@kbenoit

I agree that as.data.frame.dfm() is better used for a triple. Users should as.data.frame(as.matrix(d)) if they wanted a dense matrix.

#1212 explains why as.data.frame.dfm() exists already.

@kbenoit what do you think?

Main thought:

We should definitely not create a coercion method that purports to change one object class to another, but in fact changes the format of that object to match a particular paradigm - in this case, the "tidy" concept of a document-term matrix. Not only is that mixing classes with concept abstractions, but also, as you will see below, it's already done in tidytext for us.

Further thoughts:

1. We should integrate this as an option to `textplot_wordcloud()`.

The user should not need to worry about these details, which include the internals of the dfm. This is pretty easily done by adding an option for a covariate. I would propose

textplot_wordcloud.formula(x, ...)

where x could be mydfm ~ mydocvar to get the cloud by some covariate on the x-axis, or by categorical variables to form the comparison cloud that is currently achieved via comparison = TRUE.

2. The current `as.data.frame.dfm()` should not exist.

Right now, this fails:

as(data_dfm_lbgexample[, 1:10], "dgTMatrix") %>%
     as.data.frame()
## Error in as.data.frame.default(.) : 
##  cannot coerce class "structure("dgTMatrix", package = "Matrix")" to a data.frame

If a user wants a data.frame, they should add:

as(data_dfm_lbgexample[, 1:10], "dgTMatrix") %>%
     as.matrix() %>%
     as.data.frame()
##    A B  C  D  E  F   G   H   I   J
## R1 2 3 10 22 45 78 115 146 158 146
## R2 0 0  0  0  0  2   3  10  22  45
## R3 0 0  0  0  0  0   0   0   0   0
## R4 0 0  0  0  0  0   0   0   0   0
## R5 0 0  0  0  0  0   0   0   0   0
## V1 0 0  0  0  0  0   0   2   3  10

To make our dfm consistent with the coercion behaviours from the Matrix package objects from which a dfm inherits, we should not have a coercion method to go straight to a data.frame. Also, we are coming around quickly to the tibble/data.table view that data.frames are to be avoided because of the flaws in their design vis-a-vis row.names.

3. The coercion method to the tidy data.frame already exists in tidytext.

d <- dfm(data_corpus_inaugural, 
         remove_punct = TRUE,
         remove = stopwords("english"))
td <- tidytext::tidy(d)
td
## # A tibble: 39,347 x 3
## document        term            count
## <chr>           <chr>           <dbl>
##     1 1789-Washington fellow-citizens    1.
## 2 1797-Adams      fellow-citizens    3.
## 3 1801-Jefferson  fellow-citizens    2.
## 4 1809-Madison    fellow-citizens    1.
## 5 1813-Madison    fellow-citizens    1.
## 6 1817-Monroe     fellow-citizens    5.
## 7 1821-Monroe     fellow-citizens    1.
## 8 1841-Harrison   fellow-citizens   11.
## 9 1845-Polk       fellow-citizens    1.
## 10 1849-Taylor     fellow-citizens    1.
## # ... with 39,337 more rows

This is identical to (a slightly modified version of) what you proposed:

m <- as(d, "dgTMatrix")
testthat::expect_identical(
    tibble(document = as.character(factor(m@i, labels = rownames(d))), 
           term = as.character(factor(m@j, labels = colnames(d))), 
           count = m@x),
    tidytext::tidy(d)
)

So if you want to add docvars, you could issue a PR to tidytext to add this option as tidy.dfm(x, use_docvars = FALSE, ...), which would essentially be:

docvars(d, "document") <- docnames(d)
dplyr::left_join(td, docvars(d))
## Joining, by = "document"
## # A tibble: 39,347 x 6
## document        term            count  Year President  FirstName    
## <chr>           <chr>           <dbl> <dbl> <chr>      <chr>        
##     1 1789-Washington fellow-citizens    1. 1789. Washington George       
## 2 1797-Adams      fellow-citizens    3. 1797. Adams      John         
## 3 1801-Jefferson  fellow-citizens    2. 1801. Jefferson  Thomas       
## 4 1809-Madison    fellow-citizens    1. 1809. Madison    James        
## 5 1813-Madison    fellow-citizens    1. 1813. Madison    James        
## 6 1817-Monroe     fellow-citizens    5. 1817. Monroe     James        
## 7 1821-Monroe     fellow-citizens    1. 1821. Monroe     James        
## 8 1841-Harrison   fellow-citizens   11. 1841. Harrison   William Henry
## 9 1845-Polk       fellow-citizens    1. 1845. Polk       James Knox   
## 10 1849-Taylor     fellow-citizens    1. 1849. Taylor     Zachary      
## # ... with 39,337 more rows

ps base::merge() when you have loaded the entire tidyverse?? 😉

@koheiw

(1) is cool, but it might not be trivial to allow all options like setting x to be the modal year of a word (e.g. vs the mean year) and using LDA topic info (which is at the word x document level). I'll gladly see how far I can get and make a PR, but I guess @koheiw was already working on an overhaul of the wordcloud, right?

Didn't know about (3), cool. Although I don't really use tidytext, I just wanted to get the dfm in triple format and not too keen to import/depend on tidytext for just that operation - it just struck me that converting it to tidy format (like you have conversions for lda, stm, etc.) would give me what I needed. But if you feel that a dfm-to-triplets function does not fit within quanteda I'll just go on casting to a dgTmatrix :)

ps base::merge() when you have loaded the entire tidyverse?? wink

Yeah, old habits :). I guess inner_join is a lot faster, so I should probably use it by default...

tidytext::tidy.dfm() is already the dfm-to-triplet function. All you need after that is to merge the docvars you want. No need to cast the dgTmatrix since that is done in tidy.dfm().

Adding the option to tidytext as a PR is probably a bad idea since it underscores the inefficiency of the tidy format for things like a triplet, where the docvars repeated for every docname-featname combination, instead of being mapped to a table of docvars (to be didactic, it violates second normal form). If we build this into textplot_wordcloud() we can do the join on the fly and never need to create this structure.

But using the tidier and left_join() the user is free to use all of the tools of the tidyverse. We've worked with the tidytext maintainers to make this as easy as possible.

I still think it is good to have as.list.dfm() that returns numeric indexes for i, j, x starting from 1. This is useful for developers of quanteda extension packages, who are not familiar with Matrix packages and would never find as(x, 'dgTMatrix'). I don't want extension packages to depend on tidytext.

Regarding a new word cloud, I have created a quasi-ggplot version, but did not have time to learn how to use the grid APIs. In short, there is nothing yet.

If we were to create a wordcloud function with may different layout options, it would be a good extension package.

My issue here is that a dfm has no natural list representation; instead, the coercion to a list involves a particular model, in this case a triplet representation, that may or may not have additional information from the dfm.

If a user wants a tidy format, they should call tidytext::tidy(x). If a developer of an extension package needs this, they can use their own function, or call the tidytext function, or use textstat_frequency():

textstat_frequency(data_dfm_lbgexample, groups = docnames(data_dfm_lbgexample)) %>% head(20)
#    feature frequency rank docfreq group
# 1        I       158    1       1    R1
# 2        H       146    2       1    R1
# 3        J       146    3       1    R1
# 4        G       115    4       1    R1
# 5        K       115    5       1    R1
# 6        F        78    6       1    R1
# 7        L        78    7       1    R1
# 8        E        45    8       1    R1
# 9        M        45    9       1    R1
# 10       D        22   10       1    R1
# 11       N        22   11       1    R1
# 12       C        10   12       1    R1
# 13       O        10   13       1    R1
# 14       B         3   14       1    R1
# 15       P         3   15       1    R1
# 16       A         2   16       1    R1
# 17       Q         2   17       1    R1
# 18       N       158    1       1    R2
# 19       M       146    2       1    R2
# 20       O       146    3       1    R2

The output of as.list.dfm() does not need to keep any attributes: it will be only list(i = integer() , j = integer(), x integer()). I do not understand why this is such a big problem (seems only discouraging others to contribute).

Why integers for the first two rather than docnames and featnames?

What about:

testdfm <- dfm(c(doc1 = "a a b c d", doc2 = "b c d d"))
testdfm
# Document-feature matrix of: 2 documents, 4 features (12.5% sparse).
# 2 x 4 sparse Matrix of class "dfm"
#       features
# docs   a b c d
#   doc1 2 1 1 1
#   doc2 0 1 1 2

textstat_frequency(testdfm, groups = docnames(testdfm))[, c( "group", "feature", "frequency")] %>%
    as.list()
# $group
# [1] "doc1" "doc1" "doc1" "doc1" "doc2" "doc2" "doc2"
# 
# $feature
# [1] "a" "b" "c" "d" "d" "b" "c"
# 
# $frequency
# [1] 2 1 1 1 2 1 1

Otherwise:

as_tripletlist.dfm <- function(x) {
    x <- as(x, "dgTMatrix")
    list(i = x@i, j = x@j, x = x@x)
}

as_tripletlist.dfm(testdfm)
# $i
# [1] 0 0 1 0 1 0 1
# 
# $j
# [1] 0 1 1 2 2 3 3
# 
# $x
# [1] 2 1 1 1 1 1 2

What extension package developer cannot write that for themselves?

Character vector is fine, but I though integers are enough as far as index is starting from 1. Users can generate character vector for features or documents:

featnames(dfm)[as_tripletlist(testdfm)$j]
docnames(dfm)[as_tripletlist(testdfm)$i]

We also have pattern2id() to deal with integers directly. I am open to different names like dfm2list() as it is a utility function.

@vanatteveldt

@vanatteveldt, @kbenoit reminded me that we have convert(). I think this is the best place for the function you are proposing.

I am not sure that if we should call it "list" or "triplet", but it will look like:

convert(data_dfm_lbgexample, to = "list")

I hope you like this idea too.

How about “tripletlist”?

We should .Defunct() the as.data.frame.dfm() function - consistent with other sparse Matrix objects, they should coerce it to a matrix and then as.data.frame.matrix(). (Since we obsessing here about consistency.)

"tripletlist" sounds fine to me. convert(to = "data.frame") calls as.data.frame.dfm() 😫

And the to = “data.frame” is not a triplet... So let’s deprecate the as.data.frame.dfm() and just move its code convert(x, to = “data.frame”) - a reversal, in other words.

Add tripletlist to dfm converter for #1321

Merge pull request #1334 from quanteda/dev-tripletlist2

Thanks both!

In response to "What extension package developer cannot write that for themselves?", I thought that quanteda encourages people to use accessor functions rather than access the quanteda internals directly like that, hence the feature request.

Good point!

Navigation Menu

Repository navigation

Feature request: function to get doc-feature-frequency triples #1321

1. We should integrate this as an option to `textplot_wordcloud()`.

2. The current `as.data.frame.dfm()` should not exist.

3. The coercion method to the tidy data.frame already exists in tidytext.

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feature request: function to get doc-feature-frequency triples #1321

Description

Activity

koheiw commented on Apr 22, 2018

vanatteveldt commented on Apr 22, 2018

koheiw commented on Apr 22, 2018

kbenoit commented on Apr 22, 2018

1. We should integrate this as an option to textplot_wordcloud().

2. The current as.data.frame.dfm() should not exist.

3. The coercion method to the tidy data.frame already exists in tidytext.

vanatteveldt commented on Apr 22, 2018

kbenoit commented on Apr 22, 2018

koheiw commented on Apr 27, 2018

kbenoit commented on Apr 27, 2018

koheiw commented on Apr 27, 2018

kbenoit commented on Apr 27, 2018

koheiw commented on Apr 27, 2018

koheiw commented on May 2, 2018

kbenoit commented on May 2, 2018

koheiw commented on May 2, 2018

kbenoit commented on May 2, 2018

vanatteveldt commented on May 9, 2018

kbenoit commented on May 9, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. We should integrate this as an option to `textplot_wordcloud()`.

2. The current `as.data.frame.dfm()` should not exist.