STM Conversion runtime

[cschwem2er]

Hi,

I'm currently trying to convert a large sparse dfm object (932854 x 422225) to STM format and the conversion never finishes. Does the conversion to STM somehow scale very bad with dfm size? And is there a way to speed this up?

I was very happy to see that quanteda offers such a conversion because I highly prefer quanteda preprocessing over tm preprocessing (the STM default).

Cheers,
Carsten

[kbenoit]

Glad to hear you like the package!

I'd be happy to debug (if that's what it is) the convert function with your data if you send me a link to the object.

[cschwem2er]

Hey :) Thanks for your fast response and the offer to debug. What kind of format would you prefer? Just an R binary image?

[kbenoit]

There was a loop in the code for stm conversion that never should have been in there... Fixed now. I also made some "improvements" to your code, since you were not using some of the quanteda functions that should make it easier.

library(quanteda)

# a better way to load in and work with your files
pegidaCorpus <- textfile('pegida_fbmerged_16-13-06.csv', textField = "comment_message")
pegidaCorpus[["numdate"]] <- as.numeric(pegidaCorpus[["date_comment"]] - min(pegidaCorpus[["date_comment"]]))
pegidaCorpus[["core"]] <- pegidaCorpus[["core_user99"]]
Dfm <- dfm(pegidaCorpus, 
           ignoredFeatures = c(stopwords('german'), 'dass', 'wurde', 'wurden', 'pegida'),
           stem = TRUE, language='german')

# from the df supplied
pegidaCorpus <- corpus(df, textField = "comment_message")
pegidaCorpus
## Corpus consisting of 932,854 documents and 2 docvars.

Dfm <- dfm(pegidaCorpus, 
           ignoredFeatures = c(stopwords('german'), 'dass', 'wurde', 'wurden', 'pegida'),
           stem = TRUE, language='german')
## Creating a dfm from a corpus ...
##   ... lowercasing
##   ... tokenizing
##   ... indexing documents: 932,854 documents
##   ... indexing features: 527,208 feature types
##   ... removed 233 features, from 235 supplied (glob) feature types
##   ... stemming features (German), trimmed 104920 feature variants
##   ... created a 932854 x 422056 sparse dfm
##   ... complete. 
## Elapsed time: 209.847 seconds.

# conversion is MUCH faster now - as fast as STM
system.time(processed <- convert(Dfm, to = "stm"))
##   user  system elapsed 
## 22.647   1.538  24.196 

# some improvement if you reduce the number of features
# will certainly improve your topic model fit
Dfm2 <- trim(Dfm, minCount = 5)
## Removing features occurring fewer than 5 times: 352826
system.time(processed2 <- convert(Dfm2, to = "stm"))
##   user  system elapsed 
## 17.585   1.501  19.094 

[cschwem2er]

Many thanks for updating the conversion function :) There is a however a small problem left: As far as I understand the conversion function checks if a dfm row contains no entries (due to trimming, stopwords, whatever) and automatically removes those for the processed object. The result is an object with less documents than before, but without producing an output about which documents have been removed.

This is a problem because for some stm postestimation commands it is necessary for the metadata to have the same length than the documents used for the model fit. It would be very nice if the conversion function could somehow indicate which documents (rows) were removed, so that one can manually trim the metadata accordingly. Even better would be if one could pass the meta dataframe as additional object to convert and this object gets automatically trimmed and added as processed$meta, because at the moment $meta defaults to NULL.

Cheers,
Carsten

[kbenoit]

Added a docvars = argument to convert, now will return appropriately selected meta in the list.

I'm running against the same issue with converting to topicmodels. Could this be fixed too? Or can I just also convert my dfm to an stm object an then use that object's metadate to include it in the topicmodels object? Or will the numbering be wrong?

[kbenoit]

@jansenjoost Can you open a new issue and provide a reproducible example?

An object's metadata cannot be part of a document-term matrix that the topicmodels package can use, since this is a tm formatted dfm (with no empty documents). It does not include a container for document variables.

However, you would still be able to access the document variables from docvars(thequantedadfm).