allow sparsity reduction of tf-idf weighted dfm

[hlageek]

Currently, it is possible to perform dfm_trim() on dfm object and then add tf-idf weights, however, when I want to add tf-idf weights first by using all terms for tf-idf calculations and only then reduce the sparsity of the matrix, I get Error in docfreq.dfm(x) : this dfm has already been term weighted as:inverse10.

[kbenoit]

Please provide a reproducible example.

[hlageek]

Sorry. Here is one:

dfm.object1 <- dfm(c("the quick brown fox jumps over the lazy dog", "the quick brown foxy ox jumps over the lazy god"))
dfm.object1
dfm.trimmed1 <- dfm_trim(dfm.object1, min_docfreq = 0.9)
dfm.trimmed1
dfm.tfidf1 <- dfm_tfidf(dfm.trimmed1)
dfm.tfidf1

dfm.object2 <- dfm(c("the quick brown fox jumps over the lazy dog", "the quick brown foxy ox jumps over the lazy god"))
dfm.object2
dfm.tfidf2 <- dfm_tfidf(dfm.object2)
dfm.tfidf2
dfm.trimmed2 <- dfm_trim(dfm.tfidf2, min_docfreq = 0.9)

Document-feature matrix of: 2 documents, 11 features (22.7% sparse).
2 x 11 sparse Matrix of class "dfm"
       features
docs    the quick brown fox jumps over lazy dog foxy ox god
  text1   2     1     1   1     1    1    1   1    0  0   0
  text2   2     1     1   0     1    1    1   0    1  1   1
Document-feature matrix of: 2 documents, 6 features (0% sparse).
2 x 6 sparse Matrix of class "dfm"
       features
docs    the quick brown jumps over lazy
  text1   2     1     1     1    1    1
  text2   2     1     1     1    1    1
Document-feature matrix of: 2 documents, 6 features (0% sparse).
2 x 6 sparse Matrix of class "dfm"
       features
docs    the quick brown jumps over lazy
  text1   0     0     0     0    0    0
  text2   0     0     0     0    0    0
Document-feature matrix of: 2 documents, 11 features (22.7% sparse).
2 x 11 sparse Matrix of class "dfm"
       features
docs    the quick brown fox jumps over lazy dog foxy ox god
  text1   2     1     1   1     1    1    1   1    0  0   0
  text2   2     1     1   0     1    1    1   0    1  1   1
Document-feature matrix of: 2 documents, 11 features (22.7% sparse).
2 x 11 sparse Matrix of class "dfm"
       features
docs    the quick brown     fox jumps over lazy     dog    foxy      ox     god
  text1   0     0     0 0.30103     0    0    0 0.30103 0       0       0      
  text2   0     0     0 0           0    0    0 0       0.30103 0.30103 0.30103
 Show Traceback
Error in docfreq.dfm(x) : this dfm has already been term weighted as:inverse10

[kbenoit]

The short answer is: It's partly a bug (which I will address soon) and partly a feature, designed to prevent users from applying an inappropriate definition of the trim thresholds in dfm_trim() to a weighted dfm.

The simple reproducible error is:

data_dfm_lbgexample %>%
    dfm_tfidf() %>%
    dfm_trim()
## Error in docfreq.dfm(x) : 
##     this dfm has already been term weighted as:inverse10 

More soon.

[hlageek]

Thanks in advance for the fixing the bug part... and even bigger thanks in retrospect for quanteda :)

[kbenoit]

Should be working now as of 4a5d606. The min_docfreq = 0.9 will eliminate features that occur in fewer than 90% of your documents. In your example, with ndoc = 2, this means it will eliminate any feature that does not occur in both documents.