Sparsity should not be 100% unless a dfm is empty

[koheiw]

> mt
Document-feature matrix of: 1,373,125 documents, 1,060,452 features (100% sparse).
> topfeatures(mt)
    rt     us     de    amp    via  https     la  syria    new people 
839814  64723  64585  56413  50938  48393  38586  35449  34465  33232 

[koheiw]

> as.dfm(Matrix::rsparsematrix(1000, 1000, density = 0.0000))
Document-feature matrix of: 1,000 documents, 1,000 features (100% sparse).
> as.dfm(Matrix::rsparsematrix(1000, 1000, density = 0.0001))
Document-feature matrix of: 1,000 documents, 1,000 features (100% sparse).
> as.dfm(Matrix::rsparsematrix(1000, 1000, density = 0.001))
Document-feature matrix of: 1,000 documents, 1,000 features (99.9% sparse).

[kbenoit]

I'm not sure how you generated the first mt, can you distill this into a reprex?

I think the rsparsematrix() result is happening that way because the dgCMatrix at the heart of a dfm contains (double-precision) numerics, not integers. Last I checked there was code in the Matrix package for sparse integer objects but it was commented out. (Probably because it would be complicated to implement and test all of the methods for an integer object.) rsparsematrix() generates numeric matrices, whereas (unless weighted) a dfm will always begin life as an integer matrix.

Right now the sparsity figure is base on the length of the @x slot divided by the dimensions of the matrix (see https://github.com/quanteda/quanteda/blob/master/R/dfm-methods.R#L314).
So if you are plugging in "explicit zeros" or NAs in your experimental branch, that will lead to it being miscomputed.

> (mt <- as.dfm(matrix(c(0,0,0,0), nrow = 2)))
Document-feature matrix of: 2 documents, 2 features (100% sparse).
2 x 2 sparse Matrix of class "dfm"
       features
docs    feat1 feat2
  text1     0     0
  text2     0     0
> mt[1, 1] <- NA
> as.dfm(mt)
Document-feature matrix of: 2 documents, 2 features (75% sparse).
2 x 2 sparse Matrix of class "dfm"
       features
docs    feat1 feat2
  text1    NA     0
  text2     0     0

We never worried about this before since a dfm is not supposed to have NAs.

[kbenoit]

@koheiw is there a reproducible example for this? I don't have access to the data in the opening example.

[koheiw]

There is nothing complex. sparcity() is fine, but fomrat() is the problem.

> format(99.9, digits = 3)
[1] "99.9"
> format(99.99, digits = 3)
[1] "100"
> sparsity(as.dfm(Matrix::rsparsematrix(1000, 1000, density = 0.0001)))
[1] 0.9999
> as.dfm(Matrix::rsparsematrix(1000, 1000, density = 0.0001))
Document-feature matrix of: 1,000 documents, 1,000 features (100% sparse).

[kbenoit]

Ah, got it. I can fix that by specifying a combination of significant digits and nsmall in the format wrapper. Will get to this asap.