Error in validObject(r) : invalid class “dgTMatrix” object: length(Dimnames[2]) differs from Dim[2] which is 33411

[7804j]

Hi again,

When I create a dfm from my emails corpus, I get the error message:

Creating a dfm from a corpus ...
... lowercasing
... tokenizing
... indexing documents: 1,882 documents
... indexing features: 4,596 feature types
Error in validObject(r) :
invalid class “dgTMatrix” object: length(Dimnames[2]) differs from Dim[2] which is 33411

Below is my code. The data comes from https://www.kaggle.com/kaggle/hillary-clinton-emails/ again.

Also, interestingly, when I write LIMIT 100 at the end of the SQL query, with exactly the same code for everything else, I don't have any problem. So it might be linked with the size of the corpus.

---

Connect to db

db <- dbConnect(dbDriver("SQLite"), "output/database.sqlite")

get all emails

emails <- dbGetQuery(db, "
SELECT ExtractedBodyText body,MetadataSubject subject, MetadataDateSent date
FROM Emails e
INNER JOIN Persons p
ON e.SenderPersonId=P.Id
WHERE p.Name='Hillary Clinton'
AND e.ExtractedBodyText != ''
ORDER BY RANDOM()")

Create new column with weekdays, and column weekend

emails = emails %>% separate(date,"date", sep = "T") %>% mutate(
weekday = weekdays(as.Date(emails$date)),
weekend = ifelse(weekday %in% c('Saturday','Sunday'),1,0)
)

Clean some of the email bodies that still contain part of the header (manual inspection of emails)

emails = emails %>% mutate(body = sub("H <.Re:.\n", "", body)) %>%
mutate(body = sub("H <._Re:", "", body)) %>%
mutate(body = sub("H <._RELEASE IN.B6", "", body)) %>%
mutate(body = sub("RELEASE\nIN PART B6\n", "", body)) %>%
mutate(body = sub("RELEASE\nIN PART B6", "", body)) %>%
mutate(body = sub("RELEASE IN PART\nB6", "", body)) %>%
mutate(body = sub("RELEASE IN PART.\B1", "", body)) %>%
mutate(body = sub("H <._Fw:", "", body)) %>%
mutate(body = sub('Declassify on: 04/23/2035', "", body)) %>%
mutate(body = sub("H <._B6\nB6\n", "", body)) %>%
mutate(body = sub("H <._PM\n", "", body)) %>%
mutate(body = sub("H <._AM\n", "", body))

Create corpus

email_corpus = corpus(emails$body)

Create dfm, after stemming and removing stopwords

email_dfm <- dfm(email_corpus,ignoredFeatures = stopwords("english"), stem = TRUE)

[richard-ian-carpenter]

I am getting a similar error message. I am using quanteda v0.9.6-1.

I am doing the capstone project for the Data Science certificate offered by Coursera. I have created to training datasets based on text files. Three weeks ago, everything was working fine. I started working on the final version of the project and I'm getting that error for two of three datasets.

I did upgrade to R 3.3.0 and had to reinstall several packages, including quanteda, as well as dependencies.

Is it possible that something "broke" with the latest R update?

Example follows:
blogDfm <- dfm(blogCorpus) #, ignoredFeatures = stopwords("english"), stem = FALSE)
Creating a dfm from a corpus ...
... lowercasing
... tokenizing
... indexing documents: 134,818 documents
... indexing features: 130,551 feature types
Error in validObject(r) :
invalid class “dgTMatrix” object: length(Dimnames[2]) differs from Dim[2] which is 5526105

[kbenoit]

The email dataset works fine for me, I just tested it. Could be that your version of the Matrix package needs updating.

> email_dfm <- dfm(email_corpus, ignoredFeatures = stopwords("english"), stem = TRUE)
Creating a dfm from a corpus ...
   ... lowercasing
   ... tokenizing
   ... indexing documents: 1,882 documents
   ... indexing features: 4,765 feature types
   ... removed 155 features, from 174 supplied (glob) feature types
   ... stemming features (English), trimmed 872 feature variants
   ... created a 1882 x 3739 sparse dfm
   ... complete. 
Elapsed time: 0.133 seconds.

Rerun the code after:

update.packages(ask = FALSE)

My session info:

> sessionInfo()
R version 3.2.4 (2016-03-10)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.4 (El Capitan)

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] quanteda_0.9.6-5 tidyr_0.4.1      magrittr_1.5     dplyr_0.4.3      RSQLite_1.0.0    DBI_0.3.1       

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.4      lattice_0.20-33  XML_3.98-1.4     assertthat_0.1   SnowballC_0.5.1  chron_2.3-47     grid_3.2.4       R6_2.1.2         stringi_1.0-1    lazyeval_0.1.10 
[11] data.table_1.9.6 ca_0.64          Matrix_1.2-4     tools_3.2.4      parallel_3.2.

[7804j]

This still doesn't work for me, even after updating, as long as I have this code before creating the dfm:

emails = emails %>% mutate(body = sub("H <.Re:.\n", "", body)) %>%
mutate(body = sub("H <.Re:", "", body))

So it looks like the sub() created the bug (maybe by creating some empty documents, as you mentioned above).

Here is my sessionInfo():

R version 3.3.0 (2016-05-03)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252    LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C                            LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] gmodels_2.16.2   quanteda_0.9.6-1 tidyr_0.4.1      ggplot2_2.1.0    dplyr_0.4.3      RSQLite_1.0.0    DBI_0.4-1       

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.5      lattice_0.20-33  gtools_3.5.0     assertthat_0.1   MASS_7.3-45      chron_2.3-47     grid_3.3.0       R6_2.1.2        
 [9] plyr_1.8.3       gtable_0.2.0     magrittr_1.5     scales_0.4.0     stringi_1.0-1    lazyeval_0.1.10  gdata_2.17.0     data.table_1.9.6
[17] ca_0.64          Matrix_1.2-6     tools_3.3.0      munsell_0.4.3    parallel_3.3.0   colorspace_1.2-6

[7804j]

Should I wait for the next stable release or download the developer version perhaps?

[kbenoit]

Please reinstall Quanteda from GitHub, you need 0.9.6-5.

[richard-ian-carpenter]

Updated quanteda from GutHub using devtools. Everything is working now. Thank you!

sessionInfo():

R version 3.3.0 (2016-05-03)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.4 LTS

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] quanteda_0.9.6-7 Matrix_1.2-6 data.table_1.9.6 stringi_1.0-1 lattice_0.20-33 igraph_1.0.1 ggplot2_2.1.0

loaded via a namespace (and not attached):
[1] Rcpp_0.12.5 digest_0.6.9 withr_1.0.1 R6_2.1.2 chron_2.3-47 grid_3.3.0 plyr_1.8.3 gtable_0.2.0 git2r_0.15.0
[10] magrittr_1.5 scales_0.4.0 httr_1.1.0 curl_0.9.7 ca_0.64 devtools_1.11.1 tools_3.3.0 munsell_0.4.3 parallel_3.3.0
[19] colorspace_1.2-6 memoise_1.0.0 knitr_1.13

[kbenoit]

Great.

[hugokoopmans]

This is still an issue for me...
R 3.3.2
quanteda 0.9.9-3

> myDfm <- dfm(myCorpus, removeNumbers = TRUE) Error in validObject(r) : invalid class “dgTMatrix” object: length(Dimnames[1]) differs from Dim[1] which is 2027

with 500.000 documents it fails
with 100.000 same
with 10k all works fine

[kbenoit]

Yes this is a bug in v0.9.9-3. I just today submitted a fixed version to CRAN, which can be installed from the GitHub repository now if you don't wish to wait.

[hugokoopmans]

yep done and solved thx

[cesarmolea]

Hi,

I am having the same issue as the title on this post. When I try to create a sparse matrix it gives me the following error:

Error in validObject(r) :
invalid class “dgTMatrix” object: length(Dimnames[1]) differs from Dim[1] which is 66

I tried to install quanteda from devtools but it's not helping. Any suggestions?