"Duplicate row.names" error when transforming from TM to Quanteda Corpus

[Ninoninoninonino]

When converting from TM to Quanteda Corpus object I get the error "Error in data.frame(texts = rep(NA, length(x)), row.names = names(x), :
duplicate row.names: ". Would there be a possibility to do something about this from the Quanteda side?
Thanks a lot!

[kbenoit]

I'm afraid we need more details than that... sessionInfo()? And some details about what you are trying to convert please.

[Ninoninoninonino]

Thanks, kbenoit.
I created corpora in tm, then converted them to Quanteda corpus format for applying automatic language recognition with cldr, before checking for duplicates with RNewsflow in DTM format and passing it to stm. I realised that I can probably just stick to using either tm or Quanteda. It would just be really convenient if I didn't have to re-write my scripts ..

sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X El Capitan 10.11.6

locale:
[1] C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] tidytext_0.1.2 dplyr_0.5.0 textreuse_0.1.4
[4] RNewsflow_1.0.1 Matrix_1.2-8 igraph_1.0.1
[7] cldr_1.1.0 stm_1.1.3 foreign_0.8-67
[10] data.table_1.10.0 tm_0.6-2 NLP_0.1-10
[13] qdap_2.2.5 RColorBrewer_1.1-2 qdapTools_1.3.1
[16] qdapRegex_0.6.0 qdapDictionaries_1.0.6 stringr_1.2.0
[19] XML_3.98-1.5 RCurl_1.95-4.8 bitops_1.0-6
[22] quanteda_0.9.9-26

loaded via a namespace (and not attached):
[1] Rcpp_0.12.9 lattice_0.20-34 tidyr_0.6.1 xlsxjars_0.6.1
[5] gtools_3.5.0 psych_1.6.12 assertthat_0.1 glmnet_2.0-5
[9] foreach_1.4.3 slam_0.1-40 R6_2.2.0 plyr_1.8.4
[13] chron_2.3-50 ggplot2_2.2.1 lazyeval_0.2.0 gdata_2.17.0
[17] gender_0.5.1 splines_3.3.2 RcppProgress_0.3 munsell_0.4.3
[21] broom_0.4.2 janeaustenr_0.1.4 mnormt_1.5-5 reports_0.1.4
[25] tibble_1.2 gridExtra_2.2.1 codetools_0.2-15 matrixStats_0.51.0
[29] lda_1.4.2 SnowballC_0.5.1 openNLP_0.2-6 grid_3.3.2
[33] nlme_3.1-131 gtable_0.2.0 DBI_0.5-1 magrittr_1.5
[37] tokenizers_0.1.4 scales_0.4.1 RcppParallel_4.3.20 xlsx_0.5.7
[41] stringi_1.1.2 reshape2_1.4.2 openNLPdata_1.5.3-2 ca_0.70
[45] venneuler_1.1-0 fastmatch_1.1-0 wordcloud_2.5 iterators_1.0.8
[49] tools_3.3.2 plotrix_3.6-4 parallel_3.3.2 colorspace_1.3-2
[53] rJava_0.9-8

[kbenoit]

OK, you're using the most current quanteda alright. Can you try to provide enough material for us to reproduce the error? A link to your corpus, or a subset of it that causes the problem?

[Ninoninoninonino]

I just installed the latest version of Quanteda via install.packages("quanteda", type = "source")

The TM Corpus file is here: https://ufile.io/38cc8

My code is here:

# libraries
library(RCurl)
library(XML)
library(stringr)
library(qdap)
library(tm)
library(quanteda)
library(data.table)
library(stm)
library(cldr)
library(RNewsflow)
library(textreuse)
library(dplyr)
library(tidytext)

# load corpus object from https://ufile.io/38cc8

# convert TM corpus file into Quanteda corpus file
All.Corpora <- corpus(dtm)

Error message:

Error in data.frame(texts = rep(NA, length(x)), row.names = names(x), : duplicate row.names: 0001.html.xtr, 0002.html.xtr, 0003.html.xtr, 0004.html.xtr, 0005.html.xtr, 0006.html.xtr, 0007.html.xtr, 0008.html.xtr, 0009.html.xtr, 0010.html.xtr, 0011.html.xtr, 0012.html.xtr, 0013.html.xtr, 0014.html.xtr, 0015.html.xtr, 0016.html.xtr, 0017.html.xtr, 0018.html.xtr, 0019.html.xtr, 0020.html.xtr, 0021.html.xtr, 0022.html.xtr, 0023.html.xtr, 0024.html.xtr, 0025.html.xtr, 0026.html.xtr, 0027.html.xtr, 0028.html.xtr, 0029.html.xtr, 0030.html.xtr, 0031.html.xtr, 0032.html.xtr, 0033.html.xtr, 0034.html.xtr, 0035.html.xtr, 0036.html.xtr, 0037.html.xtr, 0038.html.xtr, 0039.html.xtr, 0040.html.xtr, 0041.html.xtr, 0042.html.xtr, 0043.html.xtr, 0044.html.xtr, 0045.html.xtr, 0046.html.xtr, 0047.html.xtr, 0048.html.xtr, 0049.html.xtr, 0050.html.xtr, 0051.html.xtr, 0052.html.xtr, 0053.html.xtr, 0054.html.xtr, 0055.html.xtr, 0056.html.xtr, 0057.html.xtr, 0058.html.xtr, 0059.html.xtr, 0060.html.xtr, 0061.html.xtr, 0062.html.xtr, 0063.html.xtr, 0064.html.xtr, 0065.html

[kbenoit]

I've tested this now, and it's an error caused by some of the documents in your corpus having the same document names. This causes an error in quanteda because document names must be unique (since the heart of the corpus structure is a data.frame), but does not create an error in tm because its core structure is a list).

In R, a list can have non-unique names for its elements:

> (tmp <- list(one = 1:3, two = 4:5, one = letters[1:3]))
$one
[1] 1 2 3

$two
[1] 4 5

$one
[1] "a" "b" "c"

> tmp["one"]
$one
[1] 1 2 3

So we need to ensure that upon conversion, data.frame row names are unique.

[Ninoninoninonino]

I didn't have a problem when I combined different Quanteda Corpora with each containing identical file names ..

[Ninoninoninonino]

Also, I didn't know how to access these names. Otherwise I could have renamed them in TM ..

[kbenoit]

Should be fixed in the v09.9.36.

Don't worry about not knowing how to access the document names in a tm corpus. The structure of a tm corpus is pretty complicated. But you might ask why the corpus has duplicated document names.

[shmuhammadd]

I had exact same issue today when I try to merge two corpus:Error: Cannot combine corpora with duplicated document names

[kbenoit]

@shmuhammad2004 can you provide a reproducible example? Perhaps a subset of your corpus where the names are identical?