std::bad_alloc when dfm() is applied to a large number of documents

[geethapriyakrish]

Hi,
I applied quanteda::dfm on a list of tokens of size 50000 and it works.

> grams[[1]] = quanteda::dfm(tokens[1:50000], ngrams = 1) --> works
> grams[[1]] = quanteda::dfm(tokens[50000:100000], ngrams = 1) --> works

When I apply it on larger list of tokens, it fails.
> grams[[1]] = quanteda::dfm(tokens[1:100000], ngrams = 1) --> throws following error

Error in qatd_cpp_tokens_ngrams(x, types(x), concatenator, n, skip + 1) : 
  std::bad_alloc

Please suggest me a solution. I would like to apply it on a list of size 500k

R version: 3.5.1
quanteda version: 1.3.3
OS version: Ubuntu 16.04.4

[kbenoit]

What happens if you omit the ngrams = 1 argument entirely? In other words,

temp <- dfm(tokens[1:100000])

Also what's the output of

summary(ntoken(tokens[1:100000]))

(this will tell us the rough distribution of the sizes in token length of your 100k documents)

[geethapriyakrish]

temp <- dfm(tokens[1:100000]) works and generates dfm with ngrams=1.
The output of summary(ntoken(tokens[1:100000])) is

 Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
     0.0     87.0    169.0    357.6    344.0 129849.0 

I am also in need of creating a dfm with ngrams=2.
But temp <- dfm(tokens[1:100000],ngrams=2) fails by throwing the same error.

[kbenoit]

Thanks. You have mostly very short documents but the largest are large (130k tokens).

@koheiw two things here:

1. We only need to call the C++ ngrams if ngrams > 1 or skip > 1 (and maybe not even if ngrams = 1 and skip = 0). We are seeing the error here even though the C++ function doesn’t need to do anything it is triggered by the ngrams = 1 even though this is the default.
2. Can we Improve the robustness of the C++ code to deal better with this issue?

[kbenoit]

Thanks. You have mostly very short documents but the largest are large (130k tokens). Some of documents are empty though, as the min token length is 0.

@koheiw two things here:

1. We only need to call the C++ ngrams if ngrams > 1 or skip > 1 (and maybe not even if ngrams = 1 and skip = 0). We are seeing the error here even though the C++ function doesn’t need to do anything it is triggered by the ngrams = 1 even though this is the default.
2. Can we Improve the robustness of the C++ code to deal better with this issue?

[koheiw]

Please share the tokens object with me. Also sessionInfo().

[geethapriyakrish]

quanteda::dfm works for ngrams=1,2 when I apply it on tokens object that is loaded from a RDS in a new session. It throws that error only when I run all lines of my code in a single session.

I guess this issue is due to RAM limitation.

[koheiw]

There is no way to investigate this issue unless you provided required information.

[geethapriyakrish]

tokens object can be downloaded from https://drive.google.com/file/d/11gah5FC_j78GacR6yvD2hkljPCNNDf9M/view?usp=drive_web

[koheiw]

Thanks, but cannot download the file. You can also send a link to my email address if you do not want to share it publicly. We also need you system information that you can get by sessionInfo().

[koheiw]

@geethapriyakrish Please test a version in the test-ngram branch if it solves your problem.