Functions fail with vectors/matrices longer than .Machine$integer.max

[kendonB]

I get the error negative length vectors are not allowed.

I've seen this talked about elsewhere on the internet, but can't find an issue here. Is it being worked on?

Thanks so much!

[eddelbuettel]

Code example, please.

Rcpp supports XLENGTH so maybe you just made the mistake of representing a number that is 'too large for integer' in an integer.

[kendonB]

Apologies for the lack of code example. Here it is:

#include <Rcpp.h>
using namespace Rcpp;

// [[Rcpp::export]]
NumericMatrix plus1(NumericVector x){

  std::size_t nrow = x.size(), ncol = 1;
  NumericMatrix out(nrow, ncol);
  for (std::size_t i = 0; i < nrow; ++i) {
    for (std::size_t j = 0; j < ncol; ++j) {
      out(i, j) = x[i] + 1;
    }
  }
  return out;
}

I sourceCpp the above. Then the Rscript:

stop("Need around 40GB")
tmp <- 1:(.Machine$integer.max + 1)*0 + 1

# works
plus1(head(tmp))
     [,1]
[1,]    2
[2,]    2
[3,]    2
[4,]    2
[5,]    2
[6,]    2

tmp2 <- plus1(tmp)
# doesn't work
# Error in .Primitive(".Call")(<pointer: 0x000000006c501de0>, x) : 
#  negative length vectors are not allowed

[thirdwing]

Personally I really think it is expected behavior.

You exceed the max length of vectors in R.

[kendonB]

I don't think this is correct. https://cran.r-project.org/bin/windows/base/old/3.0.0/NEWS.R-3.0.0.html

[eddelbuettel]

Right, and @thirdwing is actually the gentlemen who brought that very change to Rcpp. But IIRC it was also documented that your matrix elements could not be longer than 2^31-1. In a nutshell, a matrix (which is internally a vector) can have more than 2^31-1 elements provided both of its indices remain below the limit. Which your matrix out above violates.

[eddelbuettel]

Same NEWS file, just below:

    * There is some support for matrices and arrays with each dimension
      less than 2^31 but total number of elements more than that.  Only
      some aspects of matrix algebra work for such matrices, often
      taking a very long time.  In other cases the underlying Fortran
      code has an unstated restriction (as was found for complex
      svd()).

To be fair looking for '31' in that file shows some more recent changes but also clearly states you need doubles for indexing.

[kendonB]

First, thanks so much both of you for the huge contribution to open source that is Rcpp. I know it's often pretty thankless!

An update on what I can get to work.

• As suggested, big matrices work fine as output as long as the indices are less than 2^31 - 1 (plus1To10Mat).
• Big matrices also work fine as inputs as long as the indices are less than 2^31 - 1 (plus1Mat).
• Big vectors seem to work fine as inputs, but not outputs (see plus1Vec the initializer looks like it uses an int for the length input). A workaround is to output a NumericMatrix in this case (plus1VecMat). This is the only aspect I see as an issue with the package, unless there's an easy fix.

#include <Rcpp.h>
using namespace Rcpp;

// [[Rcpp::export]]
NumericVector plus1Vec(NumericVector x){
  R_xlen_t nrow = ::Rf_xlength(x);
  NumericVector out(nrow);
  for (R_xlen_t i = 0; i < nrow; ++i) {
      out[i] = x[i] + 1.0;
  }
  return out;
}

// [[Rcpp::export]]
NumericMatrix plus1VecMat(NumericVector x){
  R_xlen_t nrow = ::Rf_xlength(x) / 2;
  NumericMatrix out(nrow, 2);
  for (int i = 0; i < nrow; ++i) {
    for (int j = 0; j < 2; ++j) {
      out(i, j) = x[i + j*nrow] + 1.0;
    }
  }
  return out;
}

// [[Rcpp::export]]
NumericMatrix plus1Mat(NumericMatrix x){
  int nrow = x.nrow();
  int ncol = x.ncol();
  NumericMatrix out(nrow, ncol);
  for (int i = 0; i < nrow; ++i) {
    for (int j = 0; j < ncol; ++j) {
      out(i, j) = x(i, j) + 1.0;
    }
  }
  return out;
}

// [[Rcpp::export]]
// This one is more similar to my real world application
NumericMatrix plus1To10Mat(NumericVector x){
  R_xlen_t nrow = ::Rf_xlength(x);
  NumericMatrix out(nrow, 10);
  for (int i = 0; i < nrow; ++i) {
    for (int j = 0; j < 10; ++j) {
      out(i, j) = x[i] + 1.0 + j;
    }
  }
  return out;
}

And the R-script:

stop("Need around 40GB")
tmpVec <- 1:(.Machine$integer.max + 2)*0 + 1

# works
plus1Vec(head(tmpVec))

# Doesn't work
tmp2 <- plus1Vec(tmpVec)
# Error in .Primitive(".Call")(<pointer: 0x000000006c5030e0>, x) : 
# negative length vectors are not allowed
# But workaround is to output a NumericMatrix (works)
tmp2a <- plus1VecMat(tmpVec)
rm(tmp2a);gc()

tmpVecShort <- tmpVec[1:(ceiling(.Machine$integer.max / 10) + 1)]

tmpMat <- matrix(tmpVec, ncol = 2)
# Warning message:
# In matrix(tmpVec, ncol = 2) :
#   data length [-2147483647] is not a sub-multiple or multiple of the number of rows [1073741825]

# > tmpMat[1073741825, 2]
# Error: long vectors not supported yet: subset.c:744

# atomic subsetting works in R
# > tmpMat[.Machine$integer.max + 2]
# [1] 1
rm(tmpVec);gc()

# works
tmp3 <- plus1Mat(tmpMat)
rm(tmp3, tmpMat);gc()

# works
tmp4 <- plus1To10Mat(tmpVecShort)
rm(tmp4, tmpVecShort);gc()

[eddelbuettel]

As they say: works for me:

R> Rcpp::sourceCpp('/tmp/bigmem.cpp')

R> #stop("Need around 40GB")
R> tmpVec <- 1:(.Machine$integer.max + 2)*0 + 1

R> # works
R> plus1Vec(head(tmpVec))
[1] 2 2 2 2 2 2

R> # Doesn't work
R> tmp2 <- plus1Vec(tmpVec)
R> 

That is on a pretty absurd machine at work with 256gb of ram -- so I could get a large enough contiguous chunk of memory.

R> object.size(tmpVec)
17179869232 bytes
R> object.size(tmp2)
17179869232 bytes
R> 

[kendonB]

@eddelbuettel Can you post your sessionInfo()? Here is mine:

R version 3.2.3 (2015-12-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server 2008 R2 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Rcpp_0.12.4.3       RevoUtilsMath_3.2.3

loaded via a namespace (and not attached):
[1] tools_3.2.3

Note that I get the same result when using the normal version of R, and my machine has plenty of memory (192 GB).

[eddelbuettel]

Sure. It was in a old session via RStudio Server which is why there is ragtag mix of packages attached.

But judging from the version string it since pivoted to an updated R version. Fine. Note that I XXXX'ed out a few non-public in-house packages from work. The remainder should not matter.

R 3.2.4-revised. Rcpp 0.12.3.

Edit: I restarted. Here is a more minimal session. Still works of course.

R> sessionInfo()
R version 3.2.4 Revised (2016-03-16 r70336)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 15.10

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] tools_3.2.4    Rcpp_0.12.3    fortunes_1.5-2
R> 

fortunes is in there because my ~/.Rprofile draws from it on interactive sessions...

[kendonB]

I assume it's unlikely that the slightly older tools package, newer Rcpp package, or the change in R version can account for the difference in behavior?

Do you think it's a problem with using Windows?

[eddelbuettel]

I have something in the very back of my mind that may relate it to size_t but I have no reference.

How much RAM does that Windows computer have?

[kendonB]

192GB.

[eddelbuettel]

Yes, I guess I would blame Windows then.

If you are really motivated you could recast your examples in plain C again the C API of R and look into a bug report against R if they fail -- or conversely come back to us if they work because it is then a Rcpp-on-Windows issue.

[kendonB]

Any guidance on how to "recast your examples in plain C again the C API of R and look into a bug report against R" would be very helpful!