Rcpp is not reverse compatible

[XiangdongGu]

The newer versions of Rcpp (0.12.16 and 0.12.17) broke my codes that uses Rcpp (no error raise but the results are not right). I don't know why but once I reverse to 0.12.15 everything is working. I am not sure how to fix it. Is there any general change we should make for newer versions?

[eddelbuettel]

See the rcpp-logs repo -- we run THOUSANDS of tests before each release, and CRAN also tests incoming package against all know dependencies.

So, this is neither expected nor known. It would be polite of you to actually back up such a claim with a [minimally reproducible example[(https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) -- because otherwise it is just hearsay and nothing we can do anything about.

[XiangdongGu]

Thank you @eddelbuettel ! My code was complex, so I extract the problem part and wrap into a smaller code.

Here is the cpp file

#include <Rcpp.h>
using namespace Rcpp;

// [[Rcpp::export]]
double loglik(NumericVector y, NumericMatrix X, NumericVector beta) {
  int J = X.ncol(), n = X.nrow();
  double result = 0.0, temp;
  for (int i=0; i < n; i++) {
    temp = 0;
    for (int j=0; j < J; j++) {
      temp += X(i, j) * beta[j];
    }
    result += (y[i] - temp)*(y[i] - temp);
  }
  return result;  
}

// [[Rcpp::export]]
NumericVector gradlik(NumericVector y, NumericMatrix X, NumericVector beta) {
  int J = X.ncol(), n = X.nrow(), i, j;
  double temp;
  NumericVector result(J);
  for (i=0; i<n; i++) {
    temp = 0;
    for (j=0; j<J; j++) {
      temp += X(i, j) * beta[j];
    }
    temp += - y[i];
    for (j=0; j<J; j++) {
      result[j] += 2* X(i, j) * temp;
    }
  }
  return result;
}

// [[Rcpp::export]]
void test(NumericMatrix X, NumericVector y, Function fitfun) {
  RNGScope scope;
  int J = X.ncol(), i;
  NumericVector par(J);
  for (i=0; i<J; i++) par[i] = 0;
  for (int i=0; i<10; i++) {
    Rprintf("%8f\n", log(R::runif(0.0, 1.0)));
    par = fitfun(par, y, X);
  }
}

Here is the R script

source("rcpp_new/.Rprofile", chdir = TRUE)
#source("rcpp_old/.Rprofile", chdir = TRUE)
library(Rcpp)
sourceCpp("test.cpp")

# Generate Test Data-----------------------------------------------------------
set.seed(123)
x <- rnorm(10)
y <- x + rnorm(10, sd = 0.1)
X <- cbind(1, x)

# Optimization function--------------------------------------------------------
fitfun <- function(parm, y, X) {
  q <- optim(parm, loglik, gradlik, method="BFGS", X = X, y = y)
  q$par
}

# New version Rcpp: the random number never change
# Old version Rcpp: the random number will change
test(X, y, fitfun)

My project code was to do some bayesian analysis with optimization routine (e.g. optimize quadratic function as an illustration here). So the random number generation is essential here. In old version of Rcpp (< 0.12.15), the random function will generate different values, however in newer version of Rcpp, it always gives same random number. This problem only happens when I put random function together with optimization function in the same cpp function as shown in test function in the code.

I am not familiar C++, so maybe my usage of C++ is not standard, but it would be helpful to understand what is causing this so I can modify my code to avoid any future failure.

Thanks for the help!

[eddelbuettel]

So now we have something concrete, and know what we are talking about -- RNG state.

And YES that did change, and YES that was on purpose and YES it in the NEWS and ChangeLog.

Better news yes: we noticed the problem, and the development sources here have a fix. So please see PR #862 as well the example use in PR eddelbuettel/rcppde#15 over in RcppDE where it fixes eddelbuettel/rcppde#14.

Those, as well as the older PR #825 here addressing #823 for the original change motivation.

All this should fix your issue. And if I may, had your code been in a CRAN package with a corresponding unit test, then we would have know right at the start. So sorry for the inconvenience but there was no feedback loop back to us. This fixed a problem, we are sorry about side effects we did not know about.

[eddelbuettel]

BTW RNGScope scope; should not be needed; Rcpp adds that for you already. But look into the new SuspendRNGSynchronizationScope and how RcppDE uses it.

[XiangdongGu]

Thank you for @eddelbuettel for the clarification! I have been using old versions and waiting for new release for the fix. Recently, I have updated my R to version 3.5.1, and reinstalled all the packages including Rcpp (0.12.19). Unfortunately, the issues persist that it always gives same random number. I was trying to downgrade to older version, e.g. 0.12.13, it throws error of

In file included from Date.cpp:31:
In file included from ../inst/include/Rcpp.h:77:
../inst/include/Rcpp/Rmath.h:222:55: error: no member named 'Rf_pythag' in the global namespace; did you mean 'pythag'?
    inline double pythag(double a, double b)    { return ::Rf_pythag(a, b); }
                                                         ^~~~~~~~~~~
                                                         pythag
../inst/include/Rcpp/Rmath.h:222:19: note: 'pythag' declared here
    inline double pythag(double a, double b)    { return ::Rf_pythag(a, b); }
                  ^
1 error generated.
make: *** [Date.o] Error 1
ERROR: compilation failed for package ‘Rcpp’
* removing ‘/Volumes/Bayesian-64G/icensmis_packrat/packrat/lib/x86_64-apple-darwin15.6.0/3.5.1/Rcpp’

Really appreciate your help!

[eddelbuettel]

That error comes from

• mixing an OLD version of Rcpp (which still uses Rf_pythag), here 0.12.13
• with a new verion of R (which no longer has it).

That is your error in mixing, and your problem, caused the by the rare change in R -- they removed that symbol.

CRAN and the rest of the world happily use matching (ie "current") versions. We also just went through all of this in issue #915 which you could read in detail.

In short, you need to analyze the claim "Rcpp broke my code" and change your package to work with R 3.5.1. Or, if you insist to work with your old packages, then you either

• must use an old R that was current at the time that Rcpp version was release
• update the old Rcpp you insist on using by not relying on Rf_pythag.

[XiangdongGu]

Thank you for the explanation and point me to the issue! For the random number issue, I tried to add GetRNGstate() and PutRNGstate() for each call of random function, and it appears to be working.

It is just a little confusing to me that if my input function (fitfun in my above example) is regular function without involving optim, then the random state is changing as expected. But if the function involves optim, then the random number is not moving. This seems to be happening only for newer Rcpp versions.

But I think GetRNGstate() and PutRNGstate() solve my problem. I can use all new versions of R and Rcpp. Thanks for the great package and help!

[eddelbuettel]

If you use Rcpp Attributes (which has been available and mature for well over five years now) then that is taken care of automatically for you.

Please see e.g. our 2018 introduction paper in TAS that is also included as a vignette in case you do not have easy access to journals. It shows you how to access the R RNG easily, and reliably.

[XiangdongGu]

Hi @eddelbuettel,

After a little more investigation of my issue, I found that in the following case the RNG state is not changing: 1. the cpp function has Function type argument 2. the passed function called a cpp function.

I am illustrating a simple example as following

library(Rcpp)

sourceCpp(
  code  = '
  #include <Rcpp.h>
  using namespace Rcpp;

  // [[Rcpp::export]]
  double f(double x) {
    return x;
  }
  
  // [[Rcpp::export]]
  void test(Function myfun) {
    NumericVector y;
    for (int i = 0; i < 10; i++) {
       Rprintf("%8f\\n", runif(1)[0]);
       y = myfun(1);
    }
  }
  '
)

mfun <- function(x) f(1)

test(mfun)

In this example, mfun is R function and it calls a function defined in cpp. mfun is passed to test. In this case, the random number generated is no long changing. If I pass a function that does not call cpp function such as rnorm, then everything is fine.

I am not sure if this is known, but I would appreciate if you can give some suggestion how to best handle this.

Thanks.

[eddelbuettel]

I see. You are calling back and forth between R and C++ which is both a little unusual and rare. We did make changes in that area because we concluded that jumping back and forth like this is not the situation for which the RNG state preservations was designed.

This was discussed here in older issue tickets so please do read the succint release notes in NEWS.Rd where you will see that this was an area of work in release 0.12.18, 0.12.17 and 0.12.16 -- and each entry gives you a link to the discussion in the corresponding issue ticket. We do think the change is for the better.

In particular, see the discussion in #862 which should allow you to obtain the prior behaviour.

[byapparov]

I have replaced Rcpp section in Packrat file to this when switching from 3.3.3 to 3.5.3:

Package: Rcpp
Source: CRAN
Version: 1.0.2
Hash: d04e441a8d398e3d3a71f294c07fa94d

[eddelbuettel]

Yes, it is common practice to use current R versions (which, by the way, is now R 3.6.1) with current CRAN package versions which is after all how they are tested and quality-controlled at CRAN.

So I recommend not only to use the current Rcpp version, but also the current R version. Note that changing R versions also on occassion requires reinstallation of packages -- as documented in the release notes for R.

[byapparov]

I don't agree with the latest version recommendation.

Each team is free to decide when and why to upgrade. This is why packrat is so important.

[eddelbuettel]

This is not a useful discussion for Rcpp, so I am going to lock this. I am glad Rcpp works for you, we put a lot of effort into it.