Convert SEXP to XPtr cause lost of tag value

[Marlin-Na]

Here is a minimal example:

#include <Rcpp.h>
using namespace Rcpp;

typedef struct {
    int x;
    int y;
} mycomplex;

void myfinalizer(mycomplex* x) {
    free(x);
}

typedef XPtr<mycomplex, Rcpp::PreserveStorage, myfinalizer> XPtrMyComplex;

// [[Rcpp::export]]
SEXP init(int a, int b) {
    mycomplex* m = (mycomplex*) malloc(sizeof(mycomplex));
    m->x = a;
    m->y = b;
    XPtrMyComplex ans = XPtrMyComplex(m, true, Rf_ScalarRaw(42), R_NilValue);
    return ans;
}

// [[Rcpp::export]]
int access(SEXP robj) {
    XPtrMyComplex obj = as<XPtrMyComplex>(robj);
    mycomplex* p = obj.checked_get();
    return p->x + p->y;
}


/*** R
library(xptr)
x <- init(3, 4)
print(xptr_tag(x))
access(x)
print(xptr_tag(x))
*/

I get:

R> library(xptr)

R> x <- init(3, 4)

R> print(xptr_tag(x))
[1] 2a

R> access(x)
[1] 7

R> print(xptr_tag(x))
NULL

I am not sure if this behavior is expected. Also, I would appreciate if anyone can explain to me the semantics when doing this conversion: is a new external pointer object allocated or it is just wrapper around the original one?

Thanks!

[eddelbuettel]

This looks like it wanted to be filed against the xptr package by @randy3k rather than against Rcpp. I am not familiar with package xptr and either its tags or access() function -- sorry.

[Marlin-Na]

@eddelbuettel Not really, xptr::xptr_tag was really just a wrapper of R_ExternalPtrTag, access was the function written in Rcpp. I have replaced xptr::xptr_tag with a C function below and it essentially has the same output.

#include <Rcpp.h>
using namespace Rcpp;

typedef struct {
    int x;
    int y;
} mycomplex;

void myfinalizer(mycomplex* x) {
    free(x);
}

typedef XPtr<mycomplex, Rcpp::PreserveStorage, myfinalizer> XPtrMyComplex;

// [[Rcpp::export]]
SEXP init(int a, int b) {
    mycomplex* m = (mycomplex*) malloc(sizeof(mycomplex));
    m->x = a;
    m->y = b;
    XPtrMyComplex ans = XPtrMyComplex(m, true, Rf_ScalarRaw(42), R_NilValue);
    return ans;
}

// [[Rcpp::export]]
int access(SEXP robj) {
    XPtrMyComplex obj = as<XPtrMyComplex>(robj);
    mycomplex* p = obj.checked_get();
    return p->x + p->y;
}

// [[Rcpp::export]]
SEXP get_tag(SEXP x) {
    return R_ExternalPtrTag(x);
}


/*** R
x <- init(3, 4)
print(get_tag(x))
access(x)
print(get_tag(x))
*/

According to Writing R Extensions:

An external pointer is created by SEXP R_MakeExternalPtr(void *p, SEXP tag, SEXP prot); where p is the pointer (and hence this cannot portably be a function pointer), and tag and prot are references to ordinary R objects which will remain in existence (be protected from garbage collection) for the lifetime of the external pointer object. A useful convention is to use the tag field for some form of type identification and the prot field for protecting the memory that the external pointer represents, if that memory is allocated from the R heap. Both tag and prot can be R_NilValue, and often are.

The problem is that the tag value of the external pointer is lost with a SEXP -> XPtr conversion. Note that if I replace

XPtrMyComplex obj = as<XPtrMyComplex>(robj);
mycomplex* p = obj.checked_get();

with

mycomplex* p = (mycomplex*) R_ExternalPtrAddr(robj);

Then the tag value will not be lost.

[eddelbuettel]

Ok -- it is always possib;e that there is a bug, or missing functionality, somewhere -- but XPtr functionality is not new and used somewhat. So maybe you try to find other use cases and see if tags are deployed elsewhere.

[Marlin-Na]

Yeah, just want to report this since it took me quite some time to track down the issue. I am currently avoiding this conversion by using R_ExternalPtrTag so it was fine.

For notes, I am using this tag value for identifying the pointer type. I have several different pointer types and they are wrapped into different S4 classes. However I want to have faster identification in C/C++ via the tag value instead of via S4::is

[nbenn]

I just ran into the same issue myself. I don't think it's a bug, but rather a design choice that in combination with the example code floating around (function pointer example) causes an issue for people who use tags with Rcpp::Xptrs.

Usage of Xptr as shown in examples boils down to:

1. create an Xptr

   // [[Rcpp::export]]
   XPtr<someObj> create()
   {
     return XPtr<someObj>(new someObj());
   }

2. use an Xptr

   // [[Rcpp::export]]
   void use(SEXP xpsexp)
   {
       XPtr<someObj> xp(xpsexp);
      //... do stuff with xp
   }

If we look at the two constructors involved,

XPtr(SEXP x, SEXP tag = R_NilValue, SEXP prot = R_NilValue);
XPtr(T *p, bool set_delete_finalizer = true, SEXP tag = R_NilValue, SEXP prot = R_NilValue);

I suspect that for the use() instantiation, the tag simply gets overwritten by R_NilValue.

The problem goes away if we instantiate for example as

XPtr<someObj> xp(xpsexp, R_ExternalPtrTag(xpsexp));

@eddelbuettel can you give some guidance for the intended use of Xptr in such a scenario? For a function receiving an Xptr as SEXP object, wouldn't it make more sense for the constructor to try and honor previously set tag and prot values?

[nbenn]

For the time being my solution is a function

template <typename T>
Rcpp::XPtr<T> xptr(SEXP x)
{
  return Rcpp::XPtr<T>(x, R_ExternalPtrTag(x), R_ExternalPtrProtected(x));
}

which I then use as

// [[Rcpp::export]]
void use(SEXP xpsexp)
{
  auto xp = xptr<someObj>(xpsexp);
  //... do stuff with xp
}

Does anyone see potential problems with that?

[eddelbuettel]

@nbenn Maybe you want to submit a new example to the Rcpp Gallery?

[nbenn]

@eddelbuettel I'm sorry for my multi-week delayed answer. I'm happy to share what have been learning about using external pointers, tags, etc. in conjunction with Rcpp via an Rcpp Gallery article. I'm currently using all of this for a small project of mine and if things work out, I can write something up in a couple of weeks time. If you want to, I can contact you via your gh email address, as soon as I feel ready and we can quickly discuss scope.

[stephematician]

For the time being my solution is a function

template <typename T>
Rcpp::XPtr<T> xptr(SEXP x)
{
  return Rcpp::XPtr<T>(x, R_ExternalPtrTag(x), R_ExternalPtrProtected(x));
}

That the default behaviour of the constructor is to modify the SEXP does seem more bug than feature; as a user I would expect the XPtr constructor to (by default) do what you've implemented in xptr(SEXP)

[eddelbuettel]

Well maybe someone with a bit of time and patience wants to dive into the XPtr code and fix or correct its constructor?