Creating private$.work using compiled code prevents the ability to parallelize osqp in R

[mihaiconstantin]

I do not want to crosspost, so I would rather provide a summary here and link to the StackOverflow post that lead to this issue.

The goal is to be able to parallelize osqp, by creating an osqp object and passing it to several workers that can call methods on that object instance.

Take the following scenario:

(function() {
    # Create cluster.
    cluster <- parallel::makePSOCKcluster(parallel::detectCores() - 1)

    # Stop cluster.
    on.exit(parallel::stopCluster(cluster))

    # Bare minimum data.
    x <- matrix(rnorm(100), 10, 10)
    y <- runif(10)

    # The 'osqp' object is created in the main process.
    model <- osqp::osqp(P = crossprod(x), q = -crossprod(x, y), pars = list(verbose = FALSE))

    # Run operation.
    result <- parallel::parSapply(cluster, c(1), function(i) {
        # Calling the solver on the worker process.
        return(model$Solve()$x)
    })

    # Inspect result.
    print(result)
})()

The code above results in an error:

Error in checkForRemoteErrors(val) : 
  one node produced an error: external pointer is not valid

It appears that environment(model$Solve) contains a private environment that contains an externalptr object .work:

typeof(model$.__enclos_env__$private$.work)  ​
# "externalptr"

This pointer .work is created using compiled code, i.e., via an Rcpp export:

osqp-r/src/RcppExports.cpp

Lines 8 to 23 in c38b1de

	// osqpSetup
	SEXP osqpSetup(const S4& P, const NumericVector& q, const S4& A, const NumericVector& l, const NumericVector& u, const List& pars);
	RcppExport SEXP _osqp_osqpSetup(SEXP PSEXP, SEXP qSEXP, SEXP ASEXP, SEXP lSEXP, SEXP uSEXP, SEXP parsSEXP) {
	BEGIN_RCPP
	Rcpp::RObject rcpp_result_gen;
	Rcpp::RNGScope rcpp_rngScope_gen;
	Rcpp::traits::input_parameter< const S4& >::type P(PSEXP);
	Rcpp::traits::input_parameter< const NumericVector& >::type q(qSEXP);
	Rcpp::traits::input_parameter< const S4& >::type A(ASEXP);
	Rcpp::traits::input_parameter< const NumericVector& >::type l(lSEXP);
	Rcpp::traits::input_parameter< const NumericVector& >::type u(uSEXP);
	Rcpp::traits::input_parameter< const List& >::type pars(parsSEXP);
	rcpp_result_gen = Rcpp::wrap(osqpSetup(P, q, A, l, u, pars));
	return rcpp_result_gen;
	END_RCPP
	}

It seems that this pointer is managed by compiled code and, as such, cannot be used within the workers. It is fine to call the compiled code and create this pointer from within the workers. What is not fine is to create this pointer in another process (i.e., the main process) and then pass it to the worker processes. This is probably because each worker is created as a separate R process, with its own memory space.

This is very limiting for the cases in which one has no control over the osqp object creation and is only allowed to call methods on an already provided object. Perhaps it would be an idea to not create .work using Rccp or, if that is not possible to provide a deep clone method to somehow copy the data at that pointer.

[bnaras]

Perhaps I'm missing something. Why would you not create the osqp model in each worker? In other words, why would all parallel R processes solve the same problem?