Support SciPy sparse matrices to support reticulate?

[eddelbuettel]

The reticulate package is pretty exciting as it gives us R access to underlying Python code, and is already used by the R packages for keras and tensorflow.

Over there, a discussion started about sparse matrices.

Would it be feasible to cover one or more use cases of bringing SciPy sparse matrices to R?

[binxiangni]

I write a small program to convert SciPy sparse matrix to R. But where should I add it to? To RcppArmadillo? I have no idea how to put it in. Or should I create a new package like sparsio you mentioned before?

[eddelbuettel]

Nice!

There are a few things we could do:

• keep it in RcppArmadillo if you feel it has the best match and use there
  • have RcppArmadillo add a "Siuggests: reticulate"
  • have the script use requireNamespace("reticulate", quietlty=TRUE) to operate conditionally
  • that way the function for scipy import could be a regular function in RcppArmadillo
• make it an external function or package
  • this made sense for sparsio because it also needed extra code

Which reminds me: I meant to get back to the sparsio author and ask if we could have an additonal function or option to just import X but not y -- as I see it, it always wants to import features (X) and prediction (y). So poking @dselivanov: Is that something we could get sparsio to do?

[dselivanov]

I personally think that most sense it will be to include such conversions to reticulate itself (Since it covers most common r - python data structures conversions).

[eddelbuettel]

Could, on the other hand reticulate is a poor "connection" package free of application use.

What about my second question to use: sparsio to read just a feature matrix X, not left-hand side variable?

[dselivanov]

Why not. We read/write target only because of definition of the svmlight format.
But still I wouldn't consider basic linear algebra stuff as application... Any sparse matrix is just 3 arrays. @jjallaire any thoughts?

[eddelbuettel]

Yes, a sparse matrix's internal representation is just three vectors, and @binxiangni has done most excellent work over this Google Summer of Code adding more and more converters to RcppArmadillo.

What would help further, I thought, was to use your sparsio to read more matrices to have more conversion tests. When I looked at sparsio it could only be for sparse ML examples with both a left and a right-hand side. So what I am asking you has nothing to do with reticulate: I am simply wondering if we could use sparsio for a little more 'i' of sparse matrices...

[dselivanov]

Ok, I got the point. By default sparsio writes dummy y = 0, so it can be considered as optional. But by the definition of svmlight target variable always should be there.

Another widespread format for storing just sparse matrices (without target) is MatrixMarket - but it is not insparsio (yet).

[eddelbuettel]

This is now merged.

[dselivanov]

Still don't understand why you have decided to put it into RcppArmadillo...

[eddelbuettel]

What is "it" ?

If you wonder why SciPy sparse matrice support, well that is what the thread started with. And as optional component it makes perfect sense.

[dselivanov]

Scipy sparse matrices conversion with reticulate.

[eddelbuettel]

Well, it is our package and it made sense to us.

We are on the other hand still waiting on you and sparsio :)

[dselivanov]

Don't get me wrong, I totally agree that one can do whatever he want with own package. But mb really put such not related to Armadillo functionality into separate package (something aimed to provide utils for sparse matrices)? I have in mind couple more cases - subsetting csr matrices without conversion to csc, faster sparse * dense matrix multiplications, etc. Not sure where is the right place to discuss.

[eddelbuettel]

That's of course all possible. But to recap:

• Binxiang has done a metric ton of work to improve Armadillo sparse matrix support in RcppArmadillo
• We had some issues / potential issues with Armadillo being non-standard (was a bug, now fixed)
• Having another source of sparse matrices was beneficial for testing; Python is widely used; reticulate is ready.
• So we added this. It's still tiny: one small (currently unexported) helper function, a set of unit tests.

I see absolutely nothing wrong with that. We reached out to for help with sparse matrices via sparsio; I am still hopeful we can do that too. In the meantime we moved on with our package.

Should be on CRAN "soon".

[dselivanov]

Seems I again (somehow) lost the fact that scipy conversion feature was implemented mainly for tests. In such case it makes much more sense. Thanks for explanation.