Convert SciPy sparse matrix to R

[binxiangni]

I did some minor modification to support the conversion of SciPy sparse matrix to R. It seems that I mistook the setup in my unit test so that compiler errors occurred. Any hints to correct it?

[eddelbuettel]

Any hints to correct it?

Yes. We now use the reticulate package, but in the (older) Travis setup I still use we have to explicitly tell Travis about it in the .travis.yml file. We may of course also need SciPy and all that. But first step reticulate,

[eddelbuettel]

I get the tests to run if I

• make sure R package reticulate is installed
• Debian/Ubuntu package python-scipy is installed

But one test still fails:

>   coo <- sp$coo_matrix(mat)
>   dgT <- methods::as(M, "dgTMatrix")
>   checkEquals(dgT, py2r(coo), msg="coo2dgt")
Error in checkEquals(dgT, py2r(coo), msg = "coo2dgt") : 
  Attributes: < Component “i”: Mean relative difference: 1.2 >
Attributes: < Component “j”: Mean relative difference: 1.2 >
Attributes: < Component “x”: Mean relative difference: 0.5714286 >
coo2dgt
> 

even though

> py2r(coo)
3 x 3 sparse Matrix of class "dgTMatrix"
          
[1,] 1 . 4
[2,] . . 5
[3,] 2 3 6
> dgT
3 x 3 sparse Matrix of class "dgTMatrix"
          
[1,] 1 . 4
[2,] . . 5
[3,] 2 3 6
> M
     [,1] [,2] [,3]
[1,]    1    0    4
[2,]    0    0    5
[3,]    2    3    6
> 

If I comment the corresponding test out, and make some corrections to DESCRIPTION and NAMESPACE then the package tests.

[binxiangni]

Well, I'm checking what happens in coo2dgt.

> str(py2r(coo))
Formal class 'dgTMatrix' [package "Matrix"] with 6 slots
  ..@ i       : int [1:6] 0 0 1 2 2 2
  ..@ j       : int [1:6] 0 2 2 0 1 2
  ..@ Dim     : int [1:2] 3 3
  ..@ Dimnames:List of 2
  .. ..$ : NULL
  .. ..$ : NULL
  ..@ x       : num [1:6] 1 4 5 2 3 6
  ..@ factors : list()
> str(dgT)
Formal class 'dgTMatrix' [package "Matrix"] with 6 slots
  ..@ i       : int [1:6] 0 2 2 0 1 2
  ..@ j       : int [1:6] 0 0 1 2 2 2
  ..@ Dim     : int [1:2] 3 3
  ..@ Dimnames:List of 2
  .. ..$ : NULL
  .. ..$ : NULL
  ..@ x       : num [1:6] 1 2 3 4 5 6
  ..@ factors : list()

The problem exists because coo is row-oriented, while dgT(created by new) is column-oriented.

[eddelbuettel]

I have a commit pending which "works here" in the sense of

• having uncommented that failing test
• made corrections to NAMESPACE and DESCRIPTION
• made py2r() internal for now
• add python-scipy to Travis

But I fear we may get an old SciPy on the older Travis setup -- so even if it "works for me here" we may not get a working setup at Travis.

[eddelbuettel]

Ok, see this branch started from your PR. It just passed a test run on Travis.

There are still a number of open issues. Let's not rush this.

[binxiangni]

After checking the unit tests for sparse matrix conversion in RcppArmadillo I wrote before, str for dgTMatrix in Matrix might have 3 cases:

1. column-oriented;
2. row-oriented;
3. repeated (i, j) pairs or non-sorted.

Personally, I think the current conversion result for SciPy coo sparse matrix is fine and whether to pass the unit test just depends on the generation method of dgTMatrix and its str. So maybe we can just leave the conversion as is now and pick up another dgTMatrix with the same str in the unit test.

[eddelbuettel]

The three cases are not randomly selected though, are they?

Could we do a weaker comparison like this one:

R> methods::as(M, "dgRMatrix") == methods::as(M, "dgTMatrix")
3 x 3 Matrix of class "lgeMatrix"
     [,1] [,2] [,3]
[1,] TRUE TRUE TRUE
[2,] TRUE TRUE TRUE
[3,] TRUE TRUE TRUE
R>
R> all(methods::as(M, "dgRMatrix") == methods::as(M, "dgTMatrix"))
[1] TRUE
R>  

[binxiangni]

Most dgTMatrix is column-oriented, including created by as and operations; new can create a dgTMatrix by specifying a dgTMatrix's i, j, x, so the str of dgTMatrix created this way can be any one of cases listed above.

Not quite understood the reason for using weaker comparison. Do you mean comparing the non-zero values for different kinds of sparse matrix in this SciPy sparse matrix conversion? like the conversion results of cscmatrix and coomatrix?

[eddelbuettel]

So can you create one that would match what we get via SciPy ?

[binxiangni]

Done. And local tests pass.

[eddelbuettel]

Are you ok with my other changes? Ie py2r() is not exported as it is a) a little specialized and b) has not manual page yet.

We can leave things the way they are now, or set this up as a demo/ example instead. No strong feelings either way.

[binxiangni]

Yes, that's fine.

[coatless]

This should be removed.

[eddelbuettel]

Done. Also did some gentle renaming as py2r as bit too generic.

[eddelbuettel]

@binxiangni Can you rebase / pull one more time to get my last commit over here too?

Otherwise I can merge as is and then PR from my branch to just get its final commit over.

[binxiangni]

Done. Sorry for the late action. Still in China, so there is a time difference.

[eddelbuettel]

Oh, I was unaware you were away....