Skip to content

/ RcppArmadillo

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible memory leak with arma::sp_mat within Rcpp::List #89

Closed
gvegayon opened this issue May 3, 2016 · 6 comments
Closed

Possible memory leak with arma::sp_mat within Rcpp::List #89

gvegayon opened this issue May 3, 2016 · 6 comments

Comments

@gvegayon
Copy link
Contributor

@gvegayon gvegayon commented May 3, 2016

Hi, I don't know if this behavior is expected (I don't think so), but whenever I try to pass dgCMatrix objects within lists to RcppArmadillo via arma::sp_mat valgrind catches a memory leak. Here is a minimal example in which I create a list with dense matrices arma::mat and sparse matrices arma::sp_mat

The code for arma::sp_mat:

library(Matrix)
library(Rcpp)

code <-
'// [[Rcpp::depends(RcppArmadillo)]]
#include <RcppArmadillo.h>
using namespace Rcpp;

// [[Rcpp::export]]
List passMats(List l) {
  arma::sp_mat m1 = as< arma::sp_mat >(l[0]);
  arma::sp_mat m2 = as< arma::sp_mat >(l[1]);

  return List::create(
    _["mat1"] = m1,
    _["mat2"] = m2
  );
}'

sourceCpp(code=code)

x <- matrix(0,4,4)
x <- methods::as(x, "dgCMatrix")
passMats(list(x,x))

devtools::session_info()

And this is the output that I get from running R -d valgrind --vanilla

==3233== Memcheck, a memory error detector
==3233== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==3233== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==3233== Command: /usr/lib/R/bin/exec/R --vanilla
==3233== 

R version 3.2.5 (2016-04-14) -- "Very, Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(Matrix)
> library(Rcpp)
> 
> code <-
+ '// [[Rcpp::depends(RcppArmadillo)]]
+ #include <RcppArmadillo.h>
+ using namespace Rcpp;
+ 
+ // [[Rcpp::export]]
+ List passMats(List l) {
+   arma::sp_mat m1 = as< arma::sp_mat >(l[0]);
+   arma::sp_mat m2 = as< arma::sp_mat >(l[1]);
+ 
+   return List::create(
+     _["mat1"] = m1,
+     _["mat2"] = m2
+   );
+ }'
> 
> sourceCpp(code=code)
> 
> x <- matrix(0,4,4)
> x <- methods::as(x, "dgCMatrix")
> passMats(list(x,x))
$mat1
4 x 4 sparse Matrix of class "dgCMatrix"

[1,] . . . .
[2,] . . . .
[3,] . . . .
[4,] . . . .

$mat2
4 x 4 sparse Matrix of class "dgCMatrix"

[1,] . . . .
[2,] . . . .
[3,] . . . .
[4,] . . . .

> 
> devtools::session_info()
Session info -------------------------------------------------------------------
 setting  value                       
 version  R version 3.2.5 (2016-04-14)
 system   x86_64, linux-gnu           
 ui       X11                         
 language en_US                       
 collate  en_US.UTF-8                 
 tz       <NA>                        
 date     2016-05-03                  

Packages -----------------------------------------------------------------------
 package       * version     date       source        
 devtools        1.11.1      2016-04-21 CRAN (R 3.2.5)
 digest          0.6.9       2016-01-08 CRAN (R 3.2.3)
 lattice         0.20-33     2015-07-14 CRAN (R 3.2.1)
 Matrix        * 1.2-5       2016-04-17 CRAN (R 3.2.5)
 memoise         1.0.0       2016-01-29 CRAN (R 3.2.3)
 Rcpp          * 0.12.4      2016-03-26 CRAN (R 3.2.4)
 RcppArmadillo   0.6.700.3.0 2016-04-06 CRAN (R 3.2.5)
 withr           1.0.1       2016-02-04 CRAN (R 3.2.3)
> 
==3233== 
==3233== HEAP SUMMARY:
==3233==     in use at exit: 102,296,799 bytes in 54,646 blocks
==3233==   total heap usage: 108,090 allocs, 53,444 frees, 207,673,479 bytes allocated
==3233== 
==3233== 1,024 bytes in 1 blocks are definitely lost in loss record 171 of 2,960
==3233==    at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x12A32D8D: acquire<unsigned int> (memory.hpp:75)
==3233==    by 0x12A32D8D: acquire_chunked<unsigned int> (memory.hpp:112)
==3233==    by 0x12A32D8D: init (SpMat_meat.hpp:3964)
==3233==    by 0x12A32D8D: SpMat (SpMat_meat.hpp:76)
==3233==    by 0x12A32D8D: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (RcppArmadilloAs.h:94)
==3233==    by 0x12A305B7: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) [clone .isra.63] (as.h:81)
==3233==    by 0x12A306D8: as<arma::SpMat<double> > (as.h:144)
==3233==    by 0x12A306D8: passMats(Rcpp::Vector<19, Rcpp::PreserveStorage>) (fileca12b327c2b.cpp:7)
==3233==    by 0x12A30CF5: sourceCpp_0_passMats (fileca12b327c2b.cpp:25)
==3233==    by 0x4F0BA37: ??? (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4ACCA: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4BE56: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4A8AE: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F71D61: Rf_ReplIteration (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F720B0: ??? (in /usr/lib/R/lib/libR.so)
==3233== 
==3233== 1,024 bytes in 1 blocks are definitely lost in loss record 172 of 2,960
==3233==    at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x12A32D8D: acquire<unsigned int> (memory.hpp:75)
==3233==    by 0x12A32D8D: acquire_chunked<unsigned int> (memory.hpp:112)
==3233==    by 0x12A32D8D: init (SpMat_meat.hpp:3964)
==3233==    by 0x12A32D8D: SpMat (SpMat_meat.hpp:76)
==3233==    by 0x12A32D8D: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (RcppArmadilloAs.h:94)
==3233==    by 0x12A305B7: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) [clone .isra.63] (as.h:81)
==3233==    by 0x12A306FC: as<arma::SpMat<double> > (as.h:144)
==3233==    by 0x12A306FC: passMats(Rcpp::Vector<19, Rcpp::PreserveStorage>) (fileca12b327c2b.cpp:8)
==3233==    by 0x12A30CF5: sourceCpp_0_passMats (fileca12b327c2b.cpp:25)
==3233==    by 0x4F0BA37: ??? (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4ACCA: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4BE56: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4A8AE: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F71D61: Rf_ReplIteration (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F720B0: ??? (in /usr/lib/R/lib/libR.so)
==3233== 
==3233== 2,048 bytes in 1 blocks are definitely lost in loss record 1,212 of 2,960
==3233==    at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x12A32D61: acquire<double> (memory.hpp:75)
==3233==    by 0x12A32D61: acquire_chunked<double> (memory.hpp:112)
==3233==    by 0x12A32D61: init (SpMat_meat.hpp:3963)
==3233==    by 0x12A32D61: SpMat (SpMat_meat.hpp:76)
==3233==    by 0x12A32D61: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (RcppArmadilloAs.h:94)
==3233==    by 0x12A305B7: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) [clone .isra.63] (as.h:81)
==3233==    by 0x12A306D8: as<arma::SpMat<double> > (as.h:144)
==3233==    by 0x12A306D8: passMats(Rcpp::Vector<19, Rcpp::PreserveStorage>) (fileca12b327c2b.cpp:7)
==3233==    by 0x12A30CF5: sourceCpp_0_passMats (fileca12b327c2b.cpp:25)
==3233==    by 0x4F0BA37: ??? (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4ACCA: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4BE56: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4A8AE: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F71D61: Rf_ReplIteration (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F720B0: ??? (in /usr/lib/R/lib/libR.so)
==3233== 
==3233== 2,048 bytes in 1 blocks are definitely lost in loss record 1,213 of 2,960
==3233==    at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x12A32D61: acquire<double> (memory.hpp:75)
==3233==    by 0x12A32D61: acquire_chunked<double> (memory.hpp:112)
==3233==    by 0x12A32D61: init (SpMat_meat.hpp:3963)
==3233==    by 0x12A32D61: SpMat (SpMat_meat.hpp:76)
==3233==    by 0x12A32D61: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (RcppArmadilloAs.h:94)
==3233==    by 0x12A305B7: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) [clone .isra.63] (as.h:81)
==3233==    by 0x12A306FC: as<arma::SpMat<double> > (as.h:144)
==3233==    by 0x12A306FC: passMats(Rcpp::Vector<19, Rcpp::PreserveStorage>) (fileca12b327c2b.cpp:8)
==3233==    by 0x12A30CF5: sourceCpp_0_passMats (fileca12b327c2b.cpp:25)
==3233==    by 0x4F0BA37: ??? (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4ACCA: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4BE56: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4A8AE: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F71D61: Rf_ReplIteration (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F720B0: ??? (in /usr/lib/R/lib/libR.so)
==3233== 
==3233== LEAK SUMMARY:
==3233==    definitely lost: 6,144 bytes in 4 blocks
==3233==    indirectly lost: 0 bytes in 0 blocks
==3233==      possibly lost: 0 bytes in 0 blocks
==3233==    still reachable: 102,290,655 bytes in 54,642 blocks
==3233==         suppressed: 0 bytes in 0 blocks
==3233== Reachable blocks (those to which a pointer was found) are not shown.
==3233== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3233== 
==3233== For counts of detected and suppressed errors, rerun with: -v
==3233== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)

Using "regular" arma::mat

library(Matrix)
library(Rcpp)

code <-
'// [[Rcpp::depends(RcppArmadillo)]]
#include <RcppArmadillo.h>
using namespace Rcpp;

// [[Rcpp::export]]
List passMats(List l) {
  arma::mat m1 = as< arma::mat >(l[0]);  
  arma::mat m2 = as< arma::mat >(l[1]);

  return List::create(
    _["mat1"] = m1,
    _["mat2"] = m2
  );
}'

sourceCpp(code=code)

x <- matrix(0,4,4)
passMats(list(x,x))

devtools::session_info()

I get

==3317== Memcheck, a memory error detector
==3317== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==3317== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==3317== Command: /usr/lib/R/bin/exec/R --vanilla
==3317== 

R version 3.2.5 (2016-04-14) -- "Very, Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(Matrix)
> library(Rcpp)
> 
> code <-
+ '// [[Rcpp::depends(RcppArmadillo)]]
+ #include <RcppArmadillo.h>
+ using namespace Rcpp;
+ 
+ // [[Rcpp::export]]
+ List passMats(List l) {
+   arma::mat m1 = as< arma::mat >(l[0]);
+   arma::mat m2 = as< arma::mat >(l[1]);
+ 
+   return List::create(
+     _["mat1"] = m1,
+     _["mat2"] = m2
+   );
+ }'
> 
> sourceCpp(code=code)
> 
> x <- matrix(0,4,4)
> passMats(list(x,x))
$mat1
     [,1] [,2] [,3] [,4]
[1,]    0    0    0    0
[2,]    0    0    0    0
[3,]    0    0    0    0
[4,]    0    0    0    0

$mat2
     [,1] [,2] [,3] [,4]
[1,]    0    0    0    0
[2,]    0    0    0    0
[3,]    0    0    0    0
[4,]    0    0    0    0

> 
> devtools::session_info()
Session info -------------------------------------------------------------------
 setting  value                       
 version  R version 3.2.5 (2016-04-14)
 system   x86_64, linux-gnu           
 ui       X11                         
 language en_US                       
 collate  en_US.UTF-8                 
 tz       <NA>                        
 date     2016-05-03                  

Packages -----------------------------------------------------------------------
 package       * version     date       source        
 devtools        1.11.1      2016-04-21 CRAN (R 3.2.5)
 digest          0.6.9       2016-01-08 CRAN (R 3.2.3)
 lattice         0.20-33     2015-07-14 CRAN (R 3.2.1)
 Matrix        * 1.2-5       2016-04-17 CRAN (R 3.2.5)
 memoise         1.0.0       2016-01-29 CRAN (R 3.2.3)
 Rcpp          * 0.12.4      2016-03-26 CRAN (R 3.2.4)
 RcppArmadillo   0.6.700.3.0 2016-04-06 CRAN (R 3.2.5)
 withr           1.0.1       2016-02-04 CRAN (R 3.2.3)
> 
==3317== 
==3317== HEAP SUMMARY:
==3317==     in use at exit: 102,127,759 bytes in 54,510 blocks
==3317==   total heap usage: 107,562 allocs, 53,052 frees, 207,005,727 bytes allocated
==3317== 
==3317== LEAK SUMMARY:
==3317==    definitely lost: 0 bytes in 0 blocks
==3317==    indirectly lost: 0 bytes in 0 blocks
==3317==      possibly lost: 0 bytes in 0 blocks
==3317==    still reachable: 102,127,759 bytes in 54,510 blocks
==3317==         suppressed: 0 bytes in 0 blocks
==3317== Reachable blocks (those to which a pointer was found) are not shown.
==3317== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3317== 
==3317== For counts of detected and suppressed errors, rerun with: -v
==3317== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

For valgrind I'm using the following options:

--tool=memcheck
--memcheck:leak-check=full
--track-origins=yes
--error-limit=no

Am I doing/gettin something wrong here?

Thanks
@eddelbuettel
Copy link
Member

@eddelbuettel eddelbuettel commented May 3, 2016

Sorry, but what is your point?

Are you obsessing over

still reachable: 102,127,759 bytes in 54,510 blocks

If so, run some other R code and code. This is common as I recall. Also head the advice of

To see them, rerun with: --leak-check=full --show-leak-kinds=all
@gvegayon
Copy link
Contributor Author

@gvegayon gvegayon commented May 3, 2016

No, I'm not worried about still reachable, I'm worried about

definitely lost: 6,144 bytes in 4 blocks

This is a small example (4 by 4 matrix), so when I use bigger matrices this number goes up (big time). Should I worry about this?
@eddelbuettel
Copy link
Member

@eddelbuettel eddelbuettel commented May 3, 2016

Sorry, missed that.

Sure. Better still, can you debug the sparse code in the one or two conversion headers? I wrote most of that as a one-off as I don't actually use sparse matrices ...
@gvegayon
Copy link
Contributor Author

@gvegayon gvegayon commented May 3, 2016 •
edited

Sure, happy to help (will do what I can as I still consider myself new to C++). Should I focus on https://github.com/RcppCore/RcppArmadillo/blob/master/inst/include/RcppArmadilloWrap.h#L80-L104 and https://github.com/RcppCore/RcppArmadillo/blob/master/inst/include/RcppArmadilloAs.h#L81-L119 ?
@kevinushey
Copy link
Contributor

@kevinushey kevinushey commented May 3, 2016

FWIW I can confirm a leak is detected using the OS X Instruments 'Leak' instrument; I can help dig in later as well.
This was referenced May 4, 2016
Closed
Merged
@gvegayon
Copy link
Contributor Author

@gvegayon gvegayon commented May 4, 2016

Solved in #91
@gvegayon gvegayon closed this May 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
@eddelbuettel @gvegayon @kevinushey
You can’t perform that action at this time.