Possible memory leak with arma::sp_mat within Rcpp::List #89

gvegayon · 2016-05-03T19:13:20Z

Hi, I don't know if this behavior is expected (I don't think so), but whenever I try to pass dgCMatrix objects within lists to RcppArmadillo via arma::sp_mat valgrind catches a memory leak. Here is a minimal example in which I create a list with dense matrices arma::mat and sparse matrices arma::sp_mat

The code for arma::sp_mat:

library(Matrix)
library(Rcpp)

code <-
'// [[Rcpp::depends(RcppArmadillo)]]
#include <RcppArmadillo.h>
using namespace Rcpp;

// [[Rcpp::export]]
List passMats(List l) {
  arma::sp_mat m1 = as< arma::sp_mat >(l[0]);
  arma::sp_mat m2 = as< arma::sp_mat >(l[1]);

  return List::create(
    _["mat1"] = m1,
    _["mat2"] = m2
  );
}'

sourceCpp(code=code)

x <- matrix(0,4,4)
x <- methods::as(x, "dgCMatrix")
passMats(list(x,x))

devtools::session_info()

And this is the output that I get from running R -d valgrind --vanilla

==3233== Memcheck, a memory error detector
==3233== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==3233== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==3233== Command: /usr/lib/R/bin/exec/R --vanilla
==3233== 

R version 3.2.5 (2016-04-14) -- "Very, Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(Matrix)
> library(Rcpp)
> 
> code <-
+ '// [[Rcpp::depends(RcppArmadillo)]]
+ #include <RcppArmadillo.h>
+ using namespace Rcpp;
+ 
+ // [[Rcpp::export]]
+ List passMats(List l) {
+   arma::sp_mat m1 = as< arma::sp_mat >(l[0]);
+   arma::sp_mat m2 = as< arma::sp_mat >(l[1]);
+ 
+   return List::create(
+     _["mat1"] = m1,
+     _["mat2"] = m2
+   );
+ }'
> 
> sourceCpp(code=code)
> 
> x <- matrix(0,4,4)
> x <- methods::as(x, "dgCMatrix")
> passMats(list(x,x))
$mat1
4 x 4 sparse Matrix of class "dgCMatrix"

[1,] . . . .
[2,] . . . .
[3,] . . . .
[4,] . . . .

$mat2
4 x 4 sparse Matrix of class "dgCMatrix"

[1,] . . . .
[2,] . . . .
[3,] . . . .
[4,] . . . .

> 
> devtools::session_info()
Session info -------------------------------------------------------------------
 setting  value                       
 version  R version 3.2.5 (2016-04-14)
 system   x86_64, linux-gnu           
 ui       X11                         
 language en_US                       
 collate  en_US.UTF-8                 
 tz       <NA>                        
 date     2016-05-03                  

Packages -----------------------------------------------------------------------
 package       * version     date       source        
 devtools        1.11.1      2016-04-21 CRAN (R 3.2.5)
 digest          0.6.9       2016-01-08 CRAN (R 3.2.3)
 lattice         0.20-33     2015-07-14 CRAN (R 3.2.1)
 Matrix        * 1.2-5       2016-04-17 CRAN (R 3.2.5)
 memoise         1.0.0       2016-01-29 CRAN (R 3.2.3)
 Rcpp          * 0.12.4      2016-03-26 CRAN (R 3.2.4)
 RcppArmadillo   0.6.700.3.0 2016-04-06 CRAN (R 3.2.5)
 withr           1.0.1       2016-02-04 CRAN (R 3.2.3)
> 
==3233== 
==3233== HEAP SUMMARY:
==3233==     in use at exit: 102,296,799 bytes in 54,646 blocks
==3233==   total heap usage: 108,090 allocs, 53,444 frees, 207,673,479 bytes allocated
==3233== 
==3233== 1,024 bytes in 1 blocks are definitely lost in loss record 171 of 2,960
==3233==    at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x12A32D8D: acquire<unsigned int> (memory.hpp:75)
==3233==    by 0x12A32D8D: acquire_chunked<unsigned int> (memory.hpp:112)
==3233==    by 0x12A32D8D: init (SpMat_meat.hpp:3964)
==3233==    by 0x12A32D8D: SpMat (SpMat_meat.hpp:76)
==3233==    by 0x12A32D8D: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (RcppArmadilloAs.h:94)
==3233==    by 0x12A305B7: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) [clone .isra.63] (as.h:81)
==3233==    by 0x12A306D8: as<arma::SpMat<double> > (as.h:144)
==3233==    by 0x12A306D8: passMats(Rcpp::Vector<19, Rcpp::PreserveStorage>) (fileca12b327c2b.cpp:7)
==3233==    by 0x12A30CF5: sourceCpp_0_passMats (fileca12b327c2b.cpp:25)
==3233==    by 0x4F0BA37: ??? (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4ACCA: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4BE56: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4A8AE: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F71D61: Rf_ReplIteration (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F720B0: ??? (in /usr/lib/R/lib/libR.so)
==3233== 
==3233== 1,024 bytes in 1 blocks are definitely lost in loss record 172 of 2,960
==3233==    at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x12A32D8D: acquire<unsigned int> (memory.hpp:75)
==3233==    by 0x12A32D8D: acquire_chunked<unsigned int> (memory.hpp:112)
==3233==    by 0x12A32D8D: init (SpMat_meat.hpp:3964)
==3233==    by 0x12A32D8D: SpMat (SpMat_meat.hpp:76)
==3233==    by 0x12A32D8D: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (RcppArmadilloAs.h:94)
==3233==    by 0x12A305B7: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) [clone .isra.63] (as.h:81)
==3233==    by 0x12A306FC: as<arma::SpMat<double> > (as.h:144)
==3233==    by 0x12A306FC: passMats(Rcpp::Vector<19, Rcpp::PreserveStorage>) (fileca12b327c2b.cpp:8)
==3233==    by 0x12A30CF5: sourceCpp_0_passMats (fileca12b327c2b.cpp:25)
==3233==    by 0x4F0BA37: ??? (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4ACCA: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4BE56: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4A8AE: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F71D61: Rf_ReplIteration (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F720B0: ??? (in /usr/lib/R/lib/libR.so)
==3233== 
==3233== 2,048 bytes in 1 blocks are definitely lost in loss record 1,212 of 2,960
==3233==    at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x12A32D61: acquire<double> (memory.hpp:75)
==3233==    by 0x12A32D61: acquire_chunked<double> (memory.hpp:112)
==3233==    by 0x12A32D61: init (SpMat_meat.hpp:3963)
==3233==    by 0x12A32D61: SpMat (SpMat_meat.hpp:76)
==3233==    by 0x12A32D61: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (RcppArmadilloAs.h:94)
==3233==    by 0x12A305B7: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) [clone .isra.63] (as.h:81)
==3233==    by 0x12A306D8: as<arma::SpMat<double> > (as.h:144)
==3233==    by 0x12A306D8: passMats(Rcpp::Vector<19, Rcpp::PreserveStorage>) (fileca12b327c2b.cpp:7)
==3233==    by 0x12A30CF5: sourceCpp_0_passMats (fileca12b327c2b.cpp:25)
==3233==    by 0x4F0BA37: ??? (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4ACCA: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4BE56: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4A8AE: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F71D61: Rf_ReplIteration (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F720B0: ??? (in /usr/lib/R/lib/libR.so)
==3233== 
==3233== 2,048 bytes in 1 blocks are definitely lost in loss record 1,213 of 2,960
==3233==    at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3233==    by 0x12A32D61: acquire<double> (memory.hpp:75)
==3233==    by 0x12A32D61: acquire_chunked<double> (memory.hpp:112)
==3233==    by 0x12A32D61: init (SpMat_meat.hpp:3963)
==3233==    by 0x12A32D61: SpMat (SpMat_meat.hpp:76)
==3233==    by 0x12A32D61: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (RcppArmadilloAs.h:94)
==3233==    by 0x12A305B7: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) [clone .isra.63] (as.h:81)
==3233==    by 0x12A306FC: as<arma::SpMat<double> > (as.h:144)
==3233==    by 0x12A306FC: passMats(Rcpp::Vector<19, Rcpp::PreserveStorage>) (fileca12b327c2b.cpp:8)
==3233==    by 0x12A30CF5: sourceCpp_0_passMats (fileca12b327c2b.cpp:25)
==3233==    by 0x4F0BA37: ??? (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4ACCA: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4BE56: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F4A8AE: Rf_eval (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F71D61: Rf_ReplIteration (in /usr/lib/R/lib/libR.so)
==3233==    by 0x4F720B0: ??? (in /usr/lib/R/lib/libR.so)
==3233== 
==3233== LEAK SUMMARY:
==3233==    definitely lost: 6,144 bytes in 4 blocks
==3233==    indirectly lost: 0 bytes in 0 blocks
==3233==      possibly lost: 0 bytes in 0 blocks
==3233==    still reachable: 102,290,655 bytes in 54,642 blocks
==3233==         suppressed: 0 bytes in 0 blocks
==3233== Reachable blocks (those to which a pointer was found) are not shown.
==3233== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3233== 
==3233== For counts of detected and suppressed errors, rerun with: -v
==3233== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)

Using "regular" arma::mat

library(Matrix)
library(Rcpp)

code <-
'// [[Rcpp::depends(RcppArmadillo)]]
#include <RcppArmadillo.h>
using namespace Rcpp;

// [[Rcpp::export]]
List passMats(List l) {
  arma::mat m1 = as< arma::mat >(l[0]);  
  arma::mat m2 = as< arma::mat >(l[1]);

  return List::create(
    _["mat1"] = m1,
    _["mat2"] = m2
  );
}'

sourceCpp(code=code)

x <- matrix(0,4,4)
passMats(list(x,x))

devtools::session_info()

I get

==3317== Memcheck, a memory error detector
==3317== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==3317== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==3317== Command: /usr/lib/R/bin/exec/R --vanilla
==3317== 

R version 3.2.5 (2016-04-14) -- "Very, Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(Matrix)
> library(Rcpp)
> 
> code <-
+ '// [[Rcpp::depends(RcppArmadillo)]]
+ #include <RcppArmadillo.h>
+ using namespace Rcpp;
+ 
+ // [[Rcpp::export]]
+ List passMats(List l) {
+   arma::mat m1 = as< arma::mat >(l[0]);
+   arma::mat m2 = as< arma::mat >(l[1]);
+ 
+   return List::create(
+     _["mat1"] = m1,
+     _["mat2"] = m2
+   );
+ }'
> 
> sourceCpp(code=code)
> 
> x <- matrix(0,4,4)
> passMats(list(x,x))
$mat1
     [,1] [,2] [,3] [,4]
[1,]    0    0    0    0
[2,]    0    0    0    0
[3,]    0    0    0    0
[4,]    0    0    0    0

$mat2
     [,1] [,2] [,3] [,4]
[1,]    0    0    0    0
[2,]    0    0    0    0
[3,]    0    0    0    0
[4,]    0    0    0    0

> 
> devtools::session_info()
Session info -------------------------------------------------------------------
 setting  value                       
 version  R version 3.2.5 (2016-04-14)
 system   x86_64, linux-gnu           
 ui       X11                         
 language en_US                       
 collate  en_US.UTF-8                 
 tz       <NA>                        
 date     2016-05-03                  

Packages -----------------------------------------------------------------------
 package       * version     date       source        
 devtools        1.11.1      2016-04-21 CRAN (R 3.2.5)
 digest          0.6.9       2016-01-08 CRAN (R 3.2.3)
 lattice         0.20-33     2015-07-14 CRAN (R 3.2.1)
 Matrix        * 1.2-5       2016-04-17 CRAN (R 3.2.5)
 memoise         1.0.0       2016-01-29 CRAN (R 3.2.3)
 Rcpp          * 0.12.4      2016-03-26 CRAN (R 3.2.4)
 RcppArmadillo   0.6.700.3.0 2016-04-06 CRAN (R 3.2.5)
 withr           1.0.1       2016-02-04 CRAN (R 3.2.3)
> 
==3317== 
==3317== HEAP SUMMARY:
==3317==     in use at exit: 102,127,759 bytes in 54,510 blocks
==3317==   total heap usage: 107,562 allocs, 53,052 frees, 207,005,727 bytes allocated
==3317== 
==3317== LEAK SUMMARY:
==3317==    definitely lost: 0 bytes in 0 blocks
==3317==    indirectly lost: 0 bytes in 0 blocks
==3317==      possibly lost: 0 bytes in 0 blocks
==3317==    still reachable: 102,127,759 bytes in 54,510 blocks
==3317==         suppressed: 0 bytes in 0 blocks
==3317== Reachable blocks (those to which a pointer was found) are not shown.
==3317== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3317== 
==3317== For counts of detected and suppressed errors, rerun with: -v
==3317== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

For valgrind I'm using the following options:

--tool=memcheck
--memcheck:leak-check=full
--track-origins=yes
--error-limit=no

Am I doing/gettin something wrong here?

Thanks

The text was updated successfully, but these errors were encountered:

eddelbuettel · 2016-05-03T19:41:10Z

Sorry, but what is your point?

Are you obsessing over

still reachable: 102,127,759 bytes in 54,510 blocks

If so, run some other R code and code. This is common as I recall. Also head the advice of

To see them, rerun with: --leak-check=full --show-leak-kinds=all

gvegayon · 2016-05-03T19:48:25Z

No, I'm not worried about still reachable, I'm worried about

definitely lost: 6,144 bytes in 4 blocks

This is a small example (4 by 4 matrix), so when I use bigger matrices this number goes up (big time). Should I worry about this?

eddelbuettel · 2016-05-03T20:06:40Z

Sorry, missed that.

Sure. Better still, can you debug the sparse code in the one or two conversion headers? I wrote most of that as a one-off as I don't actually use sparse matrices ...

gvegayon · 2016-05-03T20:17:53Z

Sure, happy to help (will do what I can as I still consider myself new to C++). Should I focus on https://github.com/RcppCore/RcppArmadillo/blob/master/inst/include/RcppArmadilloWrap.h#L80-L104 and https://github.com/RcppCore/RcppArmadillo/blob/master/inst/include/RcppArmadilloAs.h#L81-L119 ?

kevinushey · 2016-05-03T20:44:30Z

FWIW I can confirm a leak is detected using the OS X Instruments 'Leak' instrument; I can help dig in later as well.

gvegayon · 2016-05-04T20:56:24Z

Solved in #91

This was referenced May 4, 2016

Allowing ARMA_64BIT_WORD if required #88

Closed

Fixing bug in SpMat exporter #91

Merged

gvegayon closed this as completed May 4, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Possible memory leak with arma::sp_mat within Rcpp::List #89

Possible memory leak with arma::sp_mat within Rcpp::List #89

gvegayon commented May 3, 2016

eddelbuettel commented May 3, 2016

gvegayon commented May 3, 2016

eddelbuettel commented May 3, 2016

gvegayon commented May 3, 2016 •

edited

Loading

kevinushey commented May 3, 2016

gvegayon commented May 4, 2016

Possible memory leak with arma::sp_mat within Rcpp::List #89

Possible memory leak with arma::sp_mat within Rcpp::List #89

Comments

gvegayon commented May 3, 2016

eddelbuettel commented May 3, 2016

gvegayon commented May 3, 2016

eddelbuettel commented May 3, 2016

gvegayon commented May 3, 2016 • edited Loading

kevinushey commented May 3, 2016

gvegayon commented May 4, 2016

gvegayon commented May 3, 2016 •

edited

Loading