[R-package] memory leaks detected by valgrind

[jameslamb]

Running the valgrind checks on the R package in #3703, I see the following.

==2176== LEAK SUMMARY:
==2176==    definitely lost: 318 bytes in 10 blocks
==2176==    indirectly lost: 576 bytes in 9 blocks
==2176==      possibly lost: 336 bytes in 1 blocks
==2176==    still reachable: 312,088,759 bytes in 55,705 blocks
==2176==                       of which reachable via heuristic:
==2176==                         newarray           : 4,264 bytes in 1 blocks
==2176==         suppressed: 0 bytes in 0 blocks
==2176== Reachable blocks (those to which a pointer was found) are not shown.
==2176== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2176== 
==2176== For lists of detected and suppressed errors, rerun with: -s
==2176== ERROR SUMMARY: 153 errors from 14 contexts (suppressed: 0 from 0)

This means that either that PR or some new changes on master have introduced issues that will prevent the next release of LightGBM from making it to CRAN. I'll test if this is related to #3699 or if it's already an issue on master and update this issue.

full log from #3703

==2176== Memcheck, a memory error detector
==2176== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2176== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==2176== Command: /usr/local/RDvalgrind/lib/R/bin/exec/R --no-readline --vanilla -f testthat.R
==2176== 
Loading required package: R6
==2176== Conditional jump or move depends on uninitialised value(s)
==2176==    at 0x1541D563: LightGBM::LinearTreeLearner::Train(float const*, float const*, bool) (linear_tree_learner.cpp:110)
==2176==    by 0x1524FB12: LightGBM::GBDT::TrainOneIter(float const*, float const*) (gbdt.cpp:401)
==2176==    by 0x15585518: LightGBM::Booster::TrainOneIter() (c_api.cpp:351)
==2176==    by 0x155771F0: LGBM_BoosterUpdateOneIter (c_api.cpp:1637)
==2176==    by 0x155A5F13: LGBM_BoosterUpdateOneIter_R (lightgbm_R.cpp:399)
==2176==    by 0x4941E37: R_doDotCall (dotcode.c:601)
==2176==    by 0x494CFE9: do_dotcall (dotcode.c:1281)
==2176==    by 0x49A06EB: bcEval (eval.c:7072)
==2176==    by 0x498C2CC: Rf_eval (eval.c:727)
==2176==    by 0x498F03B: R_execClosure (eval.c:1897)
==2176==    by 0x498ECEE: Rf_applyClosure (eval.c:1823)
==2176==    by 0x498CAC7: Rf_eval (eval.c:850)
==2176==  Uninitialised value was created by a heap allocation
==2176==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2176==    by 0x1555CB8D: LightGBM::TreeLearner::CreateTreeLearner(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, LightGBM::Config const*) (tree_learner.cpp:20)
==2176==    by 0x1524D381: LightGBM::GBDT::Init(LightGBM::Config const*, LightGBM::Dataset const*, LightGBM::ObjectiveFunction const*, std::vector<LightGBM::Metric const*, std::allocator<LightGBM::Metric const*> > const&) (gbdt.cpp:93)
==2176==    by 0x15582FCE: LightGBM::Booster::Booster(LightGBM::Dataset const*, char const*) (c_api.cpp:139)
==2176==    by 0x1557620A: LGBM_BoosterCreate (c_api.cpp:1535)
==2176==    by 0x155A54A5: LGBM_BoosterCreate_R (lightgbm_R.cpp:326)
==2176==    by 0x4941EAF: R_doDotCall (dotcode.c:607)
==2176==    by 0x494CFE9: do_dotcall (dotcode.c:1281)
==2176==    by 0x49A06EB: bcEval (eval.c:7072)
==2176==    by 0x498C2CC: Rf_eval (eval.c:727)
==2176==    by 0x498F03B: R_execClosure (eval.c:1897)
==2176==    by 0x498ECEE: Rf_applyClosure (eval.c:1823)
==2176== 
==2176== Conditional jump or move depends on uninitialised value(s)
==2176==    at 0x1541E80A: LightGBM::LinearTreeLearner::AddPredictionToScore(LightGBM::Tree const*, double*) const (linear_tree_learner.h:44)
==2176==    by 0x1525561F: LightGBM::ScoreUpdater::AddScore(LightGBM::TreeLearner const*, LightGBM::Tree const*, int) (score_updater.hpp:91)
==2176==    by 0x15250503: LightGBM::GBDT::UpdateScore(LightGBM::Tree const*, int) (gbdt.cpp:495)
==2176==    by 0x1524FCB7: LightGBM::GBDT::TrainOneIter(float const*, float const*) (gbdt.cpp:413)
==2176==    by 0x15585518: LightGBM::Booster::TrainOneIter() (c_api.cpp:351)
==2176==    by 0x155771F0: LGBM_BoosterUpdateOneIter (c_api.cpp:1637)
==2176==    by 0x155A5F13: LGBM_BoosterUpdateOneIter_R (lightgbm_R.cpp:399)
==2176==    by 0x4941E37: R_doDotCall (dotcode.c:601)
==2176==    by 0x494CFE9: do_dotcall (dotcode.c:1281)
==2176==    by 0x49A06EB: bcEval (eval.c:7072)
==2176==    by 0x498C2CC: Rf_eval (eval.c:727)
==2176==    by 0x498F03B: R_execClosure (eval.c:1897)
==2176==  Uninitialised value was created by a heap allocation
==2176==    by 0x15245A23: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate(unsigned long) (stl_vector.h:346)
==2176==    by 0x152429BC: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_create_storage(unsigned long) (stl_vector.h:361)
==2176==    by 0x1523E066: std::_Vector_base<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_Vector_base(unsigned long, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:305)
==2176==    by 0x152FF682: std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::vector(unsigned long, std::vector<double, std::allocator<double> > const&, std::allocator<std::vector<double, std::allocator<double> > > const&) (stl_vector.h:524)
==2176==    by 0x152F94D7: LightGBM::Config::GetAucMuWeights() (config.cpp:159)
==2176==    by 0x152F9CE2: LightGBM::Config::Set(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (config.cpp:219)
==2176==    by 0x1558377D: LightGBM::Booster::CheckDatasetResetConfig(LightGBM::Config const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (c_api.cpp:190)
==2176==    by 0x15575E13: LGBM_DatasetUpdateParamChecking (c_api.cpp:1499)
==2176==    by 0x155A4F52: LGBM_DatasetUpdateParamChecking_R (lightgbm_R.cpp:285)
==2176== 
==2176== 336 bytes in 1 blocks are possibly lost in loss record 161 of 2,735
==2176==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2176==    by 0x40149CA: allocate_dtv (dl-tls.c:286)
==2176==    by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==2176==    by 0x5706322: allocate_stack (allocatestack.c:622)
==2176==    by 0x5706322: pthread_create@@GLIBC_2.2.5 (pthread_create.c:660)
==2176==    by 0x56D4DDA: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==2176==    by 0x56CC8E0: GOMP_parallel (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==2176==    by 0x15574879: LGBM_DatasetCreateFromCSC (c_api.cpp:1303)
==2176==    by 0x155A36CC: LGBM_DatasetCreateFromCSC_R (lightgbm_R.cpp:91)
==2176==    by 0x4942152: R_doDotCall (dotcode.c:634)
==2176==    by 0x494CFE9: do_dotcall (dotcode.c:1281)
==2176==    by 0x49A06EB: bcEval (eval.c:7072)
==2176==    by 0x498C2CC: Rf_eval (eval.c:727)
==2176==    by 0x498F03B: R_execClosure (eval.c:1897)
==2176== 
==2176== 664 (104 direct, 560 indirect) bytes in 1 blocks are definitely lost in loss record 210 of 2,735
==2176==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2176==    by 0x1526815F: __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) (new_allocator.h:115)
==2176==    by 0x15267970: std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) (alloc_traits.h:460)
==2176==    by 0x1527CDF8: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> > >::_M_allocate_buckets(unsigned long) (hashtable_policy.h:2085)
==2176==    by 0x1527C10E: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_allocate_buckets(unsigned long) (hashtable.h:398)
==2176==    by 0x1527ABB0: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) (hashtable.h:2091)
==2176==    by 0x1527932F: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:2070)
==2176==    by 0x15277AD1: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_insert_unique_node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true>*, unsigned long) (hashtable.h:1728)
==2176==    by 0x15300CCD: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::integral_constant<bool, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (hashtable.h:1682)
==2176==    by 0x152FFD76: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (hashtable.h:788)
==2176==    by 0x152FF4E3: std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false, true>, bool> std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (unordered_map.h:388)
==2176==    by 0x152F7331: LightGBM::Config::KV2Map(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*, char const*) (config.cpp:27)
==2176== 
==2176== LEAK SUMMARY:
==2176==    definitely lost: 318 bytes in 10 blocks
==2176==    indirectly lost: 576 bytes in 9 blocks
==2176==      possibly lost: 336 bytes in 1 blocks
==2176==    still reachable: 312,088,759 bytes in 55,705 blocks
==2176==                       of which reachable via heuristic:
==2176==                         newarray           : 4,264 bytes in 1 blocks
==2176==         suppressed: 0 bytes in 0 blocks
==2176== Reachable blocks (those to which a pointer was found) are not shown.
==2176== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2176== 
==2176== For lists of detected and suppressed errors, rerun with: -s
==2176== ERROR SUMMARY: 153 errors from 14 contexts (suppressed: 0 from 0)

As a separate issue, it's problematic that despite new "definitely lost" and "indirectly loss" counts, the build https://github.com/microsoft/LightGBM/runs/1632971712?check_suite_focus=true passed. That points to a bug in the CI script which needs to be fixed.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.