Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Probable SIGSEGV due to out-of-bounds access in updateRecycledModels #76

Open
tveasey opened this issue Apr 27, 2018 · 0 comments

Comments

Projects
None yet
1 participant
@tveasey
Copy link
Contributor

commented Apr 27, 2018

We've had a crash reported against 5.6.8 (possibly due to incorrect state upgrade from 5.5 -> 5.6.8) with stack trace:

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./autodetect --jobid=webscan_rare_responses_v2 --licenseValidation=213'.
Program terminated with signal 11, Segmentation fault.
...
(gdb) bt
#0  0x00007f4b52d48eaf in ml::model::CModel::updateRecycledModels() () from /mnt/Downloads/libs/libMlModel.so
#1  0x00007f4b52d73466 in ml::model::CPopulationModel::updateRecycledModels() () from /mnt/Downloads/libs/libMlModel.so
#2  0x00007f4b52d6d6e9 in ml::model::CPopulationModel::createUpdateNewModels(long, ml::model::CResourceMonitor&) () from /mnt/Downloads/libs/libMlModel.so
#3  0x00007f4b52bb98ce in ml::model::CEventRatePopulationModel::sample(long, long, ml::model::CResourceMonitor&) () from /mnt/Downloads/libs/libMlModel.so
#4  0x00007f4b52aee6c9 in ml::model::CAnomalyDetector::sample(long, long, ml::model::CResourceMonitor&) () from /mnt/Downloads/libs/libMlModel.so
#5  0x00007f4b52af39b2 in void ml::model::CAnomalyDetector::buildResultsHelper<boost::_bi::bind_t<void, boost::_mfi::mf3<void, ml::model::CAnomalyDetector, long, long, ml::model::CResourceMonitor&>, boost::_bi::list4<boost::_bi::value<ml::model::CAnomalyDetector*>, boost::arg<1>, boost::arg<2>, boost::reference_wrapper<ml::model::CResourceMonitor> > >, boost::_bi::bind_t<void, boost::_mfi::mf1<void, ml::model::CAnomalyDetector, long>, boost::_bi::list2<boost::_bi::value<ml::model::CAnomalyDetector*>, boost::arg<1> > > >(long, long, boost::_bi::bind_t<void, boost::_mfi::mf3<void, ml::model::CAnomalyDetector, long, long, ml::model::CResourceMonitor&>, boost::_bi::list4<boost::_bi::value<ml::model::CAnomalyDetector*>, boost::arg<1>, boost::arg<2>, boost::reference_wrapper<ml::model::CResourceMonitor> > >, boost::_bi::bind_t<void, boost::_mfi::mf1<void, ml::model::CAnomalyDetector, long>, boost::_bi::list2<boost::_bi::value<ml::model::CAnomalyDetector*>, boost::arg<1> > >, ml::model::CHierarchicalResults&) ()
   from /mnt/Downloads/libs/libMlModel.so
#6  0x00007f4b52af230c in ml::model::CAnomalyDetector::buildResults(long, long, ml::model::CHierarchicalResults&) () from /mnt/libs/libMlModel.so
#7  0x00007f4b52748ea0 in ml::api::CAnomalyDetector::outputResults(long) () from /mnt/Downloads/libs/libMlApi.so
#8  0x00007f4b5274922e in ml::api::CAnomalyDetector::outputBucketResultsUntil(long) () from /mnt/Downloads/libs/libMlApi.so
#9  0x00007f4b5274a729 in ml::api::CAnomalyDetector::handleRecord(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, boost::unordered::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) ()
   from /mnt/Downloads/libs/libMlApi.so
#10 0x00007f4b527c7a71 in ml::api::CLengthEncodedInputParser::readStream(boost::function2<bool, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, boost::unordered::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&> const&) () from /mnt/Downloads/libs/libMlApi.so
#11 0x00007f4b527766d3 in ml::api::CCmdSkeleton::ioLoop() () from /mnt/Downloads/libs/libMlApi.so
#12 0x00007f4b54c8849d in main ()

The likely cause is access to m_PersonBucketCounts being out-of-range. There ought to be an invariant preventing this. As a short term fix, we should make the code defensive and log an error. (We should check other uses of recycledPersonIds which may also be problematic.) As a separate task we need to understand how we could have got into this situation.

\cc @hoigau.

@tveasey tveasey added >bug :ml labels Apr 27, 2018

tveasey added a commit that referenced this issue May 1, 2018

tveasey added a commit that referenced this issue May 1, 2018

tveasey added a commit that referenced this issue May 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.