Observing Segmentation fault during counter metric collection.

[ramesht77]

hi,

We are observing below core dump, Can you please let me know if this is a known issue?

(gdb) bt
#0 0x00007f1665707297 in local_Rb_tree_increment (__x=0x7f13bc1fbe50) at ../../../../../gcc-5.4.0/libstdc++-v3/src/c++98/tree.cc:71
#1 std::_Rb_tree_increment (__x=0x7f13bc1fbe50) at ../../../../../gcc-5.4.0/libstdc++-v3/src/c++98/tree.cc:91
#2 0x000000000298e743 in std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::operator++ (this=0x7f14106d2e80)
at /usr/local/include/c++/5.4.0/bits/stl_tree.h:284
#3 0x00000000029d5527 in std::for_each<std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, prometheus::Familyprometheus::Counter::CollectMetric(std::map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::less<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&, prometheus::Counter*) const::{lambda(std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > const&)#1}>(std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, prometheus::Familyprometheus::Counter::CollectMetric(std::map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::less<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&, prometheus::Counter*) const::{lambda(std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > const&)#1}, prometheus::Familyprometheus::Counter::CollectMetric(std::map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::less<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&, prometheus::Counter*) const::{lambda(std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > const&)#1}) (__first=..., __last=..., __f=...)
at /usr/local/include/c++/5.4.0/bits/stl_algo.h:3766
#4 0x00000000029d1b50 in prometheus::Familyprometheus::Counter::CollectMetric (this=0x7f1394001070, metric_labels=..., metric=0x7f13bc295ff0)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/core/src/family.cc:126
#5 0x00000000029d18d9 in prometheus::Familyprometheus::Counter::Collect (this=0x7f1394001070)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/core/src/family.cc:105
#6 0x00000000029dff95 in prometheus::(anonymous namespace)::CollectAll<std::vector<std::unique_ptr<prometheus::Familyprometheus::Counter > > >
(results=..., families=...)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/core/src/registry.cc:20
#7 0x00000000029dfd5f in prometheus::Registry::Collect (this=0x26c14ba0)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/core/src/registry.cc:49
#8 0x0000000002a18afb in prometheus::detail::CollectMetrics (collectables=...)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/pull/src/metrics_collector.cc:20
#9 0x0000000002a146a6 in prometheus::detail::MetricsHandler::handleGet (this=0x26c168b0, conn=0x26c15860)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/pull/src/handler.cc:149
---Type to continue, or q to quit---
#10 0x00000000029f4c57 in CivetServer::requestHandler (conn=0x26c15860, cbdata=0x26c168b0)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/CivetServer.cpp:140
#11 0x0000000002a0af19 in handle_request (conn=0x26c15860)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/civetweb.c:14407
#12 0x0000000002a0e5e4 in process_new_connection (conn=0x26c15860)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/civetweb.c:18432
#13 0x0000000002a0eeb3 in worker_thread_run (conn=0x26c15860)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/civetweb.c:18837
#14 0x0000000002a0efb9 in worker_thread (thread_func_param=0x26c15860)
at /share/build_cache/Opensource/CentOS79/CACHE1/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/civetweb.c:18892
#15 0x00007f1665239ea5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f1664f62b0d in gnu_dev_makedev () from /lib64/libc.so.6
#17 0x0000000000000000 in ?? ()
(gdb)

regards,
Ramesh

[ramesht77]

Can someone please suggest here?

[jupp0r]

This is not expected. Can you give some context how you are using the library when this happens? Are you adding metrics dynamically? Is the registry still alive? My guess would be some sort of data race.

[ramesht77]

We are creating the counter during initialization and later these counters are incremented. We are not trying to add dynamically.

[ramesht77]

Can someone please suggest here?

[gjasny]

Hello,

I’d use a modern GCC (from devtoolset if on RHEL) or clang and build with address sanitizer enabled. That should give you more clues.