Prometheus crash due to signal 11 error

[ramesht77]

hi,
Have add fixes as suggested by #557. But observing different issue now, can you please suggest.
regards,
Ramesh

(gdb) bt
#0  0x00000000015b3e55 in std::atomic<double>::load (this=0x0, __m=std::memory_order_seq_cst) at /usr/local/include/c++/5.4.0/atomic:235
#1  0x00000000015b3ed1 in std::atomic<double>::operator double (this=0x0) at /usr/local/include/c++/5.4.0/atomic:194
#2  0x00000000015b3d74 in prometheus::Gauge::Value (this=0x0)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/gauge.cc:40
#3  0x00000000015a617a in prometheus::Counter::Value (this=0x0)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/counter.cc:9
#4  0x00000000015a61b3 in prometheus::Counter::Collect (this=0x0)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/counter.cc:13
#5  0x00000000015a7520 in prometheus::Family<prometheus::Counter>::CollectMetric (this=0x7f9b48042650,
    metric_labels=std::map with 8 elements<error reading variable: Cannot access memory at address 0x17f9b484ae518>, metric=0x0)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/family.cc:110
#6  0x00000000015a7369 in prometheus::Family<prometheus::Counter>::Collect (this=0x7f9b48042650)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/family.cc:102
#7  0x00000000015b596b in prometheus::(anonymous namespace)::CollectAll<std::vector<std::unique_ptr<prometheus::Family<prometheus::Counter> > > > (results=std::vector of length 55, capacity 64 = {...}, families=std::vector of length 103, capacity 128 = {...})
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/registry.cc:20
#8  0x00000000015b5735 in prometheus::Registry::Collect (this=0x5a9f1f0)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/registry.cc:49
#9  0x00000000015ef48d in prometheus::detail::CollectMetrics (collectables=std::vector of length 3, capacity 4 = {...})
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/pull/src/metrics_collector.cc:20
#10 0x00000000015eaf0a in prometheus::detail::MetricsHandler::handleGet (this=0x5a9ae30, conn=0x5b8f560)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/pull/src/handler.cc:149
#11 0x00000000015caae7 in CivetServer::requestHandler (conn=0x5b8f560, cbdata=0x5a9ae30)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/CivetServer.cpp:140
#12 0x00000000015e176f in handle_request (conn=0x5b8f560)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/civetweb.c:14407
#13 0x00000000015e4e3a in process_new_connection (conn=0x5b8f560)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/civetweb.c:18432
#14 0x00000000015e5709 in worker_thread_run (conn=0x5b8f560)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/civetweb.c:18837
#15 0x00000000015e580f in worker_thread (thread_func_param=0x5b8f560)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/3rdparty/civetweb/src/civetweb.c:18892
#16 0x00007f9b93433ea5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f9b9315cb0d in gnu_dev_makedev () from /lib64/libc.so.6
#18 0x0000000000000000 in ?? ()
(gdb) 

(gdb) frame 6
#6  0x00000000015a7369 in prometheus::Family<prometheus::Counter>::Collect (this=0x7f9b48042650)
    at /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/family.cc:102
102     in /share/build_cache/Opensource/CACHE3/platform/build/staging/du/x86/prometheus-cpp-master/core/src/family.cc
(gdb) p this
$3 = (const prometheus::Family<prometheus::Counter> * const) 0x7f9b48042650
(gdb) p *(const prometheus::Family<prometheus::Counter> * const) 0x7f9b48042650
$4 = (prometheus::Family<prometheus::Counter>) {
  <prometheus::Collectable> = {
    _vptr.Collectable = 0x17c7cf0 <vtable for prometheus::Family<prometheus::Counter>+16>
  },
  members of prometheus::Family<prometheus::Counter>:
  metrics_ = std::unordered_map with 32 elements = {
    [std::map with 8 elements = {
      ["CUCPID"] = "851008000",
      ["CellClusterId"] = "111",
      ["DUID"] = "851008010",
      ["GroupName"] = "INTERGNBSRC",
      ["HOTYPE"] = "1",
      ["RUID"] = "27251",
      ["SRCNRCGI"] = "86465872771940547",
      ["TGTNRCGI"] = "86465872771956931"
    }] = std::unique_ptr<prometheus::Counter> = {
      get() = 0x7f9b4b2bde30
    },
....
    [std::map with 8 elements<error reading variable: Cannot access memory at address 0x17f9b484ae518>] = std::unique_ptr<prometheus::Counter> = {
      get() = 0x0
    },
    [std::map with 8 elements = {
      ["CUCPID"] = "851008000",
      ["CellClusterId"] = "111",
      ["DUID"] = "851008008",
      ["GroupName"] = "INTERGNBSRC",
      ["HOTYPE"] = "1",
      ["RUID"] = "23252",
      ["SRCNRCGI"] = "86465872771940506",
      ["TGTNRCGI"] = "86465872771944456"
    }] = std::unique_ptr<prometheus::Counter> = {
      get() = 0x7f9b4b3aa8c0
    },
......

[gjasny]

What's strange is that in stack frame 4 the this pointer is null. That means that Familiy::_metrics stored a null pointer. Normally that cannot happen. Did you disable Exceptions and ran out of memory?

Or this is a use-after-free situation. In that case building with address sanitizer might give some clues (-fsanitize=address).

That's all I can do for you right now.

[ramesht77]

hi,
Trying to create error condition of duplicate entry or invalid label name
template
T& Family::Add(const Labels& labels, std::unique_ptr object) {
std::lock_guardstd::mutex lock{mutex_};

auto insert_result =
metrics_.insert(std::make_pair(labels, std::move(object)));

if (insert_result.second) {
// insertion took place, retroactively check for unlikely issues
for (auto& label_pair : labels) {
const auto& label_name = label_pair.first;
if (!CheckLabelName(label_name)) {
metrics_.erase(insert_result.first);
throw std::invalid_argument("Invalid label name");
}
if (constant_labels_.count(label_name)) {
metrics_.erase(insert_result.first);
throw std::invalid_argument("Duplicate label name");
}
}
}
Using test application
const std::vectorstd::string CounterNames = {
COUNTER_1,
COUNTER_2
};
registerCounters("PrometheusTestCounter", CounterNames, {{ID" ,"1"}, {"DID" ,"1"}, {"UID" ,"1"}, {"NI" ,"1"}, {"Cld" ,"-1"}});
registerCounters("PrometheusTestCounter", CounterNames, {{"ID" ,"1"}, {DID" ,"1"}, {"UID" ,"1"}, {"NI" ,"1"}, {"CId" ,"-1"}});

But its not entering the error area, whats wrong here?

Please suggest.

[gjasny]

Try to add a conflicting constant label when creating the family.

[ramesht77]

pls share an example

[gjasny]

1. Set a breakpoint at the throw
2. Run the prometheus_core_test
3. Find the following:

prometheus-cpp/core/tests/family_test.cc

Lines 32 to 37 in bac0722

	TEST(FamilyTest, reject_same_label_keys) {
	auto labels = Labels{{"component", "test"}};
	Family<Counter> family{"total_requests", "Counts all requests", labels};
	EXPECT_ANY_THROW(family.Add(labels));
	}

[gjasny]

BuildCounter()
    .Name(...)
    .Help(...)
    .Labels({{"a", "A"}}) // here you set the constant labels in real-world use cases
    .Register(registry);