prometheus cause large reactor stalls #2475
Comments
|
I think this is before the optimization change we did in the metrics and
the prometheus layer
…On Mon, Jun 12, 2017 at 1:06 PM, Gleb Natapov ***@***.***> wrote:
*Installation details*
Scylla version (or git commit hash): 1.7
backtrace_buffer::append_backtrace() at /home/avi/urchin/seastar/core/reactor.cc:281
(inlined by) print_with_backtrace at /home/avi/urchin/seastar/core/reactor.cc:292
reactor::block_notifier(int) at /home/avi/urchin/seastar/core/reactor.cc:553
?? ??:0
memory::cpu_pages::allocate_small(unsigned int) at /home/avi/urchin/seastar/core/memory.cc:683
memory::allocate(unsigned long) at /home/avi/urchin/seastar/core/memory.cc:1183
operator new[](unsigned long) at /home/avi/urchin/seastar/core/memory.cc:1551
std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) at ??:?
char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) at /usr/include/c++/5/bits/basic_string.tcc:578
std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&) at ??:?
basic_sstring<char, unsigned int, 15u>::operator std::string() const at /home/avi/urchin/seastar/./core/sstring.hh:257
(inlined by) add_label at /home/avi/urchin/seastar/core/prometheus.cc:83
fill_metric at /home/avi/urchin/seastar/core/prometheus.cc:92
(inlined by) operator() at /home/avi/urchin/seastar/core/prometheus.cc:158
apply at /home/avi/urchin/seastar/./core/apply.hh:34
(inlined by) apply<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()> > at /home/avi/urchin/seastar/./core/apply.hh:42
(inlined by) apply<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()> > at /home/avi/urchin/seastar/./core/future.hh:1279
(inlined by) operator()<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()>, future_state<> > at /home/avi/urchin/seastar/./core/future.hh:884
(inlined by) operator()<future_state<> > at /home/avi/urchin/seastar/./core/scheduling.hh:157
(inlined by) run at /home/avi/urchin/seastar/./core/future.hh:391
reactor::run_tasks(reactor::task_queue&) at /home/avi/urchin/seastar/core/reactor.cc:2270
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2634
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2620
(inlined by) reactor::run() at /home/avi/urchin/seastar/core/reactor.cc:2816
operator() at /home/avi/urchin/seastar/core/reactor.cc:3742
std::function<void ()>::operator()() const at /usr/include/c++/5/functional:2267
(inlined by) posix_thread::start_routine(void*) at /home/avi/urchin/seastar/core/posix.cc:50
backtrace_buffer::append_backtrace() at /home/avi/urchin/seastar/core/reactor.cc:281
(inlined by) print_with_backtrace at /home/avi/urchin/seastar/core/reactor.cc:292
reactor::block_notifier(int) at /home/avi/urchin/seastar/core/reactor.cc:553
?? ??:0
io::prometheus::client::LabelPair::ByteSize() const at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:568
int google::protobuf::internal::WireFormatLite::MessageSizeNoVirtual<io::prometheus::client::LabelPair>(io::prometheus::client::LabelPair const&) at /usr/include/google/protobuf/wire_format_lite_inl.h:764
(inlined by) io::prometheus::client::Metric::ByteSize() const at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:2717
int google::protobuf::internal::WireFormatLite::MessageSizeNoVirtual<io::prometheus::client::Metric>(io::prometheus::client::Metric const&) at /usr/include/google/protobuf/wire_format_lite_inl.h:764
(inlined by) io::prometheus::client::MetricFamily::ByteSize() const at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:3097
write_delimited_to at /home/avi/urchin/seastar/core/prometheus.cc:48
(inlined by) operator() at /home/avi/urchin/seastar/core/prometheus.cc:160
apply at /home/avi/urchin/seastar/./core/apply.hh:34
(inlined by) apply<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()> > at /home/avi/urchin/seastar/./core/apply.hh:42
(inlined by) apply<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()> > at /home/avi/urchin/seastar/./core/future.hh:1279
(inlined by) operator()<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()>, future_state<> > at /home/avi/urchin/seastar/./core/future.hh:884
(inlined by) operator()<future_state<> > at /home/avi/urchin/seastar/./core/scheduling.hh:157
(inlined by) run at /home/avi/urchin/seastar/./core/future.hh:391
reactor::run_tasks(reactor::task_queue&) at /home/avi/urchin/seastar/core/reactor.cc:2270
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2634
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2620
(inlined by) reactor::run() at /home/avi/urchin/seastar/core/reactor.cc:2816
operator() at /home/avi/urchin/seastar/core/reactor.cc:3742
std::function<void ()>::operator()() const at /usr/include/c++/5/functional:2267
(inlined by) posix_thread::start_routine(void*) at /home/avi/urchin/seastar/core/posix.cc:50
backtrace_buffer::append_backtrace() at /home/avi/urchin/seastar/core/reactor.cc:281
(inlined by) print_with_backtrace at /home/avi/urchin/seastar/core/reactor.cc:292
reactor::block_notifier(int) at /home/avi/urchin/seastar/core/reactor.cc:553
?? ??:0
memory::free(void*, unsigned long) at /home/avi/urchin/seastar/core/memory.cc:1223
io::prometheus::client::LabelPair::SharedDtor() at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:408
io::prometheus::client::LabelPair::~LabelPair() at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:403
io::prometheus::client::LabelPair::~LabelPair() at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:404
google::protobuf::internal::GenericTypeHandler<io::prometheus::client::LabelPair>::Delete(io::prometheus::client::LabelPair*) at /usr/include/google/protobuf/repeated_field.h:336
(inlined by) void google::protobuf::internal::RepeatedPtrFieldBase::Destroy<google::protobuf::RepeatedPtrField<io::prometheus::client::LabelPair>::TypeHandler>() at /usr/include/google/protobuf/repeated_field.h:812
_ZN6google8protobuf16RepeatedPtrFieldIN2io10prometheus6client9LabelPairEED4Ev at /usr/include/google/protobuf/repeated_field.h:1019
(inlined by) io::prometheus::client::Metric::~Metric() at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:2378
io::prometheus::client::Metric::~Metric() at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:2380
google::protobuf::internal::GenericTypeHandler<io::prometheus::client::Metric>::Delete(io::prometheus::client::Metric*) at /usr/include/google/protobuf/repeated_field.h:336
(inlined by) void google::protobuf::internal::RepeatedPtrFieldBase::Destroy<google::protobuf::RepeatedPtrField<io::prometheus::client::Metric>::TypeHandler>() at /usr/include/google/protobuf/repeated_field.h:812
_ZN6google8protobuf16RepeatedPtrFieldIN2io10prometheus6client6MetricEED4Ev at /usr/include/google/protobuf/repeated_field.h:1019
(inlined by) io::prometheus::client::MetricFamily::~MetricFamily() at /home/avi/urchin/seastar/build/release/gen/proto/metrics2.pb.cc:2841
operator() at /home/avi/urchin/seastar/core/prometheus.cc:151 (discriminator 3)
apply at /home/avi/urchin/seastar/./core/apply.hh:34
(inlined by) apply<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()> > at /home/avi/urchin/seastar/./core/apply.hh:42
(inlined by) apply<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()> > at /home/avi/urchin/seastar/./core/future.hh:1279
(inlined by) operator()<prometheus::start(httpd::http_server_control&, prometheus::config)::<lambda(httpd::routes&)>::<lambda(std::unique_ptr<httpd::request>, std::unique_ptr<httpd::reply>)>::<lambda(auto:40&)> mutable [with auto:40 = std::vector<std::unordered_map<seastar::metrics::impl::metric_id, seastar::metrics::impl::metric_value> >]::<lambda()>, future_state<> > at /home/avi/urchin/seastar/./core/future.hh:884
(inlined by) operator()<future_state<> > at /home/avi/urchin/seastar/./core/scheduling.hh:157
(inlined by) run at /home/avi/urchin/seastar/./core/future.hh:391
reactor::run_tasks(reactor::task_queue&) at /home/avi/urchin/seastar/core/reactor.cc:2270
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2634
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2620
(inlined by) reactor::run() at /home/avi/urchin/seastar/core/reactor.cc:2816
operator() at /home/avi/urchin/seastar/core/reactor.cc:3742
std::function<void ()>::operator()() const at /usr/include/c++/5/functional:2267
(inlined by) posix_thread::start_routine(void*) at /home/avi/urchin/seastar/core/posix.cc:50
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#2475>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACBRv6tOA6cVoqpetlLBP6oi3irzm7X4ks5sDQ25gaJpZM4N252d>
.
|
|
I think they are backported here, but not sure. What commits are those?
…--
Gleb.
|
|
… On Mon, Jun 12, 2017 at 4:43 PM, Gleb Natapov ***@***.***> wrote:
I think they are backported here, but not sure. What commits are those?
--
Gleb.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#2475 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACBRv2xARDti261-m8KCBWpWtDGOQ0j5ks5sDUCGgaJpZM4N252d>
.
|
|
On Mon, Jun 12, 2017 at 06:50:41AM -0700, Amnon Heiman wrote:
scylladb/seastar@2a0b315
No, this one is not on that branch.
…--
Gleb.
|
|
On Mon, Jun 12, 2017 at 4:54 PM, Gleb Natapov <notifications@github.com>
wrote:
On Mon, Jun 12, 2017 at 06:50:41AM -0700, Amnon Heiman wrote:
> https://github.com/scylladb/seastar/commit/
2a0b315
>
No, this one is not on that branch.
it probably should (it's both unsafe and would have performance
improvement)
… --
Gleb.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#2475 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACBRv9EbNpQKZ_iP5BIXkkJqtvoOBR2_ks5sDUMYgaJpZM4N252d>
.
|
|
@amnonh - can we close this - or do we need to wait for your work on using streams |
avikivity
added a commit
that referenced
this issue
Aug 10, 2017
* seastar 867b7c7...b9f4568 (4): > http: removed unneeded lamda captures > Merge "Prometheus to use output stream" from Amnon > http_test: Fix an http output stream test > Merge "Add output stream to http message reply" from Amnon Fixes #2475
|
Fixed by fc683c3 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Installation details
Scylla version (or git commit hash): 1.7
The text was updated successfully, but these errors were encountered: