Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crashed when click config_dump #3329

Closed
shuoli84 opened this issue May 9, 2018 · 1 comment
Closed

crashed when click config_dump #3329

shuoli84 opened this issue May 9, 2018 · 1 comment
Labels
duplicate

Comments

@shuoli84
Copy link

shuoli84 commented May 9, 2018

Issue Template

Title: Envoy crashed when click config_dump from admin page

Description:

I started envoy with xds server, after it bootstrapped, then clicked config_dump on admin page, it generates following stack trace.

Version:
envoy version: 08712e9/1.7.0-dev/Modified/DEBUG

  • Config *:
admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 127.0.0.1, port_value: 9902 }

dynamic_resources:
  cds_config:
    api_config_source:
      api_type: GRPC
      cluster_names: [xds_cluster]

  lds_config:
    api_config_source:
      api_type: GRPC
      cluster_names: [xds_cluster]


static_resources:
  clusters:
  - name: xds_cluster
    connect_timeout: 0.25s
    type: STATIC
    lb_policy: ROUND_ROBIN
    http2_protocol_options: {}
    hosts: [
      { socket_address: { address: 10.1.62.43, port_value: 8083 }}
    ]
  • Stack trace *
    [2018-05-09 11:50:07.205][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault: 11, suspect faulting address 0x0
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:87] Backtrace obj thr<0>:
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy 0x000000010ba08026 _ZN5Envoy6Router30RouteConfigProviderManagerImpl29getStaticRouteConfigProvidersEv>
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #0 0x10ba08026:
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Initial import #1 0x10ba0b906: Envoy::Router::RouteConfigProviderManagerImpl::dumpRouteConfigs() + 1222
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Initial import #2 0x10ba1ba0e: Envoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3::operator()() const + 30
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> network filter: fix upstream host storage #3 0x10ba1b9e3: std::__1::__invoke_void_return_wrapper<std::__1::unique_ptr<google::protobuf::Message, std::__1::default_deletegoogle::protobuf::Message > >::__callEnvoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3&(std::__1::unique_ptr<google::protobuf::Message, std::__1::default_deletegoogle::protobuf::Message >, Envoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3&&&) + 51
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Failure reason desc #4 0x10ba1b89f: std::__1::__function::__func<Envoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3, std::__1::allocatorEnvoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3, std::__1::unique_ptr<google::protobuf::Message, std::__1::default_deletegoogle::protobuf::Message > ()>::operator()() + 47
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> More details around failure reasons. #5 0x10be88bf9: std::__1::function<std::__1::unique_ptr<google::protobuf::Message, std::__1::default_deletegoogle::protobuf::Message > ()>::operator()() const + 137
    [2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> fix filter manager crash #6 0x10be885e2: Envoy::Server::AdminImpl::handlerConfigDump(absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) const + 898
    [2018-05-09 11:50:07.229][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> read --build-id from root cmake project during linking #7 0x10bec2df7: Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4::operator()(absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) const + 71
    [2018-05-09 11:50:07.229][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: build/install and misc #8 0x10bec2da2: std::__1::__invoke_void_return_wrapperEnvoy::Http::Code::__call<Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4&, absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&>(Envoy::Http::Code, Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4&&&, absl::string_view&&, Envoy::Http::HeaderMap&&&, Envoy::Buffer::Instance&&&) + 178
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> add x-envoy-upstream-rq-per-try-timeout-ms router header option #9 0x10bec2c38: std::__1::__function::__func<Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4, std::__1::allocator<Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4>, Envoy::Http::Code (absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&)>::operator()(absl::string_view&&, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) + 88
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Add stats for timeouts #10 0x10bea2774: std::__1::function<Envoy::Http::Code (absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&)>::operator()(absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) const + 196
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> ci: do asan build #11 0x10be9cffa: Envoy::Server::AdminImpl::runCallback(absl::string_view, Envoy::Http::HeaderMap const&, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) + 1066
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Refactor async client #12 0x10be82f55: Envoy::Server::AdminFilter::onComplete() + 1013
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: deployment types #13 0x10be82b49: Envoy::Server::AdminFilter::decodeHeaders(Envoy::Http::HeaderMap&, bool) + 57
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: admin #14 0x10bf05959: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter*, Envoy::Http::HeaderMap&, bool) + 1657
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: misc #15 0x10bf02eaa: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::HeaderMap, std::__1::default_deleteEnvoy::Http::HeaderMap >&&, bool) + 11130
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: update landing #16 0x10bf062fa: non-virtual thunk to Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::HeaderMap, std::__1::default_deleteEnvoy::Http::HeaderMap >&&, bool) + 58
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: route matching details #17 0x10bfdd282: Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete() + 418
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: cluster stats #18 0x10bfdd418: non-virtual thunk to Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete() + 24
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: mongo proxy #19 0x10bfe1270: Envoy::Http::Http1::ConnectionImpl::$_7::operator()(http_parser*) const + 32
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Update what_is_envoy.rst #20 0x10bfe1247: Envoy::Http::Http1::ConnectionImpl::$_7::__invoke(http_parser*) + 23
    [2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Fix typo #21 0x10bfeea43: http_parser_execute + 19235
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: SSL -> TLS #22 0x10bfda775: Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long) + 101
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: README and getting help #23 0x10bfda635: Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) + 501
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Health Check Link #24 0x10bfdaad6: virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) + 38
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Change some sentences for clarity #25 0x10befa0fb: Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool) + 1355
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Add docker example #26 0x10b9c6638: Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*) + 1496
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Return x-request-id when tracing enforced. #27 0x10b9c6801: Envoy::Network::FilterManagerImpl::onRead() + 177
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Use runtime key in duration filter. #28 0x10b9b516f: Envoy::Network::ConnectionImpl::onRead(unsigned long long) + 239
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Add tracing to obs on landing page. #29 0x10b9b81a5: Envoy::Network::ConnectionImpl::onReadReady() + 405
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: do ci on docs #30 0x10b9b7fc5: Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) + 789
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Change organization of the HTTP section of Architecture overview #31 0x10b9bef5d: Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, bool)::$_2::operator()(unsigned int) const + 29
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> request shadowing #32 0x10b9bef2e: _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy7Network14ConnectionImplC1ERNS3_5Event10DispatcherEONS_10unique_ptrINS4_16ConnectionSocketENS_14default_deleteISA_EEEEONS9_INS4_15TransportSocketENSB_ISF_EEEEbE3$2jEEEvDpOT + 78
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> optimize getRawSlices() #33 0x10b9bee28: std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) + 56
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Request shadowing already present. #34 0x10b99d457: std::__1::function<void (unsigned int)>::operator()(unsigned int) const + 151
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Example configurations #35 0x10b99d3ac: Envoy::Event::FileEventImpl::assignEvents(unsigned int)::$_0::operator()(int, short, void*) const + 236
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Per zone rq (total, timing, per response code, etc) stats for upstream cluster. #36 0x10b99d2b7: Envoy::Event::FileEventImpl::assignEvents(unsigned int)::$_0::__invoke(int, short, void*) + 55
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> allow fast fail for travis build. #37 0x10c39d1da: event_process_active_single_queue + 1450
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: example configurations #38 0x10c399dad: event_base_loop + 1741
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Differentiate in x-request-id value what made it to be traceable. #39 0x10b9956a1: Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) + 97
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> fixes #40 0x10b8d64ed: Envoy::Server::InstanceImpl::run() + 813
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: fixes #41 0x10b1e8326: Envoy::MainCommonBase::run() + 150
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: sd fix #42 0x10b1b7a1b: Envoy::MainCommon::run() + 27
    [2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
@mattklein123
Copy link
Member

This was fixed by #3199 and is a duplicate of #3141

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shuoli84 @mattklein123