Envoy::Router::PrefixRouteEntryImpl::matches() Caught Segmentation fault, suspect faulting address

[benishak]

Bug Template

envoyproxy/envoy#6348

Title: One line description

Description:
I try to run Envoy on AWS EC2 c5n.9xlarge with Gloo on Kubernetes,
I run it on production to serve 2,000,000+ RPM (http+https)
after running it for 10min, it crashed.

I have 2 listeners

• HTTP
• HTTPS

both are configured with the following routers

• 0: /path1 -> static host
• 1: /path.php -> static host
• 2: / -> dynamic to Kubernetes service

gloo version: 0.11.2
Envoy version: b696ded71901e11cf4b83fb547fe4f7e5a2fdba0/1.10.0-dev/Distribution/RELEASE/BoringSSL

Repro steps:
Configure Envoy with 2 listeners and use similar routing like above and try to send it 200000 RPM/m for 10min+

Call Stack:

[2019-03-21 17:57:40.622][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x0
[2019-03-21 17:57:40.622][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<86> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2019-03-21 17:57:40.625][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #0 0xaf1c30 Envoy::Router::PrefixRouteEntryImpl::matches()
[2019-03-21 17:57:40.627][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #1 0xaf1cc1 Envoy::Router::PrefixRouteEntryImpl::matches()
[2019-03-21 17:57:40.629][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #2 0xaf3cf8 Envoy::Router::VirtualHostImpl::getRouteFromEntries()
[2019-03-21 17:57:40.631][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #3 0xaf7974 Envoy::Router::ConfigImpl::route()
[2019-03-21 17:57:40.633][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #4 0x9cef46 Envoy::Http::ConnectionManagerImpl::ActiveStream::refreshCachedRoute()
[2019-03-21 17:57:40.635][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #5 0x9d2d2e Envoy::Http::ConnectionManagerImpl::ActiveStreamFilterBase::route()
[2019-03-21 17:57:40.637][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #6 0x4793d9 Envoy::Extensions::HttpFilters::Transformation::TransformationFilter::checkResponseActive()
[2019-03-21 17:57:40.639][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #7 0x478fdb Envoy::Extensions::HttpFilters::Transformation::TransformationFilter::encodeHeaders()
[2019-03-21 17:57:40.641][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #8 0x9d096e Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeHeaders()
[2019-03-21 17:57:40.643][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #9 0xb4feb2 Envoy::Http::Utility::sendLocalReply()
[2019-03-21 17:57:40.646][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #10 0x9cd9fc Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply()
[2019-03-21 17:57:40.648][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #11 0x9cd861 Envoy::Http::ConnectionManagerImpl::ActiveStream::onIdleTimeout()
[2019-03-21 17:57:40.650][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #12 0xb97d67 event_process_active_single_queue.isra.29
[2019-03-21 17:57:40.652][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #13 0xb9845e event_process_active
[2019-03-21 17:57:40.654][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #14 0xb9b1e7 event_base_loop
[2019-03-21 17:57:40.656][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #15 0x80bb42 Envoy::Event::DispatcherImpl::run()
[2019-03-21 17:57:40.658][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #16 0x80723e Envoy::Server::WorkerImpl::threadRoutine()
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #17 0xe0b9a4 Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke()
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<86> obj</usr/glibc-compat/lib/libpthread.so.0>
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<86> #18 0x7fa0543d3d2b (unknown)
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<86> obj</usr/glibc-compat/lib/libc.so.6>
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<86> #19 0x7fa053d83e3e (unknown)
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 86

CPU and Load were fine during the run
But I noticed high DIsk Write IO around 5GB before it crashed and disk usage jumped from 2GB to 9.6GB out of total 20GB
Memory usage peak was around 15GB
This instance has around 92GB RAM and 36 Cores

[yuval-k]

looking at the code its hard to see what can be the issue - let me get a new version of envoy-gloo compiled against latest upstream, and then would you mind testing again?

[benishak]

yes sure
Thanks!

[yuval-k]

what envoy was this? the one from gloo open source? or gloo enterprise

[benishak]

gloo open souce

Envoy version: b696ded71901e11cf4b83fb547fe4f7e5a2fdba0/1.10.0-dev/Distribution/RELEASE/BoringSSL

[benishak]

FYI might be helpful
I added routes with glooctl

glooctl add route --name myhttpvs --path-prefix / --dest-name mymainupstream
glooctl add route --name myhttpvs --path-prefix /path1 --dest-name myupstream
glooctl add route --name myhttpvs --path-prefix /path.php --dest-name myupstream

[yuval-k]

i was able to reproduce this; debugging

[yuval-k]

@benishak See the root cause here: envoyproxy/envoy#6348 (comment)
I will submit a patch to envoy to fix this.

[benishak]

Thanks a lot! looking forward for the fix to be released.

[yuval-k]

fix in envoyproxy/envoy#6359

[yuval-k]

i'm incorporating the change into envoy-gloo and will release it soon

[benishak]

awesome, is it part of 0.13 release?

[yuval-k]

didnt make it to 0.13, it will be part of 0.13.1, which i will create promptly.
(this is the PR to merge in the envoy changes: #607)

[yuval-k]

fixed by #607
I released v0.13.1 that contains the fix. feel free to test it out and provide your feedback!