Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

protocol_integration_test failure on ppc64le #12

Closed
svghadi opened this issue Apr 6, 2020 · 2 comments
Closed

protocol_integration_test failure on ppc64le #12

svghadi opened this issue Apr 6, 2020 · 2 comments

Comments

@svghadi
Copy link

svghadi commented Apr 6, 2020

Description:

Test //test/integration:protocol_integration_test failed with exception

[ RUN      ] Protocols/DownstreamProtocolIntegrationTest.ManyRequestHeadersRejected/IPv6_HttpDownstream_HttpUpstream
terminate called after throwing an instance of 'Envoy::Http::CodecProtocolException'
  what():  headers size exceeds limit

Environment:

Arch: ppc64le
Os: rhel 7.6
Gcc: 7.3.1

Console Output:

//test/integration:protocol_integration_test                             FAILED in 3 out of 3 in 181.9s
  Stats over 3 runs: max = 181.9s, min = 122.2s, avg = 147.1s, dev = 25.4s
  /root/.cache/bazel/_bazel_root/eaf5b044df34a5b8a86c3ffe219d3af9/execroot/envoy/bazel-out/ppc-fastbuild/testlogs/test/integration/protocol_integration_test/shard_3_of_3/test.log
  /root/.cache/bazel/_bazel_root/eaf5b044df34a5b8a86c3ffe219d3af9/execroot/envoy/bazel-out/ppc-fastbuild/testlogs/test/integration/protocol_integration_test/shard_2_of_3/test.log
  /root/.cache/bazel/_bazel_root/eaf5b044df34a5b8a86c3ffe219d3af9/execroot/envoy/bazel-out/ppc-fastbuild/testlogs/test/integration/protocol_integration_test/shard_1_of_3/test.log

Logs:

shard_2_of_3.test.log
shard_3_of_3.test.log
shard_1_of_3.test.log
@knrc
Copy link

knrc commented Jun 23, 2020

RHEL 8 is required for building proxy, we have seen no failures on RHEL 8 other than with two of the fuzzing tests which we currently believe is a false positive.

@jwendell
Copy link
Member

Issues for this repository are tracked in Red Hat Jira. Please head to https://issues.redhat.com/browse/OSSM in order to browse or open an issue.

oschaaf pushed a commit that referenced this issue Oct 26, 2022
@benjaminp
test: run fake upstream initialization from the dispatcher thread (#2…
9aa597d 
…2856)

`//test/integration:tcp_proxy_odcds_integration_test` was observed to fail as follows:
```
==================== Test output for //test/integration:tcp_proxy_odcds_integration_test:
[==========] Running 24 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 24 tests from IpVersionsClientType/TcpProxyOdcdsIntegrationTest
[ RUN      ] IpVersionsClientType/TcpProxyOdcdsIntegrationTest.SingleTcpClient/0
[2022-08-25 20:22:46.750][3969][critical][assert] [test/integration/fake_upstream.cc:832] assert failure: !dispatcher_->isThreadSafe().
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x6b00000f81
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x4c46a78]->[0x2cf2a78] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1267
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #1: __restore_rt [0x7ffbdaa79420]->[0x7ffbd8b25420] ??:0
[2022-08-25 20:22:46.802][3969][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::FakeUpstream::assertPendingConnectionsEmpty() [0x245bf0b]->[0x507f0b] ??:0
[2022-08-25 20:22:46.846][3969][critical][backtrace] [./source/server/backtrace.h:96] #3: Envoy::(anonymous namespace)::TcpProxyOdcdsIntegrationTest_SingleTcpClient_Test::TestBody() [0x1f596cb]->[0x56cb] ??:0
[2022-08-25 20:22:46.877][3969][critical][backtrace] [./source/server/backtrace.h:96] #4: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588b61b]->[0x393761b] ??:0
[2022-08-25 20:22:46.924][3969][critical][backtrace] [./source/server/backtrace.h:96] #5: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587c2bd]->[0x39282bd] ??:0
[2022-08-25 20:22:46.966][3969][critical][backtrace] [./source/server/backtrace.h:96] #6: testing::Test::Run() [0x5864ba3]->[0x3910ba3] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:0
[2022-08-25 20:22:47.003][3969][critical][backtrace] [./source/server/backtrace.h:96] #7: testing::TestInfo::Run() [0x586576a]->[0x391176a] external/com_google_absl/absl/container/internal/raw_hash_set.h:1259
[2022-08-25 20:22:47.037][3969][critical][backtrace] [./source/server/backtrace.h:96] #8: testing::TestSuite::Run() [0x5865fbb]->[0x3911fbb] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_algo.h:1925
[2022-08-25 20:22:47.089][3969][critical][backtrace] [./source/server/backtrace.h:96] #9: testing::internal::UnitTestImpl::RunAllTests() [0x5874a28]->[0x3920a28] envoy/registry/registry.h:509
[2022-08-25 20:22:47.114][3969][critical][backtrace] [./source/server/backtrace.h:96] #10: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588ddbb]->[0x3939dbb] envoy/registry/registry.h:0
[2022-08-25 20:22:47.160][3969][critical][backtrace] [./source/server/backtrace.h:96] #11: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587e683]->[0x392a683] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:154
[2022-08-25 20:22:47.191][3969][critical][backtrace] [./source/server/backtrace.h:96] #12: testing::UnitTest::Run() [0x5874568]->[0x3920568] envoy/registry/registry.h:508
[2022-08-25 20:22:47.237][3969][critical][backtrace] [./source/server/backtrace.h:96] #13: RUN_ALL_TESTS() [0x4878d51]->[0x2924d51] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1203
[2022-08-25 20:22:47.284][3969][critical][backtrace] [./source/server/backtrace.h:96] #14: Envoy::TestRunner::RunTests() [0x48783b1]->[0x29243b1] external/com_google_googletest/googlemock/include/gmock/gmock-actions.h:485
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] #15: main [0x4874c3a]->[0x2920c3a] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1181
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] #16: __libc_start_main [0x7ffbda897083]->[0x7ffbd8943083] ??:0
================================================================================
```

This is due to the race described by envoyproxy/envoy#22855. Making sure the dispatcher thread is running before starting the test avoids this problem.

Signed-off-by: Benjamin Peterson <benjamin@engflow.com>
oschaaf pushed a commit that referenced this issue Oct 26, 2022
@benjaminp
test: fix a race in http2 flood integration test (#23094)
39934a7 
This test sends a large number of metadata frames in order to trigger a disconnect. However, it was possible for the disconnect to happen and the connection to be torn down before all the metadata frames had been sent. If that happened, ASAN detected a UAF:
```
==95==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700037e5a0 at pc 0x000004811f9e bp 0x7ffc903af990 sp 0x7ffc903af988
READ of size 8 at 0x60700037e5a0 thread T0
    #0 0x4811f9d in Envoy::IntegrationCodecClient::sendMetadata(Envoy::Http::RequestEncoder&, Envoy::Http::MetadataMap) /proc/self/cwd/test/integration/http_integration.cc:168:3
    #1 0x46ed711 in Envoy::Http2FloodMitigationTest_RequestMetadata_Test::TestBody() /proc/self/cwd/test/integration/http2_flood_integration_test.cc:1486:20
    #2 0xd380e64 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2580:10
    #3 0xd348dc2 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2655:5
    #4 0xd34a927 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2832:11
    #5 0xd34ccc4 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2986:28
    #6 0xd36f07a in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5697:44
    #7 0xd384e63 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/
gtest.cc:2580:10
    #8 0xd36dd86 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5280:10
    #9 0xa0e53a4 in Envoy::TestRunner::RunTests(int, char**) /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2485:46
    #10 0xa0e0af7 in main /proc/self/cwd/test/main.cc:34:10
    #11 0x7f442ef69082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
    #12 0x45ed36d in _start (/mnt/ssd/cas/work/1/exec/bazel-out/k8-dbg/bin/test/integration/http2_flood_integration_test.runfiles/envoy/test/integration/http2_flood_integration_test+0x45ed36d)

0x60700037e5a0 is located 48 bytes inside of 80-byte region [0x60700037e570,0x60700037e5c0)
freed by thread T0 here:
    #0 0x466f7d2 in free /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3
    #1 0x831dde8 in Envoy::Http::CodecClient::ActiveRequest::~ActiveRequest() /proc/self/cwd/./source/common/http/codec_client.h:220:10
    #2 0x5aa33f9 in std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> >::reset(Envoy::Event::DeferredDeletable*) /opt/llvm/bin/../include/c++/v1/__memory/unique_ptr.h:54:5
    #3 0xa3218e8 in Envoy::Event::DispatcherImpl::clearDeferredDeleteList() /proc/self/cwd/source/common/event/dispatcher_impl.cc:142:21
    #4 0xa3348df in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Thread::ThreadFactory&, Envoy::TimeSource&
, Envoy::Random::RandomGenerator&, Envoy::Filesystem::Instance&, Envoy::Event::TimeSystem&, std::__1::function<std::__1::unique_ptr<Envoy::Event::ScaledRangeTimerManager, std::__1::default_delete<Envoy::Event::ScaledRangeTimerManager> > (Envoy::Event::Dispatcher&)> const&
, std::__1::shared_ptr<Envoy::Buffer::WatermarkFactory> const&)::$_2&>(Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Thread::ThreadFactory&, Envoy::TimeSource&, Envoy::Rand
om::RandomGenerator&, Envoy::Filesystem::Instance&, Envoy::Event::TimeSystem&, std::__1::function<std::__1::unique_ptr<Envoy::Event::ScaledRangeTimerManager, std::__1::default_delete<Envoy::Event::ScaledRangeTimerManager> > (Envoy::Event::Dispatcher&)> const&, std::__1::s
hared_ptr<Envoy::Buffer::WatermarkFactory> const&)::$_2&) /proc/self/cwd/source/common/event/dispatcher_impl.cc:79:30
    #5 0xa334603 in std::__1::__function::__func<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Thread::ThreadFactory&, Envoy::TimeSource&, Envoy::Random::RandomGenerator&,
Envoy::Filesystem::Instance&, Envoy::Event::TimeSystem&, std::__1::function<std::__1::unique_ptr<Envoy::Event::ScaledRangeTimerManager, std::__1::default_delete<Envoy::Event::ScaledRangeTimerManager> > (Envoy::Event::Dispatcher&)> const&, std::__1::shared_ptr<Envoy::Buffe
r::WatermarkFactory> const&)::$_2, std::__1::allocator<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Thread::ThreadFactory&, Envoy::TimeSource&, Envoy::Random::RandomGenera
tor&, Envoy::Filesystem::Instance&, Envoy::Event::TimeSystem&, std::__1::function<std::__1::unique_ptr<Envoy::Event::ScaledRangeTimerManager, std::__1::default_delete<Envoy::Event::ScaledRangeTimerManager> > (Envoy::Event::Dispatcher&)> const&, std::__1::shared_ptr<Envoy:
:Buffer::WatermarkFactory> const&)::$_2>, void ()>::operator()() /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #6 0x4897039 in std::__1::__function::__value_func<void ()>::operator()() const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #7 0xa8e6aa4 in Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl(Envoy::CSmartPtr<event_base, &(event_base_free)>&, std::__1::function<void ()>)::$_0::__invoke(int, short, void*) /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #8 0xb557c5e in event_process_active_single_queue /mnt/ssd/cas/work/2/exec/external/com_github_libevent_libevent/event.c:1713:4
    #9 0xb539252 in event_process_active /mnt/ssd/cas/work/2/exec/external/com_github_libevent_libevent/event.c
    #10 0xb539252 in event_base_loop /mnt/ssd/cas/work/2/exec/external/com_github_libevent_libevent/event.c:2047:12
    #11 0xa8e1e3c in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/libevent_scheduler.cc:60:3
    #12 0xa32bd94 in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/dispatcher_impl.cc:299:19
    #13 0x480faad in Envoy::IntegrationCodecClient::flushWrite() /proc/self/cwd/test/integration/http_integration.cc:100:29
    #14 0x4811e94 in Envoy::IntegrationCodecClient::sendMetadata(Envoy::Http::RequestEncoder&, Envoy::Http::MetadataMap) /proc/self/cwd/test/integration/http_integration.cc:169:3
    #15 0x46ed711 in Envoy::Http2FloodMitigationTest_RequestMetadata_Test::TestBody() /proc/self/cwd/test/integration/http2_flood_integration_test.cc:1486:20
    #16 0xd380e64 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2580:10
    #17 0xd348dc2 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2655:5
    #18 0xd34a927 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2832:11
    #19 0xd34ccc4 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2986:28
    #20 0xd36f07a in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5697:44
    #21 0xd384e63 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2580:10
    #22 0xd36dd86 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5280:10
    #23 0xa0e53a4 in Envoy::TestRunner::RunTests(int, char**) /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2485:46
    #24 0xa0e0af7 in main /proc/self/cwd/test/main.cc:34:10
    #25 0x7f442ef69082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
```

To fix that, write all metadata frames at once.

Signed-off-by: Benjamin Peterson <benjamin@engflow.com>
oschaaf pushed a commit that referenced this issue Oct 27, 2022
@benjaminp
test: run fake upstream initialization from the dispatcher thread (#2…
97af8e5 
…2856)

`//test/integration:tcp_proxy_odcds_integration_test` was observed to fail as follows:
```
==================== Test output for //test/integration:tcp_proxy_odcds_integration_test:
[==========] Running 24 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 24 tests from IpVersionsClientType/TcpProxyOdcdsIntegrationTest
[ RUN      ] IpVersionsClientType/TcpProxyOdcdsIntegrationTest.SingleTcpClient/0
[2022-08-25 20:22:46.750][3969][critical][assert] [test/integration/fake_upstream.cc:832] assert failure: !dispatcher_->isThreadSafe().
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x6b00000f81
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x4c46a78]->[0x2cf2a78] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1267
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #1: __restore_rt [0x7ffbdaa79420]->[0x7ffbd8b25420] ??:0
[2022-08-25 20:22:46.802][3969][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::FakeUpstream::assertPendingConnectionsEmpty() [0x245bf0b]->[0x507f0b] ??:0
[2022-08-25 20:22:46.846][3969][critical][backtrace] [./source/server/backtrace.h:96] #3: Envoy::(anonymous namespace)::TcpProxyOdcdsIntegrationTest_SingleTcpClient_Test::TestBody() [0x1f596cb]->[0x56cb] ??:0
[2022-08-25 20:22:46.877][3969][critical][backtrace] [./source/server/backtrace.h:96] #4: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588b61b]->[0x393761b] ??:0
[2022-08-25 20:22:46.924][3969][critical][backtrace] [./source/server/backtrace.h:96] #5: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587c2bd]->[0x39282bd] ??:0
[2022-08-25 20:22:46.966][3969][critical][backtrace] [./source/server/backtrace.h:96] #6: testing::Test::Run() [0x5864ba3]->[0x3910ba3] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:0
[2022-08-25 20:22:47.003][3969][critical][backtrace] [./source/server/backtrace.h:96] #7: testing::TestInfo::Run() [0x586576a]->[0x391176a] external/com_google_absl/absl/container/internal/raw_hash_set.h:1259
[2022-08-25 20:22:47.037][3969][critical][backtrace] [./source/server/backtrace.h:96] #8: testing::TestSuite::Run() [0x5865fbb]->[0x3911fbb] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_algo.h:1925
[2022-08-25 20:22:47.089][3969][critical][backtrace] [./source/server/backtrace.h:96] #9: testing::internal::UnitTestImpl::RunAllTests() [0x5874a28]->[0x3920a28] envoy/registry/registry.h:509
[2022-08-25 20:22:47.114][3969][critical][backtrace] [./source/server/backtrace.h:96] #10: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588ddbb]->[0x3939dbb] envoy/registry/registry.h:0
[2022-08-25 20:22:47.160][3969][critical][backtrace] [./source/server/backtrace.h:96] #11: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587e683]->[0x392a683] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:154
[2022-08-25 20:22:47.191][3969][critical][backtrace] [./source/server/backtrace.h:96] #12: testing::UnitTest::Run() [0x5874568]->[0x3920568] envoy/registry/registry.h:508
[2022-08-25 20:22:47.237][3969][critical][backtrace] [./source/server/backtrace.h:96] #13: RUN_ALL_TESTS() [0x4878d51]->[0x2924d51] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1203
[2022-08-25 20:22:47.284][3969][critical][backtrace] [./source/server/backtrace.h:96] #14: Envoy::TestRunner::RunTests() [0x48783b1]->[0x29243b1] external/com_google_googletest/googlemock/include/gmock/gmock-actions.h:485
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] #15: main [0x4874c3a]->[0x2920c3a] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1181
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] #16: __libc_start_main [0x7ffbda897083]->[0x7ffbd8943083] ??:0
================================================================================
```

This is due to the race described by envoyproxy/envoy#22855. Making sure the dispatcher thread is running before starting the test avoids this problem.

Signed-off-by: Benjamin Peterson <benjamin@engflow.com>
oschaaf pushed a commit that referenced this issue Oct 27, 2022
@benjaminp
test: fix a race in http2 flood integration test (#23094)
860c607 
This test sends a large number of metadata frames in order to trigger a disconnect. However, it was possible for the disconnect to happen and the connection to be torn down before all the metadata frames had been sent. If that happened, ASAN detected a UAF:
```
==95==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700037e5a0 at pc 0x000004811f9e bp 0x7ffc903af990 sp 0x7ffc903af988
READ of size 8 at 0x60700037e5a0 thread T0
    #0 0x4811f9d in Envoy::IntegrationCodecClient::sendMetadata(Envoy::Http::RequestEncoder&, Envoy::Http::MetadataMap) /proc/self/cwd/test/integration/http_integration.cc:168:3
    #1 0x46ed711 in Envoy::Http2FloodMitigationTest_RequestMetadata_Test::TestBody() /proc/self/cwd/test/integration/http2_flood_integration_test.cc:1486:20
    #2 0xd380e64 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2580:10
    #3 0xd348dc2 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2655:5
    #4 0xd34a927 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2832:11
    #5 0xd34ccc4 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2986:28
    #6 0xd36f07a in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5697:44
    #7 0xd384e63 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/
gtest.cc:2580:10
    #8 0xd36dd86 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5280:10
    #9 0xa0e53a4 in Envoy::TestRunner::RunTests(int, char**) /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2485:46
    #10 0xa0e0af7 in main /proc/self/cwd/test/main.cc:34:10
    #11 0x7f442ef69082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
    #12 0x45ed36d in _start (/mnt/ssd/cas/work/1/exec/bazel-out/k8-dbg/bin/test/integration/http2_flood_integration_test.runfiles/envoy/test/integration/http2_flood_integration_test+0x45ed36d)

0x60700037e5a0 is located 48 bytes inside of 80-byte region [0x60700037e570,0x60700037e5c0)
freed by thread T0 here:
    #0 0x466f7d2 in free /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3
    #1 0x831dde8 in Envoy::Http::CodecClient::ActiveRequest::~ActiveRequest() /proc/self/cwd/./source/common/http/codec_client.h:220:10
    #2 0x5aa33f9 in std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> >::reset(Envoy::Event::DeferredDeletable*) /opt/llvm/bin/../include/c++/v1/__memory/unique_ptr.h:54:5
    #3 0xa3218e8 in Envoy::Event::DispatcherImpl::clearDeferredDeleteList() /proc/self/cwd/source/common/event/dispatcher_impl.cc:142:21
    #4 0xa3348df in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Thread::ThreadFactory&, Envoy::TimeSource&
, Envoy::Random::RandomGenerator&, Envoy::Filesystem::Instance&, Envoy::Event::TimeSystem&, std::__1::function<std::__1::unique_ptr<Envoy::Event::ScaledRangeTimerManager, std::__1::default_delete<Envoy::Event::ScaledRangeTimerManager> > (Envoy::Event::Dispatcher&)> const&
, std::__1::shared_ptr<Envoy::Buffer::WatermarkFactory> const&)::$_2&>(Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Thread::ThreadFactory&, Envoy::TimeSource&, Envoy::Rand
om::RandomGenerator&, Envoy::Filesystem::Instance&, Envoy::Event::TimeSystem&, std::__1::function<std::__1::unique_ptr<Envoy::Event::ScaledRangeTimerManager, std::__1::default_delete<Envoy::Event::ScaledRangeTimerManager> > (Envoy::Event::Dispatcher&)> const&, std::__1::s
hared_ptr<Envoy::Buffer::WatermarkFactory> const&)::$_2&) /proc/self/cwd/source/common/event/dispatcher_impl.cc:79:30
    #5 0xa334603 in std::__1::__function::__func<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Thread::ThreadFactory&, Envoy::TimeSource&, Envoy::Random::RandomGenerator&,
Envoy::Filesystem::Instance&, Envoy::Event::TimeSystem&, std::__1::function<std::__1::unique_ptr<Envoy::Event::ScaledRangeTimerManager, std::__1::default_delete<Envoy::Event::ScaledRangeTimerManager> > (Envoy::Event::Dispatcher&)> const&, std::__1::shared_ptr<Envoy::Buffe
r::WatermarkFactory> const&)::$_2, std::__1::allocator<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Thread::ThreadFactory&, Envoy::TimeSource&, Envoy::Random::RandomGenera
tor&, Envoy::Filesystem::Instance&, Envoy::Event::TimeSystem&, std::__1::function<std::__1::unique_ptr<Envoy::Event::ScaledRangeTimerManager, std::__1::default_delete<Envoy::Event::ScaledRangeTimerManager> > (Envoy::Event::Dispatcher&)> const&, std::__1::shared_ptr<Envoy:
:Buffer::WatermarkFactory> const&)::$_2>, void ()>::operator()() /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #6 0x4897039 in std::__1::__function::__value_func<void ()>::operator()() const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #7 0xa8e6aa4 in Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl(Envoy::CSmartPtr<event_base, &(event_base_free)>&, std::__1::function<void ()>)::$_0::__invoke(int, short, void*) /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #8 0xb557c5e in event_process_active_single_queue /mnt/ssd/cas/work/2/exec/external/com_github_libevent_libevent/event.c:1713:4
    #9 0xb539252 in event_process_active /mnt/ssd/cas/work/2/exec/external/com_github_libevent_libevent/event.c
    #10 0xb539252 in event_base_loop /mnt/ssd/cas/work/2/exec/external/com_github_libevent_libevent/event.c:2047:12
    #11 0xa8e1e3c in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/libevent_scheduler.cc:60:3
    #12 0xa32bd94 in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/dispatcher_impl.cc:299:19
    #13 0x480faad in Envoy::IntegrationCodecClient::flushWrite() /proc/self/cwd/test/integration/http_integration.cc:100:29
    #14 0x4811e94 in Envoy::IntegrationCodecClient::sendMetadata(Envoy::Http::RequestEncoder&, Envoy::Http::MetadataMap) /proc/self/cwd/test/integration/http_integration.cc:169:3
    #15 0x46ed711 in Envoy::Http2FloodMitigationTest_RequestMetadata_Test::TestBody() /proc/self/cwd/test/integration/http2_flood_integration_test.cc:1486:20
    #16 0xd380e64 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2580:10
    #17 0xd348dc2 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2655:5
    #18 0xd34a927 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2832:11
    #19 0xd34ccc4 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2986:28
    #20 0xd36f07a in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5697:44
    #21 0xd384e63 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2580:10
    #22 0xd36dd86 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5280:10
    #23 0xa0e53a4 in Envoy::TestRunner::RunTests(int, char**) /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2485:46
    #24 0xa0e0af7 in main /proc/self/cwd/test/main.cc:34:10
    #25 0x7f442ef69082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
```

To fix that, write all metadata frames at once.

Signed-off-by: Benjamin Peterson <benjamin@engflow.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jwendell @knrc @svghadi