Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flake in DeferredLoggingWithRetransmission #27841

Closed
alyssawilk opened this issue Jun 7, 2023 · 2 comments · Fixed by #27850
Closed

flake in DeferredLoggingWithRetransmission #27841

alyssawilk opened this issue Jun 7, 2023 · 2 comments · Fixed by #27850
Assignees

Comments

@alyssawilk
Copy link
Contributor

https://dev.azure.com/cncf/envoy/_build/results?buildId=139480&view=logs&j=4e2e4662-2d21-5726-b4bc-9ff5cd6a179b&t=da7afaec-274f-5dd3-65ad-8c8f961558bc&l=429

0x1429ac0: Envoy::Quic::QuicHttpIntegrationTest_DeferredLoggingWithRetransmission_Test::TestBody()
0xa406666: testing::internal::HandleExceptionsInMethodIfSupported<>()
0xa3ac4ff: testing::Test::Run()
0xa3b05f8: testing::TestInfo::Run()
... Google Test internal frames ...

test/integration/quic_http_integration_test.cc:1415: Failure
Expected: (std::stoi(metrics.at(1))) > (0), actual: 0 vs 0
Stack trace:
0x1428857: Envoy::Quic::QuicHttpIntegrationTest_DeferredLoggingWithRetransmission_Test::TestBody()
0x1429ac0: Envoy::Quic::QuicHttpIntegrationTest_DeferredLoggingWithRetransmission_Test::TestBody()
0xa406666: testing::internal::HandleExceptionsInMethodIfSupported<>()
0xa3ac4ff: testing::Test::Run()
0xa3b05f8: testing::TestInfo::Run()
... Google Test internal frames ...

on msan

@alyssawilk
Copy link
Contributor Author

[2023-06-06 21:36:08.701][332154][critical][backtrace] [./source/server/backtrace.h:96] #1: abort [0xffffaafccaac]
[2023-06-06 21:36:08.711][332154][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::InjectableSingleton<>::get() [0xa13ab4]
[2023-06-06 21:36:08.720][332154][critical][backtrace] [./source/server/backtrace.h:96] #3: Envoy::Network::Address::Ipv6Instance::Ipv6Instance() [0x16ea108]
[2023-06-06 21:36:08.730][332154][critical][backtrace] [./source/server/backtrace.h:96] #4: Envoy::Network::Address::InstanceFactory::createInstancePtr<>() [0x16e7f10]
[2023-06-06 21:36:08.740][332154][critical][backtrace] [./source/server/backtrace.h:96] #5: Envoy::Network::Address::addressFromSockAddr() [0x16e754c]
[2023-06-06 21:36:08.749][332154][critical][backtrace] [./source/server/backtrace.h:96] #6: Envoy::Network::Address::addressFromSockAddrOrDie() [0x16e82e4]
[2023-06-06 21:36:08.759][332154][critical][backtrace] [./source/server/backtrace.h:96] #7: Envoy::Network::IoSocketHandleImpl::recvmmsg() [0x16c9cd8]
[2023-06-06 21:36:08.769][332154][critical][backtrace] [./source/server/backtrace.h:96] #8: Envoy::Network::Utility::readFromSocket() [0x16c529c]
[2023-06-06 21:36:08.778][332154][critical][backtrace] [./source/server/backtrace.h:96] #9: Envoy::Network::Utility::readPacketsFromSocket() [0x16c636c]
[2023-06-06 21:36:08.788][332154][critical][backtrace] [./source/server/backtrace.h:96] #10: Envoy::Network::UdpListenerImpl::handleReadCallback() [0x15faf94]
[2023-06-06 21:36:08.798][332154][critical][backtrace] [./source/server/backtrace.h:96] #11: Envoy::Network::UdpListenerImpl::onSocketEvent() [0x15fad48]
[2023-06-06 21:36:08.807][332154][critical][backtrace] [./source/server/backtrace.h:96] #12: std::__1::__function::__func<>::operator()() [0x15d9c38]
[2023-06-06 21:36:08.817][332154][critical][backtrace] [./source/server/backtrace.h:96] #13: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x15dae84]
[2023-06-06 21:36:08.827][332154][critical][backtrace] [./source/server/backtrace.h:96] #14: event_process_active_single_queue [0x1a929d0]
[2023-06-06 21:36:08.836][332154][critical][backtrace] [./source/server/backtrace.h:96] #15: event_base_loop [0x1a91270]
[2023-06-06 21:36:08.846][332154][critical][backtrace] [./source/server/backtrace.h:96] #16: Envoy::Server::WorkerImpl::threadRoutine() [0xd313c4]
[2023-06-06 21:36:08.856][332154][critical][backtrace] [./source/server/backtrace.h:96] #17: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x1a89f60]
[2023-06-06 21:36:08.856][332154][critical][backtrace] [./source/server/backtrace.h:96] #18: start_thread [0xffffab126624]

INFO: Elapsed time: 1545.985s, Critical Path: 350.59s
INFO: 31912 processes: 18393 remote cache hit, 11293 internal, 1 local, 2225 processwrapper-sandbox.
//test/integration:quic_http_integration_test FAILED in 1 out of 6 in 15.5s
Stats over 6 runs: max = 15.5s, min = 5.7s, avg = 12.9s, dev = 3.7s
WARNING: //test/integration:quic_http_integration_test: Test execution time (10.6s excluding execution overhead) outside of range for LONG tests. Consider setting timeout="short" or size="small".
/build/bazel_root/base/execroot/envoy/bazel-out/aarch64-opt/testlogs/test/integration/quic_http_integration_test/shard_2_of_6/test.log

https://dev.azure.com/cncf/envoy/_build/results?buildId=139481&view=logs&j=767be981-567e-57d8-68c3-2140ede0a0bd&t=67a76341-37f4-5b49-db38-acbb8a59af82&l=658

@pksohn
Copy link
Contributor

pksohn commented Jun 7, 2023

(Note to self) For the injectable singleton problem, we probably need to add some handling like in this test: https://github.com/envoyproxy/envoy/blob/main/test/integration/protocol_integration_test.cc#L4287.

alyssawilk pushed a commit that referenced this issue Jun 8, 2023
…sion (#27850)

Fixes #27841:

Increase the sleep from 500->1000 ms.
Properly clean up the SocketInterfaceSwap.
Commit Message: Fix flaky test DeferredLoggingWithRetransmission
Risk Level: N/A, just testing

Signed-off-by: Paul Sohn <paulsohn@google.com>
asheryerm pushed a commit to asheryerm/envoy that referenced this issue Jul 5, 2023
…sion (envoyproxy#27850)

Fixes envoyproxy#27841:

Increase the sleep from 500->1000 ms.
Properly clean up the SocketInterfaceSwap.
Commit Message: Fix flaky test DeferredLoggingWithRetransmission
Risk Level: N/A, just testing

Signed-off-by: Paul Sohn <paulsohn@google.com>
Signed-off-by: asheryer <asheryer@amazon.com>
reskin89 pushed a commit to reskin89/envoy that referenced this issue Jul 11, 2023
…sion (envoyproxy#27850)

Fixes envoyproxy#27841:

Increase the sleep from 500->1000 ms.
Properly clean up the SocketInterfaceSwap.
Commit Message: Fix flaky test DeferredLoggingWithRetransmission
Risk Level: N/A, just testing

Signed-off-by: Paul Sohn <paulsohn@google.com>
Signed-off-by: Ryan Eskin <ryan.eskin89@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants