TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved flake

[alyssawilk]

https://dev.azure.com/cncf/envoy/_build/results?buildId=114045&view=logs&j=767be981-567e-57d8-68c3-2140ede0a0bd&t=1fb0b72f-6293-562d-a4d1-bb649223eaca&l=2029

[ RUN ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/3
./test/integration/server.h:494: Failure
Value of: TestUtility::waitForGaugeEq(statStore(), name, value, time_system_, timeout)
Actual: false (timed out waiting for listener_manager.total_filter_chains_draining to be 1, current value 0)
Expected: true
Stack trace:
0x9a85a0: Envoy::(anonymous namespace)::LdsStsIntegrationTest_TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved_Test::TestBody()
0x1a2a420: testing::internal::HandleExceptionsInMethodIfSupported<>()
0x1a2a2c0: testing::Test::Run()
0x1a2b454: testing::TestInfo::Run()
... Google Test internal frames ...

[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_no_extension_lookup_by_name to: true
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[ FAILED ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/3, where GetParam() = (4-byte object <01-00 00-00>, true) (10143 ms)
[----------] 4 tests from IpVersionsAndMatcher/LdsStsIntegrationTest (10750 ms total)

[----------] Global test environment tear-down
[==========] 44 tests from 7 test suites ran. (22970 ms total)
[ PASSED ] 43 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/3, where GetParam() = (4-byte object <01-00 00-00>, true)

@kyessenov looks like this was an addition from @lambdai , any chance you could take a look?

[kyessenov]

I run it with --runs_per_test=10000 with no luck.

[alyssawilk]

https://dev.azure.com/cncf/envoy/_build/results?buildId=130299&view=logs&j=b840a642-5ff3-5357-2e4b-e06e40b0cffd&t=67965174-5100-5631-9dc0-68b9f0aacb53&l=2208

[ RUN ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/3
.\test/integration/server.h(450): error: Value of: TestUtility::waitForGaugeEq(statStore(), name, value, time_system_, timeout)
Actual: false (timed out waiting for listener_manager.total_filter_chains_draining to be 1, current value 0)
Expected: true
Stack trace:
00007FF73130C71C: (unknown)
00007FF73130C606: (unknown)
00007FF731325AF8: (unknown)
00007FF731325CBD: (unknown)
00007FF73132672D: (unknown)
... Google Test internal frames ...

[alyssawilk]

https://dev.azure.com/cncf/envoy/_build/results?buildId=131741&view=logs&j=767be981-567e-57d8-68c3-2140ede0a0bd&t=8a34f9ed-4946-5fcf-7cd6-99c93070884d&l=402

not getting the detailed logging :-(

[alyssawilk]

https://dev.azure.com/cncf/envoy/_build/results?buildId=131894&view=logs&j=767be981-567e-57d8-68c3-2140ede0a0bd&t=8a34f9ed-4946-5fcf-7cd6-99c93070884d&l=402

[alyssawilk]

still flaking on arm. ARGH

[alyssawilk]

oh nm, was on old branch. may be ok?

[jmarantz]

I think this is still broken; I am having trouble getting a PR through CI and synced today and got this in envoy-presubmit (linux_arm64 Build and test)

I am wondering if some timeouts need to be increased due to Arm tests running slowly in CI.

2023-03-28T12:59:26.3882218Z [ RUN      ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/1
2023-03-28T12:59:26.3883204Z ./test/integration/server.h:450: Failure
2023-03-28T12:59:26.3884094Z Value of: TestUtility::waitForGaugeEq(statStore(), name, value, time_system_, timeout)
2023-03-28T12:59:26.3885159Z   Actual: false (timed out waiting for listener_manager.total_filter_chains_draining to be 1, current value 0)
2023-03-28T12:59:26.3886349Z Expected: true
2023-03-28T12:59:26.3887019Z Stack trace:
2023-03-28T12:59:26.3888058Z   0xa3f914: Envoy::(anonymous namespace)::LdsStsIntegrationTest_TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved_Test::TestBody()
2023-03-28T12:59:26.3889270Z   0x1bf218c: testing::internal::HandleExceptionsInMethodIfSupported<>()
2023-03-28T12:59:26.3890146Z   0x1bf202c: testing::Test::Run()
2023-03-28T12:59:26.3890937Z   0x1bf31c0: testing::TestInfo::Run()
2023-03-28T12:59:26.3891678Z ... Google Test internal frames ...
2023-03-28T12:59:26.3892111Z 
2023-03-28T12:59:26.3893065Z [external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_no_extension_lookup_by_name to: true
2023-03-28T12:59:26.3894437Z [external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
2023-03-28T12:59:26.3896265Z [  FAILED  ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/1, where GetParam() = (4-byte object <00-00 00-00>, true) (10175

[lizan]

broken in TSAN as well #26418