flaky test test_send_remote_failover_sync_non_transport_exception_error

[xrmx]

Describe the bug:

Cannot reproduce reliably still may be useful to fix:

=================================== FAILURES ===================================
_________ test_send_remote_failover_sync_non_transport_exception_error _________

should_try = <MagicMock name='should_try' id='140274647588368'>
http_send = <MagicMock name='send' id='140274647596096'>
caplog = <_pytest.logging.LogCaptureFixture object at 0x7f943c8a5c90>

    @mock.patch("elasticapm.transport.http.Transport.send")
    @mock.patch("elasticapm.transport.base.TransportState.should_try")
    def test_send_remote_failover_sync_non_transport_exception_error(should_try, http_send, caplog):
        should_try.return_value = True

        client = Client(
            server_url="http://example.com",
            service_name="app_name",
            secret_token="secret",
            transport_class="elasticapm.transport.http.Transport",
            metrics_interval="0ms",
            metrics_sets=[],
        )
        # test error
        http_send.side_effect = ValueError("oopsie")
        with caplog.at_level("ERROR", "elasticapm.transport"):
            client.capture_message("foo", handled=False)
        client._transport.flush()
        assert client._transport.state.did_fail()
        assert_any_record_contains(caplog.records, "oopsie", "elasticapm.transport")

        # test recovery
        http_send.side_effect = None
        client.capture_message("foo", handled=False)
        client.close()
>       assert not client._transport.state.did_fail()
E       assert not True
E        +  where True = <bound method TransportState.did_fail of <elasticapm.transport.base.TransportState object at 0x7f943c8ddb40>>()
E        +    where <bound method TransportState.did_fail of <elasticapm.transport.base.TransportState object at 0x7f943c8ddb40>> = <elasticapm.transport.base.TransportState object at 0x7f943c8ddb40>.did_fail
E        +      where <elasticapm.transport.base.TransportState object at 0x7f943c8ddb40> = <elasticapm.transport.http.Transport object at 0x7f943c8ddae0>.state
E        +        where <elasticapm.transport.http.Transport object at 0x7f943c8ddae0> = <elasticapm.base.Client object at 0x7f943c8a5c30>._transport

tests/client/client_tests.py:288: AssertionError
----------------------------- Captured stderr call -----------------------------
{"@timestamp":"2024-03-06T16:09:14.917Z","log.level":"error","message":"Failed to submit message: 'oopsie'","ecs":{"version":"1.6.0"},"log":{"logger":"elasticapm.transport","origin":{"file":{"line":318,"name":"base.py"},"function":"handle_transport_fail"},"original":"Failed to submit message: 'oopsie'"},"process":{"name":"MainProcess","pid":905083,"thread":{"id":140274711856704,"name":"eapm event processor thread"}},"service":{"name":"app_name"}}
{"@timestamp":"2024-03-06T16:09:14.917Z","log.level":"error","message":"Failed to submit message: 'oopsie'","ecs":{"version":"1.6.0"},"log":{"logger":"elasticapm.transport","origin":{"file":{"line":318,"name":"base.py"},"function":"handle_transport_fail"},"original":"Failed to submit message: 'oopsie'"},"process":{"name":"MainProcess","pid":905083,"thread":{"id":140274711856704,"name":"eapm event processor thread"}},"service":{"name":"app_name"}}
------------------------------ Captured log call -------------------------------
{"@timestamp":"2024-03-06T16:09:14.917Z","log.level":"error","message":"Failed to submit message: 'oopsie'","ecs":{"version":"1.6.0"},"log":{"logger":"elasticapm.transport","origin":{"file":{"line":318,"name":"base.py"},"function":"handle_transport_fail"},"original":"Failed to submit message: 'oopsie'"},"process":{"name":"MainProcess","pid":905083,"thread":{"id":140274711856704,"name":"eapm event processor thread"}},"service":{"name":"app_name"}}
=============================== warnings summary ===============================

To Reproduce

1. pytest

Environment (please complete the following information)

• OS: [e.g. Linux] Linux
• Python version: 3.10.12
• Framework and version [e.g. Django 2.1]:
• APM Server version:
• Agent version:

Additional context

None

[basepi]

This is definitely one of the most flakey tests in the suite. Luckily, it was mostly only flakey on windows in CI, so I haven't spent a ton of time digging into the internals. I've followed the code paths in the past trying to track this down and nothing seems amiss. I can't figure out how you can call client._transport.flush() and not have the transport correctly set its state.