LDClient.close() blocks forever when DNS resolution hangs

[aviadr1]

Describe the bug

LDClient.close() can block forever. There is no timeout anywhere on its shutdown path, so if delivery of the final analytics event payload stalls, close() never returns and the calling thread is stuck permanently.

The trigger we hit in production is hanging DNS resolution. This is not covered by HTTPConfig(connect_timeout=..., read_timeout=...): urllib3 calls socket.getaddrinfo() in util/connection.py before it applies sock.settimeout(), so name resolution is outside both timeouts. When a resolver blackholes requests rather than returning NXDOMAIN, a flush worker sits in getaddrinfo() indefinitely and the whole shutdown path queues up behind it.

This bit us hard because close() is typically called from an atexit / interpreter-shutdown hook. A process that can never finish close() can never exit. We had worker pods sitting alive for hours to days after finishing their work, each holding a concurrency slot that was never released.

The unbounded path

Every wait in this chain is untimed:

1. LDClient.close() → self._event_processor.stop() — ldclient/client.py:356
2. DefaultEventProcessor.stop() → self._post_message_and_wait('stop') — ldclient/impl/events/event_processor.py:212
3. _post_message_and_wait() → reply.wait() with no timeout — event_processor.py:233-236. Note this method also does a blocking self._inbox.put(...) on a bounded queue, which is a second unbounded wait.
4. the reply is only set after _do_shutdown() returns — event_processor.py:150-151
5. _do_shutdown() → self._flush_workers.wait() — event_processor.py:163-165
6. FixedThreadPool.wait() → self._event.wait() with no timeout — ldclient/impl/fixed_thread_pool.py:41-47
7. which only returns once the in-flight EventPayloadSendTask finishes — and that is the task stuck in getaddrinfo().

To reproduce

This uses the real SDK and the real urllib3 stack. The only thing faked is the resolver — socket.getaddrinfo is made to hang for the events host, which is exactly what a blackholed resolver looks like to the process. Note connect_timeout and read_timeout are both set to 1 second and do not help.

import socket, sys, threading, time

HANG_SECONDS = 600
_real_getaddrinfo = socket.getaddrinfo

def hanging_getaddrinfo(host, port, *args, **kwargs):
    if "launchdarkly" in str(host):
        print(f"    [resolver] getaddrinfo({host!r}) -> hanging (simulated DNS blackhole)", flush=True)
        time.sleep(HANG_SECONDS)
    return _real_getaddrinfo(host, port, *args, **kwargs)

socket.getaddrinfo = hanging_getaddrinfo

from ldclient.config import Config, HTTPConfig
from ldclient.context import Context
from ldclient.impl.events.event_processor import DefaultEventProcessor
from ldclient.impl.events.types import EventInputIdentify
from ldclient.version import VERSION

config = Config(
    "fake-sdk-key",
    use_ldd=True,
    diagnostic_opt_out=True,
    http=HTTPConfig(connect_timeout=1, read_timeout=1),
)

print(f"ldclient version: {VERSION}", flush=True)

ep = DefaultEventProcessor(config)
ep.send_event(EventInputIdentify(int(time.time() * 1000), Context.create("user-key")))
ep.flush()
time.sleep(1.0)  # let the flush worker pick up the payload and get stuck

print("calling stop() -- this is what LDClient.close() does first", flush=True)
finished = threading.Event()
started = time.time()
threading.Thread(target=lambda: (ep.stop(), finished.set()), daemon=True).start()

if finished.wait(20.0):
    print(f"RESULT: stop() returned after {time.time() - started:.1f}s -- BOUNDED", flush=True)
    sys.exit(0)
print("RESULT: stop() has not returned after 20s -- UNBOUNDED (bug reproduced)", flush=True)
sys.exit(1)

Output on 9.16.1

ldclient version: 9.16.1
    [resolver] getaddrinfo('events.launchdarkly.com') -> hanging (simulated DNS blackhole)
calling stop() -- this is what LDClient.close() does first
RESULT: stop() has not returned after 20s and is waiting on a 600s resolver -- UNBOUNDED (bug reproduced)

The thread blocked inside close():

  File "repro.py", line 62, in <lambda>
    threading.Thread(target=lambda: (ep.stop(), finished.set()), daemon=True).start()
  File "ldclient/impl/events/event_processor.py", line 212, in stop
    self._post_message_and_wait('stop')
  File "ldclient/impl/events/event_processor.py", line 236, in _post_message_and_wait
    reply.wait()
  File "threading.py", line 669, in wait
    signaled = self._cond.wait(timeout)

and the flush worker it is waiting on:

  File "ldclient/impl/fixed_thread_pool.py", line 63, in _run_worker
    item()
  File "ldclient/impl/events/event_processor.py", line 54, in run
    resp = self._do_send(output_events)
  File "ldclient/impl/events/event_processor.py", line 261, in _post_events_with_retry
    r = http_client.request('POST', uri, headers=hdrs, body=data, timeout=urllib3.Timeout(...), retries=0)
  ...
  File "urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):

Expected behavior

close() should always return within a bounded time. Analytics events are already best-effort — they are dropped when the inbox is full, when the outbox overflows, and when all flush workers are busy — so dropping a final payload that cannot be delivered is consistent with existing behavior, and far better than never returning.

Logs

Nothing is logged; the SDK simply never returns from close().

SDK version

9.16.1 (also present in 9.15.x and 9.16.0; the relevant code is unchanged on main as of 5da1515)

Language version, developer tools

Python 3.12 and 3.14, urllib3 2.7.0

OS/platform

Linux (reproduced on 6.18 x86_64). Originally hit on Kubernetes, where a degraded cluster resolver made this routine.

Additional context

DefaultAsyncEventProcessor.stop() has the same unbounded shape — await self._post_message_and_wait('stop') with no timeout (ldclient/impl/events/async_event_processor.py:245), while flush_and_wait() directly above it already bounds its wait with asyncio.wait_for. I have not reproduced a hang there, so I left it alone, but it looks worth a look.

I have opened a PR fixing the sync path: it adds a shutdown_timeout config option (defaulting to 5 seconds) and threads it through the shutdown waits.