zeromq.RequestClient leaks socketpair FDs under sustained re-auth / refresh_pillar churn

[dwoz]

Summary

Minions under sustained saltutil.refresh_pillar / AsyncAuth re-auth
churn leak ~902 file descriptors (~451 socketpairs) per minion, tripping
the 1024-file ulimit "critical" threshold and eventually the minion's
own sock-throttle logic. FDs accumulate monotonically until the process
is restarted.

Root cause

salt/transport/zeromq.py:RequestClient.close (~line 2100 on 3008.x)
tears down the ZMQ REQ socket and destroys the context synchronously,
racing the still-running _send_recv coroutine. The task's coroutine
locals still hold a reference to the socket after close() runs; when
GC eventually runs during io_loop teardown, the socketpair backing the
REQ socket and its internal mailbox never get released. AsyncAuth
and refresh_pillar create + tear down a RequestClient per operation;
under churn the leak dominates the minion's FD footprint. AsyncReqMessageClient
already got a graceful-shutdown fix in the twangboy #68637 chain — the
newer pooled RequestClient path was missed.

Affected branches

3008.x only. 3006.x has the older AsyncReqMessageClient hardened by
#68637.

Reproduction

tests/monitoring/stress_test_no_api.sh for 10 min against a 3008.x
master with 50 minions running back-to-back saltutil.refresh_pillar
and re-auth cycles. Sample per-minion FD count of pid 9 in each minion
container every 30s.

Validation of fix

Pre-patch (50 minions, ~1h soak): median FD = 924, with the entire
minion cohort clamped at 924 (see
agents/reports/bug7-deploy/pre_restart.txt — 132/163 minions at 924,
tail up to 1024). Post-patch (same rig, 10 min stress after full
restart): median FD = 35, count of samples over 500 = 0/50 minions,
count over 900 = 0/50 (see
agents/reports/bug7-deploy/stress_10min_fd_samples_20260809_024317.tsv
— every sampled point across the 10 min window is 35/35/35 for
min/median/p90).

Fix mirrors AsyncReqMessageClient's graceful shutdown: capture the
per-task _send_recv_exit_future up front, resolve it in _send_recv's
finally, and defer socket/context teardown into an async task that
first awaits the exit future (bounded by 5s wait_for + shield).
Loop-not-running fallback closes synchronously.

Related PRs

TBD.