Skip to content

kqueue: close(2) returns EBADF inside thread_perform; something closed the fd first, and the cancel path can't say who (it ignores flags.threadpool entirely) #232

Description

@mattrobenolt

Pure-libxev reproducer, committed:
integrations/ztls-xev/probe/two_conn_cancel.zig (imports only std,
builtin, xev). Two AF_UNIX socketpair ends on one xev.Loop with a thread
pool; end A arms a read, then cancels it and closes; end B has a read armed.
On macOS/kqueue:

.BADF => unreachable
  src/posix.zig:290 in close
  src/backend/kqueue.zig:1285 in perform   (xev_posix.close(op.fd))
  src/backend/kqueue.zig:960 in thread_perform

What is established (code-verified against 9ce8e8e6):

  • With the static xev.Kqueue.TCP the probe uses, reads are EVFILT_READ
    kevents performed on the loop thread; only close is threadpooled, and
    unconditionally so
    (src/watcher/stream.zig:434-436). The abort site is
    that threadpooled close.
  • The cancel path closes nothing: process_cancellations
    (kqueue.zig:275-296) dispatches on the target's state and routes .active
    targets to stop_completion (:902-950), which completes the target with
    ECANCELED via the .deleting submit branch (:194-210) and EV_DELETE.
    No close anywhere.
  • Neither the probe nor libxev's close paths perform a second close of A's fd
    in this scenario. EBADF at posix.zig:290 means the fd was not open when
    the worker ran: something closed it first, and we could not identify that
    closer.
    "Double close" / duplicate scheduling of the one close op is the
    leading hypothesis, stated as such.

The concrete suspect, provable by reading without a macOS run:
stop_completion is threadpool-blind. Completion.kevent()
(kqueue.zig:1053-1130) does not consult flags.threadpool, while the disarm
logic (kqueue.zig:424-431) explicitly treats threadpooled completions as
never-in-kqueue. So canceling an in-flight threadpooled op splits two ways:

  • Kevent-shaped threadpooled op (the dynamic xev.TCP, File, any
    options.threadpool user): stop_completion takes the kevent branch and
    writes c.result and pushes the completion onto the loop's intrusive queue
    (:204-207) while a worker still owns it and will later write
    c.result again (:960) and push the same node into the MPSC queue
    (:963). Completion.next is a single field shared by queue.zig (whose
    push asserts next == null) and queue_mpsc.zig. That is a data race
    with two visible outcomes: a completion delivered twice, and queue-link
    corruption, the only mechanism we found that can schedule the same close
    completion twice and produce the reported stack.
  • Non-kevent threadpooled op (e.g. .close itself): stop_completion
    falls to else => {} (:949). The target is never completed, active
    never decremented, the callback never fires. That is a stall shape, and it
    matches the ztls-side symptom this was minimized from (a close that never
    completes after a cancel).

Related already-open work: #169 (v.next is not proper cleared for re-enqueue a completion between different queues) and #224 (kqueue: fix tick(0) EV_DELETE flush, rearm state, and submit queue corruption; #227 is the IOCP
counterpart) are the same shared-Completion.next corruption family. They do
not cover the cancel path's threadpool-blindness: #224's rearm fix is about
rearm state, not stop_completion completing a threadpooled target through
the kevent branch. If those land, the corrupting push sites in the cancel path
remain.

Two asks:

  1. Make stop_completion respect flags.threadpool the way the disarm path
    does: complete threadpooled targets through the worker's own completion
    path (or refuse explicitly), never through the kevent branch.
  2. posix.zig:290-291 (close: .BADF => unreachable, else => unreachable)
    should surface the errno through the existing CloseError channel
    (kqueue.zig:1407-1414, which already maps e.g. .PERM). An EBADF here is
    diagnostic, and aborting destroys the evidence.

Explicitly unestablished: which close ran first; whether the completion was
double-scheduled or op.fd was stale; whether the EV_DELETE for the
canceled read interacts with the already-closed fd; and whether this abort
shares a root cause with the ztls-visible stall (noted as unestablished; the
stall's shape matches the non-kevent branch above, but ztls uses the static
API, whose reads are not threadpooled, so the double-schedule race does not
obviously apply to it).

For comparison, the same reproducer on Linux: io_uring completes the sequence
in 2 spins; epoll stalls, which is the dup-fd leak from report B, reproduced
here without ztls's workaround.


Disclosure: this report was investigated and drafted with AI assistance (Kimi
K3 ran the probes and drafted; Opus 5 ran adversarial fact-checks on every
claim). Human-reviewed before filing.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions