Thread starvation in ThreadLimitHandler

[arcticmosquito]

Jetty version(s)
12.0.x

Jetty Environment
core (ThreadLimitHandler) + ee8 (ServletContextHandler)

Java version/vendor (use: java -version)

openjdk 21.0.2 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-21.0.2.13.1 (build 21.0.2+13-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.2.13.1 (build 21.0.2+13-LTS, mixed mode, sharing)

OS type/version
macOS Sonoma 14.4.1

Description
ThreadLimitHandler might block a request execution forever if write operation is completed from another thread (e.g., when TCP tx buffer is saturated). Suppose that ThreadLimitHandler is set to 1 thread per remote, there is a single inbound request, the thread handling the request obtains a permit and tries to write a response ServletOutputStream if write operation could not be finished by the handling thread it will block waiting for a callback invocation (could not find ee8 version, but I assume ee10 is subjected to the same problem - HttpOutput). When the write operation is finished the notification thread will try obtaining a permit (ThreadLimitHandler) however this attempt will never succeed as the only permit is held by the processing thread.
This issue could reproduce with any number of permits as long as there are as many in progress requests.