VirtualThreadPool can create unlimited threads while doc mentions otherwise

[luvk1412]

Jetty version(s)

12.0.31

We are evaluating shifting our spring boot applications to virtual threads. In the process i was going through
VirtualThreadPool doc of which says:

It is possible to specify the max number of concurrent virtual threads that can be spawned, to help limiting resource usage in applications

But after reading the code, it seems like unlimited virtual thread's can be spawned as semaphore is acquired post virtual thread creation. So this essentially mean that as many virtual threads can be created but concurrently only VirtualThreadPool#maxThreads task can run. This behaviour also can also lead to resource exhaustion, while the doc says otherwise. I was expecting a RejectedExecutionException to be thrown if threads more than maxThreads are spawned, hence this looks like a bug to me. Please correct me if i am misunderstanding anything here.

[joakime]

So this is a theoretical issue, one borne from interpreting the code.
Do you have a testcase that can replicate this theory?

[sbordet]

If you have maxThreads=1 and you submit 10 tasks, in a platform thread pool you will have 1 running thread and 9 tasks queued waiting to be run.

For the same case with a virtual thread pool you will have 1 running thread, and 9 virtual threads queued waiting to be run.
As virtual threads are cheap, when they block on the semaphore, they likely occupy a similar amount of memory than a Runnable queued in a queue of the platform thread case.

Like the comment in the code says, we cannot block the caller thread.

We cannot throw RejectedExecutionException either, because the task may be a task that would eventually free up other thread(s), or a task that must be executed.

I guess we can tweak the wording of the documentation, but I think the code is correct as is.

[luvk1412]

hey @joakime @sbordet , thanks for your quick replies.

Do you have a testcase that can replicate this theory?

Sure, here are some quick tests i wrote using claude to demonstrate my two points:

1. unlimited virtual thread's can be spawned as semaphore is acquired post virtual thread creation
2. This behaviour can also lead to resource exhaustion

Show Tests

Show Tests Code

import org.eclipse.jetty.util.thread.VirtualThreadPool;
import org.junit.jupiter.api.Test;

import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;

import static org.junit.jupiter.api.Assertions.assertTrue;

public class VirtualThreadPoolBehaviourSmokeTests {


  @Test
  void testUnboundedThreadsCreation() throws Exception {
    int maxThreads = 200;
    int submittedTasks = 10_000;

    System.out.println("=== Unbounded Thread Creation TEST ===");
    System.out.println("Max heap: " + Runtime.getRuntime().maxMemory() / 1024 / 1024 + " MB");
    System.out.println("Max concurrent threads: " + maxThreads);
    System.out.println("Tasks to submit: " + submittedTasks);

    System.gc();
    Thread.sleep(500);
    long baselineMemory = getUsedMemory();

    VirtualThreadPool pool = new VirtualThreadPool(maxThreads);
    pool.setTracking(true); // Enable thread tracking to count created threads
    pool.start();

    CountDownLatch blockTasks = new CountDownLatch(1);
    AtomicInteger executingCount = new AtomicInteger();

    System.out.println("\n=== SUBMITTING TASKS ===");
    // Submit tasks that will block
    for (int i = 0; i < submittedTasks; i++) {
      pool.execute(() -> {
        executingCount.incrementAndGet();
        try {
          blockTasks.await(); // Block until test completes
        } catch (InterruptedException e) {
          Thread.currentThread().interrupt();
        }
      });

      // Monitor progress
      if (i % 2000 == 0 && i > 0) {
        System.out.println("Submitted: " + i +
          ", Pool tracked threads: " + pool.getThreads() +
          ", Executing: " + executingCount.get() +
          ", Memory: " + getUsedMemory() / 1024 / 1024 + " MB");
      }
    }

    Thread.sleep(2000); // Let all threads spawn

    System.gc();
    Thread.sleep(500);
    long afterMemory = getUsedMemory();

    System.out.println("\n=== RESULTS ===");
    System.out.println("Pool tracked threads: " + pool.getThreads());
    System.out.println("Threads executing concurrently: " + executingCount.get());
    System.out.println("Baseline memory: " + baselineMemory / 1024 / 1024 + " MB");
    System.out.println("After submission: " + afterMemory / 1024 / 1024 + " MB");
    System.out.println("Memory increase: " + (afterMemory - baselineMemory) / 1024 / 1024 + " MB");

    int trackedThreads = pool.getThreads();
    if (trackedThreads > 0) {
      long memoryPerThread = (afterMemory - baselineMemory) / trackedThreads;
      System.out.println("Approximate memory per virtual thread: ~" + memoryPerThread + " bytes");
    }

    System.out.println("\n=== ANALYSIS ===");
    assertTrue(pool.getThreads() > 9000, "Expected ~10,000 threads created, got: " + pool.getThreads());
    assertTrue(executingCount.get() <= maxThreads, "Expected max " + maxThreads + " executing, got: " + executingCount.get());
    System.out.println("✓ Confirmed: ALL " + pool.getThreads() + " virtual threads were created");
    System.out.println("✓ Confirmed: Only " + executingCount.get() + " executing concurrently (semaphore limit)");

    blockTasks.countDown();
    pool.stop();
  }

  @Test
  void testOOMWithUnboundedCreation() throws Exception {
    System.out.println("=== OOM STRESS TEST ===");
    System.out.println("Max heap: " + Runtime.getRuntime().maxMemory() / 1024 / 1024 + " MB");

    VirtualThreadPool pool = new VirtualThreadPool(200);
    pool.setTracking(true);
    pool.start();

    CountDownLatch blockTasks = new CountDownLatch(1);
    AtomicInteger submitted = new AtomicInteger();
    AtomicInteger executed = new AtomicInteger();

    try {
      for (int i = 0; i < 1_000_000; i++) {
        pool.execute(() -> {
          executed.incrementAndGet();
          try {
            blockTasks.await();
          } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
          }
        });
        submitted.incrementAndGet();

        if (i % 10000 == 0) {
          System.out.println("Submitted: " + submitted.get() +
            ", Pool tracked: " + pool.getThreads() +
            ", Executing: " + executed.get() +
            ", Memory: " + getUsedMemory() / 1024 / 1024 + " MB");
        }
      }

      System.out.println("\n✓ Successfully submitted " + submitted.get() +
        " tasks without OOM");
      System.out.println("Pool tracked threads: " + pool.getThreads());

    } catch (OutOfMemoryError e) {
      System.err.println("\n✗ OOM after submitting " + submitted.get() + " tasks");
      System.err.println("Pool tracked threads: " + pool.getThreads());
      System.err.println("Executing: " + executed.get());
      System.err.println("\n⚠ This proves VirtualThreadPool's unbounded creation can cause OOM!");
      throw e;

    } finally {
      blockTasks.countDown();
      pool.stop();
    }
  }

  private long getUsedMemory() {
    Runtime runtime = Runtime.getRuntime();
    return runtime.totalMemory() - runtime.freeMemory();
  }
}

Show Tests output

mvn test -Dtest=VirtualThreadPoolBehaviourSmokeTests#testOOMWithUnboundedCreation -DargLine="-Xmx128m -Xms128m"`

=== Unbounded Thread Creation TEST ===
Max heap: 128 MB
Max concurrent threads: 200
Tasks to submit: 10000

=== SUBMITTING TASKS ===
Submitted: 2000, Pool tracked threads: 1855, Executing: 200, Memory: 18 MB
Submitted: 4000, Pool tracked threads: 3792, Executing: 200, Memory: 22 MB
Submitted: 6000, Pool tracked threads: 5350, Executing: 200, Memory: 26 MB
Submitted: 8000, Pool tracked threads: 7968, Executing: 200, Memory: 30 MB

=== RESULTS ===
Pool tracked threads: 10000
Threads executing concurrently: 200
Baseline memory: 7 MB
After submission: 30 MB
Memory increase: 23 MB
Approximate memory per virtual thread: ~2415 bytes

=== ANALYSIS ===
✓ Confirmed: ALL 10000 virtual threads were created
✓ Confirmed: Only 200 executing concurrently (semaphore limit)

mvn test -Dtest=VirtualThreadPoolBehaviourSmokeTests#testOOMWithUnboundedCreation -DargLine="-Xmx128m -Xms128m"

=== OOM STRESS TEST ===
Max heap: 128 MB
Submitted: 1, Pool tracked: 0, Executing: 0, Memory: 40 MB
Submitted: 10001, Pool tracked: 7188, Executing: 200, Memory: 30 MB
Submitted: 20001, Pool tracked: 17304, Executing: 200, Memory: 52 MB
Submitted: 30001, Pool tracked: 28690, Executing: 200, Memory: 70 MB
Submitted: 40001, Pool tracked: 38217, Executing: 200, Memory: 86 MB
Submitted: 50001, Pool tracked: 49948, Executing: 200, Memory: 106 MB
Submitted: 60001, Pool tracked: 59586, Executing: 200, Memory: 116 MB
Submitted: 70001, Pool tracked: 59610, Executing: 200, Memory: 121 MB
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create name string at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 838

---

If you have maxThreads=1 and you submit 10 tasks, in a platform thread pool you will have 1 running thread and 9 tasks queued waiting to be run.
For the same case with a virtual thread pool you will have 1 running thread, and 9 virtual threads queued waiting to be run.
As virtual threads are cheap, when they block on the semaphore, they likely occupy a similar amount of memory than a Runnable queued in a queue of the platform thread case.

I totally agree with the above, my only concern is that i cannot control how much can be queued. Currently while creating QueuedThreadPool for our server we provide a LinkedBlockingQueue with a certain capacity. So in your example when maxThreads=1, and capacity=15, we will have 1 running thread and 9 tasks queued but if the queuing continues, 17th task would get rejected. As I understand there is no way to achieve this currently with VirtualThreadPool.

In practical sense for us, when one of our application server is under stress and due to it, if we have already reached a maximum concurrency, as well as maximum queuing capacity, it doesn't make sense for our use case to queue more incoming client requests as it's not going to help the server in anyway and in such situation we would want the server to auto recover and hence we would want all such requests to fast fail and clients understand this fast fail, hence they also give the server some breathing room to recover instead of bombarding further. With VTP this would just lead to either server getting OOM killed eventually or clients waiting for long time which eventually might result in 5xx or 2xx, so it would have been better that we give client a response early, than rather making him wait and still providing an error in a 5xx case.

We cannot throw RejectedExecutionException either, because the task may be a task that would eventually free up other thread(s), or a task that must be executed.

I am curious about this part because I was under the impression this can be done. Are you here talking about some of the jetty's internal tasks which will be submitted to the VTP ? Is there some doc where I can read about it that which are these tasks and what would be repercussion of rejecting those because as for tasks corresponding to http requests, I as consumer would be fine with rejecting them. If rejection isn't acceptable, then is currently what we are doing by providing LinkedBlockingQueue with a certain capacity to QueuedThreadPool as mentioned above also wrong. We have been running with this configuration in production for some years now and it has worked fine and we have seen fast fail and auto recovery working as well. I was under the impression that Rejection is fine because there is no additional contract defined on Executor#execute() by ThreadPool impl jetty expects.

---

Also while working and experimenting with VTP, I found out one more possible problem which is unrelated to above issue I raised but just pointing it out here. Jetty server startup by default submits 6 long running tasks(on my machine at least) to the ThreadPool per connector, 5 for selector and 1 for acceptor. So if someone starts a server with 2 connectors(which would consume 12 semaphore permits) and maxThreads<=12, then jetty server essentially will not be able to server any requests. I am not sure if this should be documented or server startup should fail in such cases or something else, just pointing it here as I observed it.

[sbordet]

Currently while creating QueuedThreadPool for our server we provide a LinkedBlockingQueue with a certain capacity. So in your example when maxThreads=1, and capacity=15, we will have 1 running thread and 9 tasks queued but if the queuing continues, 17th task would get rejected.

This is not a recommended configuration:
https://jetty.org/docs/jetty/12.1/programming-guide/arch/threads.html#thread-pool-queue

In practical sense for us, when one of our application server is under stress and due to it, if we have already reached a maximum concurrency, as well as maximum queuing capacity, it doesn't make sense for our use case to queue more incoming client requests as it's not going to help the server in anyway and in such situation we would want the server to auto recover and hence we would want all such requests to fast fail and clients understand this fast fail, hence they also give the server some breathing room to recover instead of bombarding further.

The model you explain above is too simplistic and assumes that every task submitted to the thread pool is an HTTP request, so limiting queues or threads provides a limiter to HTTP request.
That's not the case.

If you want to limit requests and fail fast in case of too much concurrency, use QoSHandler:
https://jetty.org/docs/jetty/12.1/programming-guide/server/http.html#handler-use-qos

Jetty server startup by default submits 6 long running tasks(on my machine at least) to the ThreadPool per connector, 5 for selector and 1 for acceptor. So if someone starts a server with 2 connectors(which would consume 12 semaphore permits) and maxThreads<=12, then jetty server essentially will not be able to server any requests.

It is documented here:
https://jetty.org/docs/jetty/12.1/programming-guide/arch/threads.html#thread-pool-configuration

[luvk1412]

This is not a recommended configuration

So all this time in my org it is being used wrong ig and even so from before the time i even joined.
Thanks for all the above resources, i will go through them and hopefully correct the behaviour across all services.

Is there a specific case or side effect documented or some past gh issue you can point me to, which was caused by using the incorrect rejection approach we are currently using. It would be helpful to for me as i can try reproducing it and demonstrate internally. If not then also it's fine, thanks a lot for your help.

[sbordet]

I'm not sure what you exactly mean by "rejection approach we are currently using".
If you mean bounding the queue of the thread pool, then yes it is likely wrong.

In the Jetty code there are hundreds of places where we submit internal tasks to the thread pool.
If there is a chance that the task is rejected, then it is a problem, because we would write extremely defensive code throughout the whole codebase: for every call to execute(Runnable) catch RejectedExecutionException.
Likely this cannot be factored out in a utility method, because the behavior of what to do would depend on the caller, which might me layers away -- the source code would just become a mess.
But that is also just the tip of the iceberg: what do you do if a task is rejected? There are tasks that must eventually be executed to keep the server running, and running them in the caller thread is often not an option, as they might block a thread that cannot be blocked (or the server would hang).

As I said, having a too simple mental model of how a server work (1 task == 1 HTTP request) maybe worked in the past, but not recently and not for Jetty.
Trying to bound the thread pool queue to bound the number of concurrent request is the wrong approach.
It probably was never the correct one, but alas it became widespread.

I'll give you a simple example.
Let's assume the case of an HTTP request with content: the headers have been received, but not the content yet.
A thread is dispatched to handle that request, and the application code blocks the thread trying to read the request content that it is not available yet.
Meanwhile, the server is busy and the thread pool queue fills up.
Then, the request content arrives. The server would queue a task that would unblock the processing thread, but it can't: RejectedExecutionException.
Imagine this happening for all the threads, and now you have all the server threads blocked, no more threads to clear the queue (they are all blocked in application code), and no way to unblock the threads that are blocked, because the tasks that would unblock them have been rejected.
If you run the task in the caller thread, it may happen that you unblock the application thread, that reads some content, but it is still not all of it, so it blocks again. But now, you have blocked the caller thread, which is likely the selector thread, and now not only you have 1 request blocked, but the whole selector managing many sockets is not able to process them at all, and again you have locked up the server.

There are many other (more complicated) examples, especially with HTTP/2.

Now, what happens with an unbounded queue? That the tasks would be queued up, but hopefully a thread, later, will be available to process them, freeing other threads.
This is why having a moderately short idle timeout is important: it would allow the server to unblock threads waiting to read request content that never arrives.

What if you queue up until OOME? Well, you have an under-provisioned server for your load.

The right solution is to have a moderately over-provisioned thread pool with an unbounded queue (say maxThreads=200), and have a more specific mechanism to limit the request concurrency: the QoSHandler (configured to allow at most 180 requests or such).
Requests beyond 180 concurrent would be immediately rejected (even before arriving to the application), and your system will be stable because you actually limit requests, not the number of tasks in a thread pool (which has a much more vague relation with the number of concurrent requests).

HTH.

[luvk1412]

@sbordet thanks a lot for the detailed explanation, this was really helpful. Apologies if I picked your brain a bit too much 😅

[sbordet]

@luvk1412 I updated the documentation in #14359, mind to review it?