Virtual Threads

Virtual Threads are lightweight threads supporting high-throughput concurrency. Most changes related to virtual threads in JDK 21 are invisible at an API level because virtual threads are instances of the familiar Thread class. To illustrate the changes, we will look at several examples of concurrent programs.

Sleep Sort

Virtual threads are especially beneficial in the presence of blocking operations like sleep. To compare different ways to execute tasks concurrently, we use them through a common interface ExecutorService which has implementations for different execution models such as pooling threads or creating new threads for each submitted task.

Sleep Sort is a tongue-in-cheek sorting algorithm that uses concurrent tasks to print each sorted item after waiting some time proportional to the item's size. The idea is to start all concurrent tasks simultaneously so they print their corresponding items in the correct order.

One Thread per Task

Here is a Java implementation of the Sleep Sort algorithm.

public record SleepSort(ExecutorService executor) {
    public static void main(String[] args) {
        new SleepSort().sortInputs();
    }
    private SleepSort() {
        this(Executors.newThreadPerTaskExecutor(Thread::new));
    }
    private void sortInputs() {
        try (BufferedReader stdin = new BufferedReader(new InputStreamReader(System.in))) {
            stdin.lines().map(Integer::parseInt).forEach(num -> {
                executor.submit(() -> {
                    TimeUnit.SECONDS.sleep(num);
                    System.out.println(num);
                    return null;
                });
            });

            executor.shutdown();
            executor.awaitTermination(1, TimeUnit.MINUTES);
        } catch (IOException | InterruptedException e) {
            executor.shutdownNow();
        }
    }
}

We use Executors.newThreadPerTaskExecutor to create a new platform thread for every input number. In Java, each platform thread is mapped to an operating system thread. So, we create as many operating system threads as there are input numbers. For a small number of inputs, this approach works fine. The following shell command generates ten random numbers between 1 and 10 and then sorts them using our program.

me@java-dev:~/java21-demo$ shuf -r -i 1-10 -n 10 | java -cp target/classes sebfisch.SleepSort
1
2
3
3
5
5
5
8
9
10

When sorting a larger number of inputs, however, the time for creating as many operating system threads (for example 10000) seems to interfere with their sleeping time. As a result, numbers are not printed in sorted order.

me@java-dev:~/java21-demo$ time shuf -r -i 1-10 -n 10000 | java -cp target/classes sebfisch.SleepSort
[skipped some output]
9
6
9
8
9
8
7
8
8
8
8
7
6
9
[skipped some output]

real	0m13.356s
user	0m4.547s
sys	0m3.750s

We also print timing information for this run so we can compare it with that of a later implementation.

Thread Pooling

It is common practice to limit the number of created operating system threads by creating a pool of threads where each thread is used to execute multiple tasks. We can change a single line in the constructor of SleepSort to use this execution model with a fixed number of threads.

     private SleepSort() {
-        this(Executors.newThreadPerTaskExecutor(Thread::new));
+        this(Executors.newFixedThreadPool(10));
     }

This approach works fine for Sleep Sort as long as the number of sorted inputs does not exceed the number of threads. We can still successfully sort ten random numbers using ten threads as follows.

me@java-dev:~/java21-demo$ shuf -r -i 1-10 -n 10 | java -cp target/classes sebfisch.SleepSort
1
3
3
4
4
4
5
6
6
8

However, a thread pool is typically created to use a small number of threads to execute a much larger number of tasks. This approach is problematic when the executed tasks contain blocking operations, like in Sleep Sort. Even twenty random numbers are sorted incorrectly when using a thread pool of size ten.

me@java-dev:~/java21-demo$ shuf -r -i 1-10 -n 20 | java -cp target/classes sebfisch.SleepSort
1
2
2
3
5
6
7
9
7
9
9
5
4
2
2
8
10
10
4
9

The first few numbers in the output are sorted correctly. But towards the end of the output, smaller numbers appear between larger ones. While the corresponding tasks have waited the appropriate time, they have been started much too late because the number of threads is insufficient to start all tasks at the same time. Half of the tasks are started later because all threads are blocked when executing the first ten tasks.

In general, using thread pools with tasks that use blocking operations can significantly reduce the throughput of an application. Blocked threads are not available to execute other tasks that would not be blocked.

One Virtual Thread per Task

We can change our program again to create one virtual thread per sorted input.

     private SleepSort() {
-        this(Executors.newFixedThreadPool(10));
+        this(Executors.newVirtualThreadPerTaskExecutor());
     }

This implementation is very similar to the first one. The only difference is that each task is executed using a new virtual thread instead of a new platform thread. Let's try using the program to sort 10K random numbers again.

me@java-dev:~/java21-demo$ time shuf -r -i 1-10 -n 10000 | java -cp target/classes sebfisch.SleepSort
[skipped output]

real	0m10.139s
user	0m3.097s
sys	0m1.458s

All numbers are printed in the correct order now, so we do not show the output here. The footprint of 10K virtual threads is small enough to start all of them roughly at the same time. The "real" time is now around the expected ten seconds required for waiting for the largest number. Even 100K numbers can be sorted with this program correctly and the "real" time is still not much more than ten seconds.

me@java-dev:~/java21-demo$ time shuf -r -i 1-10 -n 100000 | java -cp target/classes sebfisch.SleepSort
[skipped output]

real	0m10.670s
user	0m15.926s
sys	0m12.898s

We can use Java's built-in Flight Recorder to collect more detailed information about the execution of our Sleep-Sort program.

me@java-dev:~/java21-demo$ shuf -r -i 1-10 -n 10000 | java -cp target/classes -XX:StartFlightRecording:filename=SleepSort.jfr sebfisch.SleepSort
[skipped output]

We can then use JDK Mission Control (JMC) to open the generated file SleepSort.jfr and get a detailed overview of the program's execution. JMC has a Threads section where virtual threads are grouped into a specific thread group.

An important difference between platform threads and virtual threads is how they are mapped to operating system threads. While each new platform thread is executed on a new operating system thread, a large number of virtual threads are executed on a usually much smaller pool of operating system threads, so-called carrier threads.

The other important difference is that blocked virtual threads do not block the operating system thread they are executed on. Instead, virtual threads are mounted on their carrier thread to execute them, and unmounted when they block, to free the carrier thread for the execution of another virtual thread. When a virtual thread is unblocked, it is eventually remounted on a (potentially different) carrier thread to resume execution.

Essentially, blocking operations on virtual threads are implemented with corresponding non-blocking operations, and the Java VM implements asynchronous execution of virtual threads in a way that looks like synchronous execution with blocking operations to application programmers. As a consequence, programmers can use this familiar programming model with a much larger number of threads than was previously possible in Java.