Will web server libraries use virtual threads with 9.3.8.0?

[mattwelke]

I was reading https://github.com/jruby/jruby/releases/tag/9.3.8.0 and noticed:

Fibers can now be configured to use OpenJDK Project Loom's virtual threads, which are lightweight and do not consume a native platform thread. Virtual threads make it possible for JRuby applications to create thousands of concurrently-executing fibers. Enable this feature on Java 19 using the JVM flag --enable-preview. (#7328)

My understanding is that this means under the hood, fibers will use virtual threads. Based on my understanding of Java so far, playing around with the preview releases, I know that to use virtual threads in Java instead of platform threads, you must explicitly make your new thread a virtual thread as you create it (ex. with Thread.ofVirtual()). So this makes sense to me as how the JRuby maintainers may have implemented it. This new version of JRuby, when configured to do so (by starting the JVM with --enable-preview) will create threads this way for fibers.

But what does this mean for web servers libraries like Puma and unicorn? I remember when using Rails a while ago that Puma, for example, was popular at the time because it used multiple threads to handle more requests concurrently. Will the devs of libraries like it need to switch their code to use fibers to take advantage of virtual threads when they're run using JRuby? Have they already been aware of this for a while now, so that their code uses fibers (which I would assume would be powered by JVM platform threads in JRuby right now) so that those libraries are ready to go right now?

[headius]

For frameworks that use Fibers (and Enumerators with next, which requires a Fiber) users will not have to do anything but enable virtual threads in the JVM to get those advantages. That means frameworks like async, and the defunct celluloid should "just work" and be able to scale out far more fibers than before.

As for what tools like web servers should do going foward... that depends how they want to achieve concurrency and parallelism.

Native threads (Thread.new threads in Ruby) will always get a "platform" thread and run in parallel so long as there's enough cores. Virtual threads (Fiber.new in Ruby) may run in parallel with native threads (Ruby ties each Fiber to one parent platform thread) but generally will not run in parallel with fibers on the same platform thread.

Fiber-heavy servers like Async tend to use only one core (or very few cores) but use that core to drive many fibers, achieving very high concurrency but not very high parallelism. Thread-based servers like Puma use a native thread per request handler to always achieve parallelism but this potentially limits how many concurrent connections can be processed.

A gross simplification might be:

• Async = many thousands concurrent, very few parallel
• Puma = many dozens parallel, no more than that concurrently.

I believe both approaches will be valid going forward, as well as hybrid approaches that use both platform and virtual threading.

[mattwelke]

Thanks for the info @headius.

If you don't mind, there's some more stuff I'm curious about. Concurrency is something I've been interested in lately.

Why was it implemented in JRuby so that in order to use a Java virtual thread, you need to use a Ruby fiber? My understanding of virtual threads in Java is that one of the main design goals was to continue to be able to use the straightforward, preemptive multitasking style (as opposed to the cooperative multitasking style which is often considered more difficult to use). With virtual threads, you just run them and rely on the JVM to preempt them (efficiently) when it is the best time for them to be preempted.

In the Ruby world, it seems to me like the Thread class aligns well with that programming model. Why wouldn't JRuby be designed so that Ruby threads would be backed by Java virtual threads instead of Java platform threads? Wouldn't doing it that way instead make all the existing web server libraries using Ruby threads (like Puma) use Java virtual threads automatically when they're run with JRuby?

Is the idea that we don't think that would even be desirable in the first place? For example, because, as you mention, they use a thread-per-request model, they might pool Ruby threads. And pooling Java virtual threads is an anti-pattern.

Or, is the idea that we still want to allow JRuby users to use Java platform threads, so keeping Ruby's Thread class implemented by Java platform threads achieves that goal?

[headius]

in order to use a Java virtual thread, you need to use a Ruby fiber

Well, technically you can just call java.lang.Thread.ofVirtual().new from Ruby and end up with a virtual thread just like what's inside a fiber, with the bonus that it can migrate across native threads and get both concurrency and parallelism. But I don't think that's what you're asking.

The reason we back regular Ruby threads with a full native "platform thread" is simple: that's how it is in CRuby. You spin up a Thread, you get a native pthread backing it and guarantees of parallelism across cores. If you want fine-grained "threadlet" concurrency within that native thread, you use Fiber instances which are multiplexed on that thread, and that's where JDK virtual threads come in.

At some point, a native thread somewhere needs to drive the virtual threads you create. We are not explicitly binding the virtual threads to a single native thread (e.g. using a virtual thread Executor backed by native threads) but simply launch virtual threads and use our existing synchronization mechanisms to ensure only one Fiber from a given Thread runs at a time. As you know, this is an artificial limitation, but it matches CRuby semantics.

Now the more interesting experiment I'd like to try would be to have each Ruby Thread actually encompass a virtual thread Executor backed by exactly one native thread. Instead of us doing a Java new Thread(...), we would create a one-thread executor, start up the first virtual thread (which would be the "main" fiber for the associated native thread), and then any future fibers just get scheduled alongside the main fiber.

Most of this is academic, however; we still need to honor the semantics of Ruby Fiber, which are:

• A fiber lives within exactly one native thread
• A thread may have many live fibers that multiplex access to the native thread
• Fibers cannot migrate across threads

So moving to an executor-per-Thread with a pool of vthreads actually driving the Ruby code is just an implementation detail, and would not really change anything apparent for Ruby devs.