Using JDK21 virtual threads as compute pool

[jacum]

Dear maintainers,

May I bother you with a question?

I did some research recently
https://jacum.medium.com/running-cats-effect-on-virtual-threads-of-jdk21-7a68ea97fd65
TLDR:
The current way of dealing with occasional blocking contexts in Cats Effect is sub-optimal, when compared to native fiber scheduling implemented on JDK21.

Of course - JDK21 is still fresh, and may not (yet) available for all and every JVM workload, but it is clearly very efficient. It is also next LTS after JDK 17, and likely will be most widely used one in a year or two.

I wonder what is your take on the future of the runtime on JDK21?

Is the whole runtime going to be replaced to thread-per-runnable?

Or just maybe rewrite blockOn() method

cats-effect/core/jvm/src/main/scala/cats/effect/unsafe/WorkerThread.scala

Line 817 in 623178c

override def blockOn[T](thunk: => T)(implicit permission: CanAwait): T = {

to run that 'thunk' on Executors.newVirtualThreadPerTaskExecutor() and leave the rest of non-blocking executor to use the same Thread API?

[armanbilge]

The current way of dealing with occasional blocking contexts in Cats Effect is sub-optimal, when compared to native fiber scheduling implemented on JDK21.

Surprisingly, actually this is not true at all :) It is very subtle and nuanced, but in most cases the handling of blocking tasks in Cats Effect is near optimal and probably better than Loom. We recently discussed this in #3869 (comment). The tl;dr is that Loom doesn't and cannot fix I/O blocking such as file I/O or DNS and using virtual threads in for those operations instead of the Cats Effect threadpool would create unnecessary thread shifting and contention. Loom does fix blocking-style APIs that use non-blocking I/O (e.g. network sockets) under-the-hood, such as JDBC.

Based on the discussion in that issue, we did add a mechanism so that blocking tasks can be run on virtual threads.

• Block in-place if running on a virtual thread #3870

I wonder what is your take on the future of the runtime on JDK21?

Daniel wrote a very nice Reddit post detailing why Loom doesn't matter so much for Cats Effect. tl;dr we will not be replacing the compute pool with virtual threads.

https://www.reddit.com/r/scala/comments/sa927v/comment/htsoydn/

Further reading/watching about threads, threadpools, and performance:

• https://typelevel.org/cats-effect/docs/core/starvation-and-tuning
• https://www.youtube.com/watch?v=PLApcas04V0

Let us know if you have more questions :)

[armanbilge]

and yet, due to .unsafeRunSync(), it will receive all the ritual of creating a Future and awaiting for its result on a lock, albeit briefly.

Actually, this is also not correct. That is SyncIO, so unsafeRunSync() runs in-place without involving any Futures or locks. For reference the implementation is here.

cats-effect/core/shared/src/main/scala/cats/effect/SyncIO.scala

Line 223 in 623178c

def unsafeRunSync(): A = {

However, if there are rogue IO#unsafeRunSync() calls in http4s please open an issue. I would consider that a serious bug.

---

even if this happens to be a fully local effect in the same JVM (e.g. evaluation with MonadThrow)

For these cases, have you looked at the IO#syncStep API? It attempts to transform an IO to a SyncIO which you can run in-place, without thread-shifting.

---

With sufficiently large bursts of such .unsafeRunSync()-s, this causes hundreds of such threads spawned, and this can choke even workloads given luxury of 16 cpu cores.

I fully understand that it's far better not to use .unsafeRunSync() at all, but practically it is impossible to achieve.

I know its not very helpful, but unfortunately this sounds like a poorly architected codebase :( even if you cannot refactor entirely to IO, using unsafeToFuture would be much much better.

---

but in our benchmarks that run the complete services, just replacing IORuntime with the (rather naive) JDK21 virtual threads executor did yield a substantial performance gain, attributed to much fewer threads.

Nice, I'm glad this is an fruitful option for you, in which case the PR I linked above adding in-place blocking support for virtual threads would be very applicable.

So I think I should clarify my comments: in general, it does not make sense to replace the Cats Effect runtime with Loom / Virtual threads. However in very specific cases such as yours, it might be advantageous, and we are definitely open to changes like #3870 that will enhance support for and interop with virtual threads.

[jacum]

Thank you so much Arman, very insightful!

[jacum]

Is #3870 going to be released with 3.5.3?

[armanbilge]

At the moment, I'm afraid not. Personally I think I'd be okay to backport without an explicit test for it, what does @djspiewak think?

This PR is targeted at series/3.x because we are pinned to Scala 3.2 in series/3.5.x and that doesn't support JDK 21. We could backport the implementation changes here to 3.5.x but we cannot test them.

[djspiewak]

I'm leery about doing this without an explicit test. It has some relatively nontrivial machinery in order to avoid breaking Java 8 support, so it would be pretty easy to accidentally break. With that said though, I'm releasing a quick snapshot of 3.6 that you can try if you would like, @jacum. It should be pretty drop-in and things will evict forward, though the usual caveats apply w.r.t. stability and production use.

Update: Release has been published as version 3.6-623178c.

I would definitely echo @armanbilge's sentiments about unsafeRunSync(), with some additional contextual color. First off, I assume you're aware that Dispatcher#unsafeRunSync(...) is significantly faster than IO#unsafeRunSync()? This is because the latter has some complicated (read: expensive) machinery needed to safely transport fatal errors (including those which leave the JVM in an undefined state) back to the call site while the former is exactly what you would expect it is.

Secondarily, I really don't agree that unsafeRunSync is practically unavoidable. I've worked on a large number of non-trivial Cats Effect applications at scale and the only place where unsafeRunSync was necessary was when dealing with particularly recalcitrant interop cases (i.e. where non-CE-based third-party frameworks need to call CE-using code and do not support asynchronous call patterns like unsafeRunAsync). These cases do happen but 1) they are fairly uncommon in my experience due to the relative completeness of the ecosystem, and 2) they are always avoidable in the critical path (meaning their performance impact is not high).

If you can share, I'd be curious as to what the primary drivers of these calls are.

[djspiewak]

A couple quick comments on the (excellent!) blog post:

There were no substantial difference measured on non-blocking effects, but for the unsafeRunSync(), JDK 21 beats default executor with flying colors.

Winning on unsafeRunSync() doesn't surprise me at all, but the "no substantial difference measured on non-blocking effects" really surprises me quite a bit. I haven't had a chance to look too closely at your test code, but my guess is that you're not actually stressing the scheduling algorithm. You can take a look at some of the scheduler JMH benchmarks in the repo for inspiration. They're highly synthetic and result in magnified scores relative to what you would actually see in production, but they do a better job of comparing ForkJoinPool (which underlies virtual threads) and WorkStealingThreadPool. In our tests, the latter pretty consistently destroys the former, particularly in more compute-bound scenarios.

The only caveat is that by default Cats Effect tracing attaches something to each thread, and with JDK 21 virtual thread, it causes a memory leak. With -Dcats.effect.tracing.mode=NONE, fiber tracing is disabled and there is no leak anymore.

This sounds like a bug to me! Would you mind opening an issue for it?

[jacum]

The test code being executed is just

 def doSomethingBlocking(dispatcher: Dispatcher[IO]): IO[Unit] =
    for {
      _ <- IO.delay {  dispatcher.unsafeRunSync(IO(UUID.randomUUID())) }
    } yield ()

i.e. the only thing tested is the parallel processing of blocking contexts.
Thanks, I will check out the benchmark you mentioned!

---

"no substantial difference measured on non-blocking effects" - this observation has been made when the IORuntime of some of real services was changed, and comparing results by running our production load simulation suites.

Some more background on our context:

• We have many dfferent services, some of them are pure tagless-final Cats Effect, whereas other include earlier generation libraries and frameworks, e.g. Akka and its custom extensions, to which there is no (yet) suitable replacement in F[_] ecosystem.

• Unfortunately, very much appropriate SyncIO is not used, and *[_] types across the codebase often represent both evaluations (like the crypto thing in http4s I mistakenly thought to be Future/await), and true non-local effects such as an I/O operation.
  When such Sync[_] effect code happen to run inside Akka command handler or some custom Akka extension, written in single-thread semantics, the value is synchronously extracted by Dispatcher#unsafeRunSync - which is, in this case, truly and genuinely as 'unsafe' as it could be.

• When load increases (and it comes in bursts by nature of the application), there are spikes of threads in TIMED_WAITING state (of which all are io-compute-blocker-xxx ones), which become more and more pronounced with higher load - until this starts badly affecting the latency of process execution. Allocating more CPUs helped just to a very limited extent.

• The codebase is rather large and critical enough, but could not be refactored without substantial risks.

We went looking for some stopgap solution. It was indeed somewhat similar to SyncIO. JDK21 was yet another option on stopgaps list. When running on JDK21 as is, just with replaced IORuntime, the dreadful spikes in TIMED_WAITING threads do not appear and latency isn't affected.

Running services that were not affected by such bad code combination on JDK21 vs. default Cats Effect WorkStealingPool won't reveal any substantial difference - likely the bottleneck isn't in the runtime anymore.

---

Going to test your 3.6 snapshot some time soon.

#3929 - code to reproduce leaking fiber monitors.