[question] Is caffeine virtual thread / loom friendly?

[wyhasany]

I wonder if caffeine is loom-friendly - e.g. does not pin the platform threads?

[He-Pin]

Yes and no
Concurrenthashmap is blocking the carrier thread. They plan to fix it,but not jdk23

[ben-manes]

Currently in Java, synchronized pins the virtual thread. That is being worked on currently and will be resolved soonish (status).

Caffeine uses ConcurrentHashMap which is built on synchronized blocks, such as for computeIfAbsent. Since the issue will be resolved soon and the Java team is not rewriting all of their collections towards ReentrantLock, we'll following their lead by waiting for the platform to catch up. There are a few other areas where virtual threads are penalized compared to platform threads such as the lack of preemption, network address resolution (JEP 418), need for better async io (io_ring), scalable timers, and more. It will be a long transition until VT are preferred in all cases.

In the meantime, you can use AsyncCache which decouples the computation from the map operation. Then the map's synchronized blocks are short cpu work and the I/O is performed by a CompletableFuture (which is VT friendly). See
#779 for an example.

[He-Pin]

Or maybe just set a larger pool size。

[wyhasany]

@ben-manes AsyncCache seems to be a perfect solution from my perspective 🎉 do you plan to rewrite synchronized blocks to ReentrantLocks in caffeine codebase?

[He-Pin]

I think we may better follow Hotspot team's lead .

[wyhasany]

@He-Pin what do you mean by follow?

[ben-manes]

Since ConcurrentHashMap will pin the thread in its synchronized blocks, there is not much of an incentive to change Caffeine's usage. Theirs is per-hashbin whereas ours is per-entry. For short cpu work needed by AsyncCache it is perfectly fine to pin as VTs do not preempt. For a synchronized cache then ConcurrentHashMap's compute will pin regardless, so any effort we make will not be helpful.

A few years ago I did chat with Doug Lea who was pondering what a change to ConcurrentHashMap might look like. He had mused using "separate bitwise-trie-like structure of AQS objects that serve as the inflated forms. One TBD is how/when to kill them off", where AQS is the synchronization primitive underlying ReentrantLock. I haven't seen any progress there whereas the Loom github shows steady progress on mutex support.

You might follow loom-dev and the github branches to keep an eye on progress.

[He-Pin]

I just upgrade my system to jdk21 and use both virtual thread, and removed some old guava code , swith to locksupport etc and switch to caffeine, after I trace the pin.

I was trying to submit pr to cchm too , but seems not much good way, otherwise it will wast memory.

I also asked on Loom dev, they are working on it , but no timeline，and cchm will not be rewritten.

I see 10％ cpu reduce and 5% mem reduce on 16core 32gb memory box, and at the workload 60% and where with VT is 50% on the same workload. This is a mission critical system of taobao live.

That's what I can share and I think you can just follow author's advice, that what we can best do for now.

[ben-manes]

FYI - The new EA builds no longer pin on synchronization (announcement) and they are asking for usage feedback.

[jkvargas]

hi, quick question
what is the actual status of caffeine and virtual threads?

[ben-manes]

Per the link above, the Java team's work should resolve this in an upcoming release. You can try their early access build to provide feedback. For the time being, virtual threads should be used judiciously in limited scenarios until many of the show stopper problems have been resolved by the JDK. Likely the next LTS (25) will be a good time to consider using it.