Compare caching implementation with Caffeine

[azell]

It might be simpler to drop in https://github.com/ben-manes/caffeine that writing your own cache implementation.

[ben-manes]

A quick benchmark shows the new cache to be about 2x the throughput of a synchronized LinkedHashMap in LRU mode. Its crawling thread skews benchmarks a bit as there doesn't appear to be a clean way to halt it afterwards, which hurts other caches by stealing its cpu time. So I can't check this in as it is too disruptive to my tooling.

Benchmark	LinkedHashMap	Rapidoid	Caffeine
read	8 M/s	21 M/s	145 M/s
read/write	8 M/s	15 M/s	110 M/s
write	6 M/s	15 M/s	65 M/s

The efficiency shows the cache uses a FIFO-like policy. Its slightly worse than FIFO due to using buckets (aka segments) which may cause premature evictions. The multi2 trace is probably fair for what users might experience, whereas gli is loopy so it shows robustness against a more hostile user.

Policy	multi2	gli
Fifo	28.97 %	0.96 %
Lru	36.37 %	0.98 %
Optimal	53.85 %	35.13 %
Rapidoid	28.22 %	0.95 %
Caffeine	48.86 %	33.82 %

[nmihajlovski]

I apologize for the late reply.

The benchmark results for Caffeine look impressive, congrats @ben-manes ! It would be nice to see the same benchmarks executed with 256 / 512 / 1024 threads, just to check for performance degradation.

I am going to provide an easy way to stop the crawling thread, and I will write again when it is ready. It would be great to see Rapidoid cache officially in the benchmarks. :)

Regarding the integration of Caffeine into Rapidoid:

• Rapidoid is still targeting JDK 7+ (as a framework), looks like Caffeine is for JDK 8.
• Rapidoid's cache implementation is very small and simple, it's good to have it as default in the framework - without external dependencies,
• Soon I would expose API for cache customization in the framework, it would be easy to integrate external cache implementations (e.g. Caffeine),
• I would provide out-of-the-box integration for Caffeine,
• After these steps, I will have more experience and insights into Caffeine, so I will experiment and consider including it as primary cache implementation for the dockerized Rapidoid Platform.

[ben-manes]

Perhaps then you might prefer embedding ConcurrentLinkedHashMap instead. That was a popular library that we ported parts into Guava to add caching functionality. It's small and easy to embed, e.g. Groovy. Most users, e.g. Cassandra, have upgraded to Caffeine. CLHM doesn't use any threads and amortizes maintenance on calling threads.

That would give you most of the performance. However it is strictly a concurrent LRU map. You would need to add memoization and expiation as decorators. The library's code is fairly straightforward so you could prune away the general map methods. If you were up to it, you could add expiration directly into the map if use a time bounded fifo approach.

All that might sound complex until you understand the design; slides. It uses a write ahead log to record and replay events. This avoids contention as the number of threads or cores increase, as it's appending to a buffer rather than locking.

If you go down that approach, generalizing the api to plugin Caffeine might not be necessary. You'd have a small, built in, and fast cache.

[nmihajlovski]

Thanks for sharing the resources/slides, the design/architecture looks smart and interesting.

Since Rapidoid's built-in cache is already good-enough as a default, I wouldn't switch to CLHM, because it requires significant effort for me to reimplement the same features again on top of CLHM.

Last week I made some improvements and implemented proper resource clean-up of the Rapidoid's cache. Starting from v5.3.3, simply calling Caching.shutdown() will terminate the scheduler threads. It would be interesting to see how it performs, compared to the other cache implementations. I would be happy to help to add it to your benchmarks...

Finally, I am still optimistic about including Caffeine into the Rapidoid platform in future (the framework targets JDK 7+ for now, but the platform is on JDK 8).

[ben-manes]

I'll try to get to it in the next day or two. If you'd like to take a stab at it, see this commit adding ExpiringMap. The only difference would be to override default methods to call shutdown.

[nmihajlovski]

Sure, I would give it a try later this week...