[question] The lazyness of expiration

[ClSlaid]

Hello maintainer, I'd like to know whether the expiration of cache is lazy or active.
By lazy I mean not until getting on the expired entry or other scenarios that have to remove it, the entry will be preserved in the cache;
Corresponsively by active I mean if the entry outlived its TTL or IDLE, the entry will be instantly evicted.
I've also used this crate to build a server-side connection pool, and I hope any outlived connections should be dropped immediately. Nothing about this was in the README, but by invalidate_entries_if() method I guess this cache should be lazy?

[tatsuya6502]

Hi. Thank you for using Moka.

1. The expiration is lazy by design, so expired entries will be preserved in the cache for a short period of time.

   • As of v0.8.3, they will be usually removed from the cache within 300 milliseconds after expiration, but this delay can get longer if the cache is receiving very heavy reads and/or writes from clients.
   • The 300 ms delay (actually a clean-up interval) is currently hard-coded, but we can make it configurable in the future.
   • If you are curious about internals, there is a (little, work-in-progress) doc about it: https://docs.rs/moka/latest/moka/#implementation-details

2. There is also a known issue in an underlying library (crossbeam-epoch) that will prevent entries from dropping for much longer period after the removal from the cache.

   • Our cache internally uses a lock-free hash table to store the entries and it uses crossbeam-epoch for epoch-based garbage collection (GC): https://docs.rs/crossbeam-epoch/latest/crossbeam_epoch/
   • crossbeam-epoch's garbage collector is not optimal for some use cases and may keep ~100 garbages (objects waiting to be dropped) in the heap. We are hitting this issue and some entries will not be dropped immediately.

As for 1, please let me know whether you are okay with the current delay or not. We could make the delay configurable per cache.

As for 2, let me explore some ideas to resolve/mitigate the GC issue.

[ClSlaid]

Hi. Thank you for using Moka.

1. The expiration is `lazy` by design, so expired entries will be preserved in the cache for a short period of time.
   
   * As of v0.8.3, they will be usually removed from the cache within 300 milliseconds after expiration, but this delay can get longer if the cache is receiving very heavy reads and/or writes from clients.
   * The 300 ms delay (actually a clean-up interval) is currently hard-coded, but we can make it configurable in the future.
   * If you are curious about internals, there is a (little, work-in-progress) doc about it: https://docs.rs/moka/latest/moka/#implementation-details

2. There is also a known issue in an underlying library (crossbeam-epoch) that will prevent entries from dropping for much longer period after the removal from the cache.
   
   * Our cache internally uses a lock-free hash table to store the entries and it uses crossbeam-epoch for epoch-based garbage collection (GC): https://docs.rs/crossbeam-epoch/latest/crossbeam_epoch/
   * crossbeam-epoch's garbage collector is not optimal for some use cases and may keep ~100 garbages (objects waiting to be dropped) in the heap. We are hitting this issue and some entries will not be dropped immediately.

As for 1, please let me know whether you are okay with the current delay or not. We could make the delay configurable per cache.

As for 2, let me explore some ideas to resolve/mitigate the GC issue.

Thanks for your reply, I think preserving an evicted connection for less than 3s is ok (after all the connection pool is just a toy project), and it seems hardly occur.

And I would like to take a deeper look into Moka's internal functionalities, as when I graduate from university. :)

[tatsuya6502]

as when I graduate from university. :)

Congratulations! 🎉

I would like to take a deeper look into Moka's internal functionalities,

I wish I had more time to write design docs, so all you can read now are:

• The one that I already mentioned: https://docs.rs/moka/latest/moka/#implementation-details
• A wiki page about admission and eviction policies: https://github.com/moka-rs/moka/wiki
• And the source code.

Moka's design is heavily inspired by the Caffeine library for Java. So you can learn the base design from Caffeine's publications: https://github.com/ben-manes/caffeine#in-the-news

I started to develop Moka because I needed a concurrent cache for the storage layer of a NoSQL database (as my hobby project) and found there was no such cache library in Rust that would meet my needs. I thought I will finish implementing Moka in a couple of months so I can go back to my original project quickly. I never thought it would take some years (but I am enjoying).

[SimonSchneider]

crossbeam-epoch's garbage collector is not optimal for some use cases and may keep ~100 garbages (objects waiting to be dropped) in the heap. We are hitting this issue and some entries will not be dropped immediately.

This one could be something we are actually noticing. Our cache entries are quite large and not that many so having about ~100 of them hanging around might look like a memory leak to us. Is there any way to mitigate this?

[tatsuya6502]

@SimonSchneider

Sorry for not getting back to you sooner.

I wonder if we could mitigate it by using the flush method of crossbeam_epoch::Guard.

https://docs.rs/crossbeam-epoch/0.9.9/crossbeam_epoch/struct.Guard.html#method.flush

Clears up the thread-local cache of deferred functions by executing them or moving into the global cache.

Call this method after deferring execution of a function if you want to get it executed as soon as possible. Flushing will make sure it is residing in in the global cache, so that any thread has a chance of taking the function and executing it.

We use some "deferred functions" to drop entries removed from cache.

Let me try it in next few days. I will measure performance impacts for both memory usage and speed.

[SimonSchneider]

No worries, thanks for getting back to me.

Thank you for looking into it! Sounds like an approach that could work.

[tatsuya6502]

As for the flush stuff, I am working on it here: #169

[tatsuya6502]

I merged #169 into the master branch. I will release it as a part of v0.9.3.

Closing this issue.

[tatsuya6502]

Sorry, it took longer than I thought. But I have published Moka v0.9.3 to crates.io.