fix(cache): replace in-memory table with an lru-cache to limit memory consumption

[Tieske]

As the intent is to replace the caching all together, I did not make its size configurable

See code comments on some rough calculations. Configurability can be added if required, feedback welcome

[p0pr0ck5]

How hard would it be to pull our our implementation of TTLs with lua-resty-lrucache's TTL mechanism?

[p0pr0ck5]

what do we expect the average cache hit ratio to look like? if we expect a lot of cache misses and purges, should we consider resty.lrucache.pureffi (or making it configurable?)

[Tieske]

Depends on customer use. I'd expect it to be relatively high, based on "sessions", what I mean is that you'll most likely get multiple requests for a specific use. In that case the apis, cunsomers, credentials etc. will be the same for that session. Only when done the next session comes up.

but as said very speculative.

[Tieske]

How hard would it be to pull our our implementation of TTLs with lua-resty-lrucache's TTL mechanism?

Impossible without modifying the lru library. We need to share the ttl between the lru cache and the shm cache. But if we get an existing entry from shm, we cannot get its remaining ttl. Hence we must have our own ttl field inserted.

[subnetmarco]

The size should still be configurable. We currently support mem_cache_size in the configuration file, so it should be configurable from there (or another property)?

[thibaultcha]

How about using the same value as mem_cache_size? There should however, be an upper limit as we know the LuaJIT VM memory limit is quite low, but as long as we are in a 1...500mb range, I think it's reasonably to have the shm and the Lua-land LRU cache to be the same size. Thoughts?

[p0pr0ck5]

The value of mem_cache_size doesn't quite make sense, because lrucache wants a number of items, not a size limit, so the limits of the shdict and the lrucache will not be the same. And given that the usage of the lrucache will vary based entity size, which isn't something that can easily be documented (remember, people want hard numbers, which isnt something that can easily be done here), I think making this configurable will only confuse users. I'd vote to keep it small (maybe a few thousand elements) and leave it alone- it's a transparent cache tier, and doesn't need to be exposed on such a granular level to users.

[subnetmarco]

Also, mem_cache_size was being used once in the master process, while the LRU cache has a size limit for each worker. Which means that the actual total memory used would be:

(mem_cache_size * num_workers) + master_shd_size

[thibaultcha]

The same way we compute the number of items in the current patch with a hard-coded estimated item size, we could just as well do that by replacing this PR's MEM_SIZE with mem_cache_size.

In any case, I'm even more in favor of simply hard-coding this value.

[Tieske]

There should however, be an upper limit as we know the LuaJIT VM memory limit is quite low, but as long as we are in a 1...500mb range

That's exactly why I picked the current size. The item size of 1024 is probably a lot bigger than we actually use. The growth we suspect is mostly based on the db-cache misses (standard data does not grow, it, at its maximum, consists of the entire datastore contents, and never more)
The db-miss sentinels are small, so my guess is that the maximum memory consumption per worker will be less than 250mb anyway.

In any case, I'm even more in favor of simply hard-coding this value.

so am I.