Caching Improvements (Old version with memory size limit, discard) #279
Conversation
The theory behind caching aborted Operations was that they would fail again anyway. However, under memory pressure Operations may fail due to failing allocations and will succeed in the future once memory pressure drops, if we kept them in the cache we wouldn't try again. While we are at it, improve wording and use an extra abort() method on Operation.
Instead of limiting the number of elements in the cache limit the total memory used by the cached elements. This currently uses a free template function to let elements give their size in memory. For the results in QLever we currently ignore the memory used by RuntimeInformation. We also ignore the memory used by the cache itself
|
I'm really not happy with the second commit. I think limiting the memory size makes most cache operations fundamentally O(n) where n is the number of elements in the cache, which can be quite large if we have a lot of small elements. As a new value with size approaching or exceeding the capacity is added we are forced to delete everything else. In a first test on Wikidata Full even with a cache size of 20 GB completion times rise from 70 ms on average to over 128 ms. Even more problematic when we have several large tables that are often accessed but don't all fit in the cache we will constantly be recomputing those that didn't fit before. With an element based capacity on the other hand we can simply remove the longest untouched value in O(1) for any new value once we reach capacity. What do you think @hannahbast @floriankramer? |
niklas88
changed the title
The theory behind caching aborted Operations was that they would fail
again anyway. However, under memory pressure Operations may fail due to
failing allocations and will succeed in the future once memory pressure
drops, if we kept them in the cache we wouldn't try again.
While we are at it, improve wording and use an extra abort() method on
Operation.
As a next step we should make the cache limited by the amount of memory not the number of cached results.