Increase QET cache and STXXL memory size #96
Conversation
|
I also tried with a cache size of 100000 but that used too much memory running out of RAM even on vulcano. |
There was a problem hiding this comment.
-
I think we have saved enough RAM during the index creation to allow this extra GB for STXXL
-
when I got you correctly, it takes a lot of time to (stxxl)-sort the text indices. Would even more RAM help there? If so we could somehow pass this information in at runtime if we know that we have say 20 GB to spare.
-
I don't know much about the implications of the query execution tree cache size and trust you there. Could it in theory be beneficial to also store subtree results on disk? I think there could be subtrees which are much faster to read from disk than to compute and this could be good for systems that run a lot of similar queries (is AQQU one of those?)
But for the current state this is of course fine.
We're saving a lot of RAM with more efficient storage of the OPS/OSP meta data and soon with vocabulary improvements. So we should put some of that saved memory to good use. Thus this commit increases the size of the query execution tree (QET) cache ten fold. It also doubles the amount of memory STXXL may use. When testing with Aqqu with a pure QLever backend this increases query performance by at least an order of magnitude with minimal increase im RAM use.
|
I'm still figuring out a good cache size but let's keep this PR open to track it. |
niklas88
moved this from Approved waiting on decision/external
to Review Approved
in QLever
niklas88
moved this from Review Approved
to Approved waiting on decision/external
in QLever
|
Ok so I'm still not sure this is the ideal size but 1000 subtrees works well and doesn't easily run out of memory. |
We're saving a lot of RAM with more efficient storage of the OPS/OSP
meta data and soon with vocabulary improvements. So we should put some
of that saved memory to good use.
Thus this commit increases the size of the query execution tree (QET)
cache ten fold. It also doubles the amount of memory STXXL may use.
When testing with Aqqu with a pure QLever backend this increases query
performance by at least an order of magnitude with minimal increase im
RAM use.
That said the 10-fold is currently quite arbitrary and likely still orders of magnitude less than what our virtuoso instance for Freebase has when combined with the varnish cache in front. Especially since we're caching just subtrees.