LQP and/or PQP caches take up GBs

[mrks]

I was investigating why the memory consumption was increasing constantly and not dropping to idle levels between different TPC-H items:

The biggest jump in the orange line is TPC-H 15. I then ran ./hyriseBenchmarkTPCH -q 14,15,16 --metrics -o bothcaches.json on the current master:

For Q15, you can see the point where all query variations are cached. As there are only 58 of them, it seems as if each takes 13 MB.

Here it is with the PQP cache disabled:

And here without any plan caching:

Thoughts:

• Check why the LQP cache entries are that big in the first place.
• Evict more aggressively?
• Track actual cache size to help with eviction decisions?
• Does query parametrization help?
• How to detect these issues in the future?

[mrks]

More thoughts:

• Some operators (Delete, Insert, UnionPositions?) store their input tables. Operators should not do that, as it prevents tables from being released. If they do, they should at least implement _on_cleanup. Not sure how to enforce this.
• This gets aggravated when these operators are cached. Should we move to a model where a deep copy of the PQP is stored in the PQP cache? That would prevent operators that do not properly clean up after themselves from wasting memory once they are in the cache and only used for reproduction (hihi). However, it might make @Bouncner and @Bensk1 sad.
• The last operator's output is cached indefinitely as OperatorTask only calls clear_output on predecessors.
• Work done so far can be found in mrks/caching_explosion

[mrks]

Should ~SQLPipelineStatement call clear_output on the top operator? @julianmenzler

[julianmenzler]

For Q15, you can see the point where all query variations are cached. As there are only 58 of them, it seems as if each takes 13 MB.

The current changes of #2254 seem to have reduced the memory consumption for Q15. Per query variation, from 13 MB down to ~1.5 MB.

[mrks]

@julianmenzler Fixed in #2254?

[julianmenzler]

@julianmenzler Fixed in #2254?

I think so:

(TPC-H SF 1, 7cfa1a5)