PHOENIX-6751 Force using range scan vs skip scan when using large IN clause #1495
Conversation
|
@tkhurana Please review |
1 similar comment
|
@tkhurana Please review |
| // is below the configured max (maxInListSkipScanSize). | ||
| // We shall force a range scan if the configured max is exceeded. | ||
| // cnfStartPos => is the start slot of this IN list | ||
| if (checkMaxSkipScanCardinality) { |
There was a problem hiding this comment.
Can this conversion of skip scan to range scan configurable because range scan on bigger data sets is slow?
There was a problem hiding this comment.
There's already a config param to control how many elements triggers the conversion, which could be used to turn it off by setting it very high. However, when we get in this state (high cardinality skip scans) we find that we get OOM exceptions even with large client-side heaps, which is worse than a slow query.
There was a problem hiding this comment.
@chrajeshbabu - do you still have concerns about this patch or is it ready to be merged?
There was a problem hiding this comment.
@chrajeshbabu For some queries especially with RVC expression and mixed sort orders, the cost of optimization results in huge memory allocations and sometimes even exceeds the number of KEY_RANGES allowed This JIRA provides a framework for opting out the optimization path when a certain threshold is reached. We will be working towards an algorithm that is more linear in nature than combinatorial as is the case today.
@gjacoby126 The difference is the use of biginteger to avoid overflow issues when determining whether to use skip scan or range scan. |
No description provided.