The value-based segment pruners (ColumnValueSegmentPruner and BloomFilterSegmentPruner) skip segment pruning for IN (...) predicates when the clause contains more than inpredicate.threshold values. The default is 10 (Server.DEFAULT_VALUE_PRUNER_IN_PREDICATE_THRESHOLD).
For queries with large IN lists the current behavior is that pruners are bypassed entirely and every segment is scanned, which is slow and in large Pinot cluster, reduces overall capacity for other queries. The only way to raise the threshold today is the per-server config
pinot.server.query.executor.pruner.<pruner>.inpredicate.threshold, which requires restarting the servers, which can take a while.
I propose adding a boolean query option like so:
SET forceInPredicatePruning=true;
When true, the value pruners always attempt to prune segments for IN predicates regardless of the clause size. Default is false, preserving existing behavior.
The value-based segment pruners (
ColumnValueSegmentPrunerandBloomFilterSegmentPruner) skip segment pruning forIN (...)predicates when the clause contains more thaninpredicate.thresholdvalues. The default is 10 (Server.DEFAULT_VALUE_PRUNER_IN_PREDICATE_THRESHOLD).For queries with large IN lists the current behavior is that pruners are bypassed entirely and every segment is scanned, which is slow and in large Pinot cluster, reduces overall capacity for other queries. The only way to raise the threshold today is the per-server config
pinot.server.query.executor.pruner.<pruner>.inpredicate.threshold, which requires restarting the servers, which can take a while.I propose adding a boolean query option like so:
When true, the value pruners always attempt to prune segments for
INpredicates regardless of the clause size. Default is false, preserving existing behavior.