[enhancement](Nereids) Speedup PartitionPrunner #31970
Conversation
|
Thank you for your contribution to Apache Doris. |
|
run buildall |
|
Any performance data? |
924060929
force-pushed
the
optimize_partition_prunner
branch
from
5d0a639
to
08c4e0f
Compare
|
run buildall |
|
run buildall |
100 threads parallel continuous send this sql which query an empty table, test in my mac machine(m2 chip, 8 core) select count(1),date_format(time_col,'%Y%m%d'),varchar_col1
from tbl
where partition_date>'2024-02-15' and (varchar_col2 ='73130' or varchar_col3='73130') and time_col>'2024-03-04'
and time_col<'2024-03-05'
group by date_format(time_col,'%Y%m%d'),varchar_col1
order by date_format(time_col,'%Y%m%d') desc, varchar_col1 desc,count(1) asc
limit 1000before: 1600 peak QPS, about 1300 avg QPS |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
| this.slotToType = IntStream.range(0, partitionSlots.size()) | ||
| .mapToObj(index -> Pair.of(partitionSlots.get(index), partitionSlotTypes.get(index))) | ||
| .collect(ImmutableMap.toImmutableMap(Pair::key, Pair::value)); | ||
| this.slotToType = Maps.newHashMapWithExpectedSize(partitionSlots.size() * 2); |
There was a problem hiding this comment.
Minor: use Maps.newHashMapWithExpectedSize(partitionSlots.size()); may good enough, can update the code in next pr
There was a problem hiding this comment.
I will change to 16 in next pr, because the num of partitionSlots usually <= 3
This pr imporve the high QPS query by speed up PartitionPrunner 1. remove useless Date parse/format, use LocalDate instead 2. fast evaluate path for single value partition 3. change Collection.stream() to ImmutableXxx.builderWithExpectedSize(n) to skip useless method call and collection resize 4. change lots of if-else to switch 5. don't parse to string to compare dateLiteral, use int field compare instead
Proposed changes
This pr imporve the high QPS query by speed up PartitionPrunner
LocalDateinsteadCollection.stream()toImmutableXxx.builderWithExpectedSize(n)to skip useless method call and collection resize