Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

query performance (version 23.7.2) #53027

Closed
yangshike opened this issue Aug 4, 2023 · 15 comments
Closed

query performance (version 23.7.2) #53027

yangshike opened this issue Aug 4, 2023 · 15 comments
Labels
performance

Comments

@yangshike
Copy link

yangshike commented Aug 4, 2023 •
edited by novikd

today ,I upgraded from 23.5 to 23.7.2，because ：23.7 （Fix for moving 'IN' conditions to PREWHERE #51610 (Alexander Gololobov)）

Unfortunately，a disaster has occurred.
query in 23.7 ： 616 rows in set. Elapsed: 27.013 sec. Processed 517.96 million rows, 177.16 GB
query on 23.5: 616 rows in set. Elapsed: 0.166 sec. Processed 384.01 thousand rows, 135.14 MB

this is my query：

SELECT 
   live_start_time,
   camp_date_id, course_id , class_id , min(`camp_date_name`) camp_date_name, min(`class_name`) class_name, min(`order_number`) `order_number`, min(`flow_channel`) `flow_channel`, min(`flow_type`) `flow_type`, min(`flow_source`) `flow_source`, min(`channel_name`)`channel_name` , min(`sub_channel_name`) `sub_channel_name`, sum(apply_count) apply_count, sum(join_count) join_count, sum(recall_count) recall_count, sum(come_count) come_count, sum(come_convert_count) come_convert_count, sum(effective_count) effective_count, sum(thirty_count) thirty_count, sum(sixty_count) sixty_count, sum(two_hour_count) two_hour_count, sum(base_live_high_order_count) base_live_high_order_count, sum(popup_base_live_high_order_count) popup_base_live_high_order_count, sum(live_high_order_count) live_high_order_count, sum(live_high_order_amount) live_high_order_amount, sum(sale_high_order_count) sale_high_order_count, sum(sale_high_order_amount) sale_high_order_amount, sum(refund_high_order_count) refund_high_order_count, sum(refund_high_order_amount) refund_high_order_amount, sum(unpaid_order_count) unpaid_order_count, ifnull(sum(new_unpaid_count),0) new_unpaid_count, min(`created_time`) `created_time`
   FROM t_bi_live_watcher_stat right join 
      (select
         camp_date_id camp_date_id_version , live_start_time live_start_time_version, max(version) max_version from t_bi_live_watcher_stat where camp_date_id in (96475154519,4421504077,2399864193058,180928 ) 
      group by camp_date_id, live_start_time) t_version_one
      ON t_bi_live_watcher_stat.camp_date_id = t_version_one.camp_date_id_version and t_bi_live_watcher_stat.live_start_time = t_version_one.live_start_time_version and t_bi_live_watcher_stat.version = t_version_one.max_version
where camp_date_id in (475154519,3074421504077,864193058,30214180928 )
group by live_start_time, camp_date_id, course_id , class_id order by order_number asc, live_start_time asc FORMAT TabSeparatedWithNamesAndTypes;
@yangshike yangshike added the potential bug To be reviewed by developers and confirmed/rejected. label Aug 4, 2023
@yangshike
Copy link
Author

Is that the reason？？
Can I disable this function？
Grace Hash Join algorithm is now applicable to FULL and RIGHT JOINs. #49483. #51013 (lgbo).

@yangshike
Copy link
Author

yangshike commented Aug 4, 2023 •
edited by novikd

This is main trace_log on 23.5:

[clickhouse01] 2023.08.04 16:32:19.244778 [ 28705 ] {f08f7512-ab5a-43d1-855e-a057c362667f} <Debug> executeQuery: Query span trace_id for opentelemetry log: 00000000-0000-0000-0000-000000000000
[clickhouse01] 2023.08.04 16:32:19.245448 [ 28705 ] {f08f7512-ab5a-43d1-855e-a057c362667f} <Debug> executeQuery: (from 127.0.0.1:43544)  SELECT live_start_time, camp_date_id, course_id , class_id , min()
camp_date_name, min() class_name, min() , min() , min() , min() , min() , min() , sum(apply_count) apply_count, sum(join_count) join_count, sum(recall_count) recall_count, sum(come_count) come_count, sum(
come_convert_count) come_convert_count, sum(effective_count) effective_count, sum(thirty_count) thirty_count, sum(sixty_count) sixty_count, sum(two_hour_count) two_hour_count, sum(base_live_high_order_cou
nt) base_live_high_order_count, sum(popup_base_live_high_order_count) popup_base_live_high_order_count, sum(live_high_order_count) live_high_order_count, sum(live_high_order_amount) live_high_order_amount
, sum(sale_high_order_count) sale_high_order_count, sum(sale_high_order_amount) sale_high_order_amount, sum(refund_high_order_count) refund_high_order_count, sum(refund_high_order_amount) refund_high_orde
r_amount, sum(unpaid_order_count) unpaid_order_count, ifnull(sum(new_unpaid_count),0) new_unpaid_count, min() FROM t_bi_live_watcher_stat right join ( select camp_date_id camp_date_id_version , live_start
_time live_start_time_version, max(version) max_version from t_bi_live_watcher_stat where camp_date_id in (733443096475154519,733443074421504077,733442399864193058,733441730214180928 ) group by camp_date_
id, live_start_time ) t_version_one on t_bi_live_watcher_stat.camp_date_id = t_version_one.camp_date_id_version and t_bi_live_watcher_stat.live_start_time = t_version_one.live_start_time_version and t_bi_
live_watcher_stat.version = t_version_one.max_version where camp_date_id in (733443096475154519,733443074421504077,733442399864193058,733441730214180928 )group by live_start_time, camp_date_id, course_id
, class_id order by order_number asc, live_start_time asc (stage: Complete)
[clickhouse01] 2023.08.04 16:32:19.247476 [ 28705 ] {f08f7512-ab5a-43d1-855e-a057c362667f} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Looking for mutations for part 2481921
26369976320_2600_2624_9_2625 (part data version 2625, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565489 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57938776219489
1269_606_611_1 (part data version 606, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565530 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Looking for mutations for part 5793877
62194891269_612_619_2 (part data version 612, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565561 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57938776219489
1269_612_619_2 (part data version 612, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565600 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Looking for mutations for part 5793885
31615269102_607_620_2 (part data version 607, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565628 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57938853161526
9102_607_620_2 (part data version 607, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565668 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Looking for mutations for part 5793892
63496114355_611_624_2 (part data version 611, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565702 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57938926349611
4355_611_624_2 (part data version 611, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565740 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Looking for mutations for part 5793899
94798182621_608_621_2 (part data version 608, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565762 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57938999479818
2621_608_621_2 (part data version 608, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565798 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Looking for mutations for part 5793906
24409383262_602_612_2 (part data version 602, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565831 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57939062440938
3262_602_612_2 (part data version 602, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565866 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Looking for mutations for part 5793906
24409383262_613_615_1 (part data version 613, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565895 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57939062440938
3262_613_615_1 (part data version 613, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565935 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Looking for mutations for part 5808466
34080727850_116_118_1 (part data version 116, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.565967 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 58084663408072
7850_116_118_1 (part data version 116, part metadata version 0)
[clickhouse01] 2023.08.04 16:33:33.576396 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "camp_date_id IN (733443096475154519, 73344307
4421504077, 733442399864193058, 733441730214180928)" moved to PREWHERE
[clickhouse01] 2023.08.04 16:33:33.576786 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> ContextAccess (default): Access granted: SELECT(camp_date_id, live_start_time, version) ON click_kuaicaib
i.t_bi_live_watcher_stat
[clickhouse01] 2023.08.04 16:33:33.578161 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
[clickhouse01] 2023.08.04 16:33:33.578318 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> TableJoin: Left JOIN converting actions: empty
[clickhouse01] 2023.08.04 16:33:33.578326 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> TableJoin: Right JOIN converting actions: empty
[clickhouse01] 2023.08.04 16:33:33.578364 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> HashJoin: (0x7f3d47dfe098) Datatype: EMPTY, kind: Right, strictness: All, right header: camp_date_id_vers
ion Int64 Int64(size = 0), live_start_time_version Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), max_version Int64 Int64(size = 0)
[clickhouse01] 2023.08.04 16:33:33.578393 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> HashJoin: (0x7f3d47dfe098) Keys: [(camp_date_id, live_start_time, version) = (camp_date_id_version, live_
start_time_version, max_version)]
[clickhouse01] 2023.08.04 16:33:33.579483 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> ContextAccess (default): Access granted: SELECT(camp_date_id, class_id, camp_date_name, class_name, cours
e_id, live_start_time, order_number, flow_channel, flow_type, flow_source, channel_name, sub_channel_name, apply_count, join_count, recall_count, come_count, come_convert_count, effective_count, thirty_co
unt, sixty_count, two_hour_count, base_live_high_order_count, live_high_order_count, live_high_order_amount, sale_high_order_count, sale_high_order_amount, unpaid_order_count, new_unpaid_count, created_ti
me, refund_high_order_count, refund_high_order_amount, version, popup_base_live_high_order_count) ON click_kuaicaibi.t_bi_live_watcher_stat
[clickhouse01] 2023.08.04 16:33:33.580571 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
[clickhouse01] 2023.08.04 16:33:33.580803 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> JoiningTransform: Before join block: 'camp_date_id Int64 Int64(size = 0), class_id Nullable(Int64) Nullab
le(size = 0, Int64(size = 0), UInt8(size = 0)), camp_date_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), class_name Nullable(String) Nullable(size = 0, String(size = 0), UInt
8(size = 0)), course_id Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_start_time Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), order_number Nullable(In
t32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_channel Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_type Nullable(Int32) Nullable(size = 0, Int32(size = 0),
UInt8(size = 0)), flow_source Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), channel_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), sub_channel_name Nu
llable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), apply_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), join_count Nullable(Int64) Nullable(size = 0, Int64(
size = 0), UInt8(size = 0)), recall_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_conver
t_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), effective_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), thirty_count Nullable(Int64) Nullable(si
ze = 0, Int64(size = 0), UInt8(size = 0)), sixty_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), two_hour_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size =
 0)), base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high
_order_amount Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_amount Nulla
ble(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), unpaid_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), new_unpaid_count Nullable(Int64) Nullable(size = 0
, Int64(size = 0), UInt8(size = 0)), created_time DateTime UInt32(size = 0), refund_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), refund_high_order_amount Nullable
(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), version Int64 Int64(size = 0), popup_base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0))'
[clickhouse01] 2023.08.04 16:33:33.580939 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> JoiningTransform: After join block: 'camp_date_id Int64 Int64(size = 0), class_id Nullable(Int64) Nullabl
e(size = 0, Int64(size = 0), UInt8(size = 0)), camp_date_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), class_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8
(size = 0)), course_id Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_start_time Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), order_number Nullable(Int
32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_channel Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_type Nullable(Int32) Nullable(size = 0, Int32(size = 0), U
Int8(size = 0)), flow_source Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), channel_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), sub_channel_name Nul
lable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), apply_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), join_count Nullable(Int64) Nullable(size = 0, Int64(s
ize = 0), UInt8(size = 0)), recall_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_convert
_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), effective_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), thirty_count Nullable(Int64) Nullable(siz
e = 0, Int64(size = 0), UInt8(size = 0)), sixty_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), two_hour_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size =
0)), base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_
order_amount Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_amount Nullab
le(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), unpaid_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), new_unpaid_count Nullable(Int64) Nullable(size = 0,
 Int64(size = 0), UInt8(size = 0)), created_time DateTime UInt32(size = 0), refund_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), refund_high_order_amount Nullable(
Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), version Int64 Int64(size = 0), popup_base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0))'
[clickhouse01] 2023.08.04 16:33:33.584465 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Key condi
tion: (column 0 in 4-element set)
[clickhouse01] 2023.08.04 16:33:33.591318 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): MinMax in
dex condition: (column 0 in 4-element set)
[clickhouse01] 2023.08.04 16:33:33.591578 [ 19840 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part 733442399864193058_797_797_0 with 3 steps
[clickhouse01] 2023.08.04 16:33:33.591599 [ 19937 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part 733441730214180928_797_797_0 with 1 steps
[clickhouse01] 2023.08.04 16:33:33.591604 [ 19902 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part 733443096475154519_800_800_0 with 25 steps
[clickhouse01] 2023.08.04 16:33:33.591605 [ 19829 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part 733443096475154519_801_801_0 with 4 steps
[clickhouse01] 2023.08.04 16:33:33.591614 [ 17196 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part 733443074421504077_799_799_0 with 29 steps
[clickhouse01] 2023.08.04 16:33:33.594209 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Key condi
tion: (column 0 in 4-element set)
[clickhouse01] 2023.08.04 16:33:33.601347 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): MinMax in
dex condition: (column 0 in 4-element set)
[clickhouse01] 2023.08.04 16:33:33.601557 [ 19957 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part p1.733441730214180928_797_797_0 with 1 steps
[clickhouse01] 2023.08.04 16:33:33.601559 [ 8922 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gener
ic exclusion search over index for part p1.733442399864193058_797_797_0 with 1 steps
[clickhouse01] 2023.08.04 16:33:33.601614 [ 19899 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part p1.733443074421504077_799_799_0 with 1 steps
[clickhouse01] 2023.08.04 16:33:33.601610 [ 19891 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part p1.733443096475154519_800_800_0 with 1 steps
[clickhouse01] 2023.08.04 16:33:33.601607 [ 16830 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Used gene
ric exclusion search over index for part p1.733443096475154519_801_801_0 with 1 steps
[clickhouse01] 2023.08.04 16:33:33.603127 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Key condi
tion: unknown
[clickhouse01] 2023.08.04 16:33:33.611194 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): MinMax in
dex condition: (column 0 in 4-element set)
[clickhouse01] 2023.08.04 16:33:33.611589 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Selected
5/4266 parts by partition key, 5 parts by primary key, 49/49 marks by primary key, 49 marks to read from 5 ranges
[clickhouse01] 2023.08.04 16:33:33.611607 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Spreading
 mark ranges among streams (default reading)
[clickhouse01] 2023.08.04 16:33:33.612094 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> MergeTreeReadPool: min_marks_for_concurrent_read=24
[clickhouse01] 2023.08.04 16:33:33.612170 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Reading a
pprox. 383950 rows with 5 streams
[clickhouse01] 2023.08.04 16:33:33.613055 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Selected
5/4266 parts by partition key, 5 parts by primary key, 5/5 marks by primary key, 5 marks to read from 5 ranges
[clickhouse01] 2023.08.04 16:33:33.613075 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Spreading
 mark ranges among streams (default reading)
[clickhouse01] 2023.08.04 16:33:33.613199 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> MergeTreeReadPool: min_marks_for_concurrent_read=24
[clickhouse01] 2023.08.04 16:33:33.613219 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (8fba6f47-99e2-4250-8f11-f564bfa81ae3) (SelectExecutor): Reading a
pprox. 57 rows with 5 streams
[clickhouse01] 2023.08.04 16:33:33.613890 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> JoiningTransform: Before join block: 'camp_date_id Int64 Int64(size = 0), class_id Nullable(Int64) Nullab
le(size = 0, Int64(size = 0), UInt8(size = 0)), camp_date_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), class_name Nullable(String) Nullable(size = 0, String(size = 0), UInt
8(size = 0)), course_id Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_start_time Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), order_number Nullable(In
t32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_channel Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_type Nullable(Int32) Nullable(size = 0, Int32(size = 0),
UInt8(size = 0)), flow_source Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), channel_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), sub_channel_name Nu
llable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), apply_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), join_count Nullable(Int64) Nullable(size = 0, Int64(
size = 0), UInt8(size = 0)), recall_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_conver
t_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), effective_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), thirty_count Nullable(Int64) Nullable(si
ze = 0, Int64(size = 0), UInt8(size = 0)), sixty_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), two_hour_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size =
 0)), base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high
_order_amount Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_amount Nulla
ble(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), unpaid_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), new_unpaid_count Nullable(Int64) Nullable(size = 0
, Int64(size = 0), UInt8(size = 0)), created_time DateTime UInt32(size = 0), refund_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), refund_high_order_amount Nullable
(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), version Int64 Int64(size = 0), popup_base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0))'
[clickhouse01] 2023.08.04 16:33:33.614014 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> JoiningTransform: After join block: 'camp_date_id Int64 Int64(size = 0), class_id Nullable(Int64) Nullabl
e(size = 0, Int64(size = 0), UInt8(size = 0)), camp_date_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), class_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8
(size = 0)), course_id Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_start_time Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), order_number Nullable(Int
32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_channel Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_type Nullable(Int32) Nullable(size = 0, Int32(size = 0), U
Int8(size = 0)), flow_source Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), channel_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), sub_channel_name Nul
lable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), apply_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), join_count Nullable(Int64) Nullable(size = 0, Int64(s
ize = 0), UInt8(size = 0)), recall_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_convert
_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), effective_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), thirty_count Nullable(Int64) Nullable(siz
e = 0, Int64(size = 0), UInt8(size = 0)), sixty_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), two_hour_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size =
0)), base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_
order_amount Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_amount Nullab
le(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), unpaid_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), new_unpaid_count Nullable(Int64) Nullable(size = 0,
 Int64(size = 0), UInt8(size = 0)), created_time DateTime UInt32(size = 0), refund_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), refund_high_order_amount Nullable(
Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), version Int64 Int64(size = 0), popup_base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0))'
[clickhouse01] 2023.08.04 16:33:33.622561 [ 9181 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.622581 [ 19823 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.622589 [ 9181 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse01] 2023.08.04 16:33:33.622591 [ 17125 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.622608 [ 19823 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse01] 2023.08.04 16:33:33.622618 [ 17125 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse01] 2023.08.04 16:33:33.622612 [ 19886 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.622635 [ 9152 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.622641 [ 19886 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse01] 2023.08.04 16:33:33.622666 [ 9181 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 13 to 13 rows (from 455.00 B) in 0.009151646 sec. (1420.509 rows/sec., 4
8.55 KiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.622665 [ 17125 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 2 to 2 rows (from 70.00 B) in 0.009144052 sec. (218.721 rows/sec., 7.48
 KiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.622670 [ 19823 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 14 to 14 rows (from 490.00 B) in 0.009156581 sec. (1528.955 rows/sec.,
52.26 KiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.622669 [ 9152 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse01] 2023.08.04 16:33:33.622683 [ 19886 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 14 to 14 rows (from 490.00 B) in 0.009196997 sec. (1522.236 rows/sec.,
52.03 KiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.622705 [ 9152 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 14 to 14 rows (from 490.00 B) in 0.009158329 sec. (1528.663 rows/sec., 5
2.25 KiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.622716 [ 9152 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Merging aggregated data
[clickhouse01] 2023.08.04 16:33:33.628082 [ 19823 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.628113 [ 19823 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=623, median_size=98
[clickhouse01] 2023.08.04 16:33:33.628128 [ 19823 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse01] 2023.08.04 16:33:33.634869 [ 17125 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.634915 [ 17125 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=623, median_size=98
[clickhouse01] 2023.08.04 16:33:33.634937 [ 17125 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse01] 2023.08.04 16:33:33.635225 [ 9181 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.635251 [ 9181 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=623, median_size=98
[clickhouse01] 2023.08.04 16:33:33.635266 [ 9181 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse01] 2023.08.04 16:33:33.635602 [ 19886 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.635636 [ 19886 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=623, median_size=98
[clickhouse01] 2023.08.04 16:33:33.635658 [ 19886 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse01] 2023.08.04 16:33:33.636184 [ 9152 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregating
[clickhouse01] 2023.08.04 16:33:33.636206 [ 9152 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=623, median_size=98
[clickhouse01] 2023.08.04 16:33:33.636224 [ 9152 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse01] 2023.08.04 16:33:33.637107 [ 9152 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 8192 to 12 rows (from 2.70 MiB) in 0.021741704 sec. (376787.394 rows/sec
., 124.25 MiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.643913 [ 19823 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 30700 to 51 rows (from 10.05 MiB) in 0.028434479 sec. (1079675.137 rows
/sec., 353.39 MiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.652222 [ 17125 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 34818 to 87 rows (from 10.99 MiB) in 0.036693236 sec. (948894.232 rows/
sec., 299.52 MiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.713497 [ 9181 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 139264 to 218 rows (from 45.83 MiB) in 0.097879127 sec. (1422816.123 row
s/sec., 468.21 MiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.729415 [ 19886 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> AggregatingTransform: Aggregated. 170976 to 254 rows (from 56.38 MiB) in 0.113991293 sec. (1499904.032 ro
ws/sec., 494.56 MiB/sec.)
[clickhouse01] 2023.08.04 16:33:33.729431 [ 19886 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> Aggregator: Merging aggregated data
[clickhouse01] 2023.08.04 16:33:33.730834 [ 19894 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> MergingSortedTransform: Merge sorted 2 blocks, 616 rows in 0.112002137 sec., 5499.895060038006 rows/sec.,
 2.03 MiB/sec
[clickhouse01] 2023.08.04 16:33:33.793053 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> executeQuery: Read 384007 rows, 128.88 MiB in 0.667191 sec., 575557.8237716036 rows/sec., 193.16 MiB/sec.
[clickhouse01] 2023.08.04 16:33:33.795411 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Trace> HashJoin: (0x7f3d47dfe098) Join data is being destroyed, 740535 bytes and 56 rows in hash table
[clickhouse01] 2023.08.04 16:33:33.799457 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> MemoryTracker: Peak memory usage (for query): 404.98 MiB.
[clickhouse01] 2023.08.04 16:33:33.799487 [ 28705 ] {ff467752-2c43-4c10-8bca-41dd788fb5f4} <Debug> TCPHandler: Processed in 0.674116979 sec.

@yangshike
Copy link
Author

yangshike commented Aug 4, 2023 •
edited by novikd

This is trace_log on 23.7

[clickhouse02] 2023.08.04 16:37:17.147251 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> executeQuery: (from 127.0.0.1:36882)  SELECT live_start_time, camp_date_id, course_id , class_id , min(`c
amp_date_name`) camp_date_name, min(`class_name`) class_name, min(`order_number`) `order_number`, min(`flow_channel`) `flow_channel`, min(`flow_type`) `flow_type`, min(`flow_source`) `flow_source`, min(`c
hannel_name`)`channel_name` , min(`sub_channel_name`) `sub_channel_name`, sum(apply_count) apply_count, sum(join_count) join_count, sum(recall_count) recall_count, sum(come_count) come_count, sum(come_con
vert_count) come_convert_count, sum(effective_count) effective_count, sum(thirty_count) thirty_count, sum(sixty_count) sixty_count, sum(two_hour_count) two_hour_count, sum(base_live_high_order_count) base
_live_high_order_count, sum(popup_base_live_high_order_count) popup_base_live_high_order_count, sum(live_high_order_count) live_high_order_count, sum(live_high_order_amount) live_high_order_amount, sum(sa
le_high_order_count) sale_high_order_count, sum(sale_high_order_amount) sale_high_order_amount, sum(refund_high_order_count) refund_high_order_count, sum(refund_high_order_amount) refund_high_order_amount
, sum(unpaid_order_count) unpaid_order_count, ifnull(sum(new_unpaid_count),0) new_unpaid_count, min(`created_time`) `created_time` FROM t_bi_live_watcher_stat right join ( select camp_date_id camp_date_id
_version , live_start_time live_start_time_version, max(version) max_version from t_bi_live_watcher_stat where camp_date_id in (733443096475154519,733443074421504077,733442399864193058,733441730214180928
) group by camp_date_id, live_start_time ) t_version_one on t_bi_live_watcher_stat.camp_date_id = t_version_one.camp_date_id_version and t_bi_live_watcher_stat.live_start_time = t_version_one.live_start_t
ime_version and t_bi_live_watcher_stat.version = t_version_one.max_version where camp_date_id in (733443096475154519,733443074421504077,733442399864193058,733441730214180928 )group by live_start_time, cam
p_date_id, course_id , class_id order by order_number asc, live_start_time asc; (stage: Complete)
[clickhouse02] 2023.08.04 16:37:17.149239 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 24819212636997
6320_2600_2624_9_2625 (part data version 2625, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.149255 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 24875478390071
7056_0_0_0_2429 (part data version 2429, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.149271 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 24875881904346
3168_0_0_0_2429 (part data version 2429, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.149292 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 24922168918309
6832_0_0_0_2429 (part data version 2429, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.347938 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57938853161526
9102_607_620_2 (part data version 607, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.347973 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57938926349611
4355_611_624_2 (part data version 611, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.347999 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57938999479818
2621_608_621_2 (part data version 608, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.348038 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57939062440938
3262_602_612_2 (part data version 602, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.348077 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 57939062440938
3262_613_615_1 (part data version 613, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.348115 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (ReplicatedMergeTreeQueue): Got 0 commands for part 58084663408072
7850_116_118_1 (part data version 116, part metadata version 0)
[clickhouse02] 2023.08.04 16:37:17.349611 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "camp_date_id IN (733443096475154519, 73344307
4421504077, 733442399864193058, 733441730214180928)" moved to PREWHERE
[clickhouse02] 2023.08.04 16:37:17.349905 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> ContextAccess (default): Access granted: SELECT(camp_date_id, live_start_time, version) ON click_kuaicaib
i.t_bi_live_watcher_stat
[clickhouse02] 2023.08.04 16:37:17.350970 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
[clickhouse02] 2023.08.04 16:37:17.351111 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> TableJoin: Left JOIN converting actions: empty
[clickhouse02] 2023.08.04 16:37:17.351147 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> TableJoin: Right JOIN converting actions: empty
[clickhouse02] 2023.08.04 16:37:17.351204 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> HashJoin: (0x7f6354bf8398) Datatype: EMPTY, kind: Right, strictness: All, right header: camp_date_id_vers
ion Int64 Int64(size = 0), live_start_time_version Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), max_version Int64 Int64(size = 0)
[clickhouse02] 2023.08.04 16:37:17.351243 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> HashJoin: (0x7f6354bf8398) Keys: [(camp_date_id, live_start_time, version) = (camp_date_id_version, live_
start_time_version, max_version)]
[clickhouse02] 2023.08.04 16:37:17.352253 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> ContextAccess (default): Access granted: SELECT(camp_date_id, class_id, camp_date_name, class_name, cours
e_id, live_start_time, order_number, flow_channel, flow_type, flow_source, channel_name, sub_channel_name, apply_count, join_count, recall_count, come_count, come_convert_count, effective_count, thirty_co
unt, sixty_count, two_hour_count, base_live_high_order_count, live_high_order_count, live_high_order_amount, sale_high_order_count, sale_high_order_amount, unpaid_order_count, new_unpaid_count, created_ti
me, refund_high_order_count, refund_high_order_amount, version, popup_base_live_high_order_count) ON click_kuaicaibi.t_bi_live_watcher_stat
[clickhouse02] 2023.08.04 16:37:17.353259 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
[clickhouse02] 2023.08.04 16:37:17.353460 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> JoiningTransform: Before join block: 'camp_date_id Int64 Int64(size = 0), class_id Nullable(Int64) Nullab
le(size = 0, Int64(size = 0), UInt8(size = 0)), camp_date_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), class_name Nullable(String) Nullable(size = 0, String(size = 0), UInt
8(size = 0)), course_id Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_start_time Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), order_number Nullable(In
t32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_channel Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_type Nullable(Int32) Nullable(size = 0, Int32(size = 0),
UInt8(size = 0)), flow_source Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), channel_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), sub_channel_name Nu
llable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), apply_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), join_count Nullable(Int64) Nullable(size = 0, Int64(
size = 0), UInt8(size = 0)), recall_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_conver
t_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), effective_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), thirty_count Nullable(Int64) Nullable(si
ze = 0, Int64(size = 0), UInt8(size = 0)), sixty_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), two_hour_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size =
 0)), base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high
_order_amount Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_amount Nulla
ble(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), unpaid_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), new_unpaid_count Nullable(Int64) Nullable(size = 0
, Int64(size = 0), UInt8(size = 0)), created_time DateTime UInt32(size = 0), refund_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), refund_high_order_amount Nullable
(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), version Int64 Int64(size = 0), popup_base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0))'
[clickhouse02] 2023.08.04 16:37:17.353556 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> JoiningTransform: After join block: 'camp_date_id Int64 Int64(size = 0), class_id Nullable(Int64) Nullabl
e(size = 0, Int64(size = 0), UInt8(size = 0)), camp_date_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), class_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8
(size = 0)), course_id Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_start_time Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), order_number Nullable(Int
32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_channel Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_type Nullable(Int32) Nullable(size = 0, Int32(size = 0), U
Int8(size = 0)), flow_source Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), channel_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), sub_channel_name Nul
lable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), apply_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), join_count Nullable(Int64) Nullable(size = 0, Int64(s
ize = 0), UInt8(size = 0)), recall_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_convert
_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), effective_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), thirty_count Nullable(Int64) Nullable(siz
e = 0, Int64(size = 0), UInt8(size = 0)), sixty_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), two_hour_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size =
0)), base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_
order_amount Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_amount Nullab
le(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), unpaid_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), new_unpaid_count Nullable(Int64) Nullable(size = 0,
 Int64(size = 0), UInt8(size = 0)), created_time DateTime UInt32(size = 0), refund_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), refund_high_order_amount Nullable(
Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), version Int64 Int64(size = 0), popup_base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0))'
[clickhouse02] 2023.08.04 16:37:17.356725 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Key condi
tion: (column 0 in 4-element set)
[clickhouse02] 2023.08.04 16:37:17.363167 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): MinMax in
dex condition: (column 0 in 4-element set)
[clickhouse02] 2023.08.04 16:37:17.363329 [ 4250 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part 733442399864193058_797_797_0 with 3 steps
[clickhouse02] 2023.08.04 16:37:17.363367 [ 6630 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part 733441730214180928_797_797_0 with 6 steps
[clickhouse02] 2023.08.04 16:37:17.363387 [ 5739 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part 733443096475154519_801_801_0 with 4 steps
[clickhouse02] 2023.08.04 16:37:17.363414 [ 4359 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part 733443096475154519_800_800_0 with 25 steps
[clickhouse02] 2023.08.04 16:37:17.363432 [ 6583 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part 733443074421504077_799_799_0 with 29 steps
[clickhouse02] 2023.08.04 16:37:17.365994 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Key condi
tion: (column 0 in 4-element set)
[clickhouse02] 2023.08.04 16:37:17.367916 [ 6568 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.248192126369976320_2600_2624_9_2625 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.367936 [ 3387 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.249221689183096832_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.367919 [ 7853 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.248754783900717056_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.367998 [ 7853 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.248758819043463168_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.368009 [ 6763 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.251799997439594496_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.368036 [ 7853 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.259730226321608704_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.368064 [ 7853 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.262709070196490240_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.368042 [ 6582 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.256861626011275264_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.368063 [ 7769 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.261531831996043264_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.368078 [ 3415 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.258898846767435776_0_0_0_2429 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419369 [ 3387 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738074651055079569_138_142_2 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419369 [ 6703 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738074677638379546_143_143_0 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419551 [ 6763 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518231238385924_30_30_0 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419577 [ 7853 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518284157882511_0_18_9 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419610 [ 3495 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518284157882511_19_29_5 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419594 [ 4532 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518323353680056_28_28_0 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419611 [ 6582 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518323353680056_30_30_0 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419633 [ 6703 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518284157882511_30_30_0 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419653 [ 6823 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518323353680056_29_29_0 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419642 [ 3494 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518323353680056_27_27_0 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419676 [ 3387 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518231238385924_29_29_0 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.419652 [ 7769 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Used gener
ic exclusion search over index for part p1.738518323353680056_0_26_3 with 1 steps
[clickhouse02] 2023.08.04 16:37:17.422635 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Key condi
tion: unknown
[clickhouse02] 2023.08.04 16:37:17.429751 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): MinMax in
dex condition: unknown
[clickhouse02] 2023.08.04 16:37:17.485364 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Selected
4272/4272 parts by partition key, 4272 parts by primary key, 69891/69891 marks by primary key, 69891 marks to read from 4272 ranges
[clickhouse02] 2023.08.04 16:37:17.489677 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Spreading
 mark ranges among streams (default reading)
[clickhouse02] 2023.08.04 16:37:17.753492 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> MergeTreeReadPool: min_marks_for_concurrent_read=24
[clickhouse02] 2023.08.04 16:37:17.755835 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Reading a
pprox. 527871692 rows with 16 streams
[clickhouse02] 2023.08.04 16:37:17.759270 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Selected
4272/4272 parts by partition key, 5 parts by primary key, 5/4272 marks by primary key, 5 marks to read from 5 ranges
[clickhouse02] 2023.08.04 16:37:17.759298 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Spreading
 mark ranges among streams (default reading)
[clickhouse02] 2023.08.04 16:37:17.759429 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> MergeTreeReadPool: min_marks_for_concurrent_read=24
[clickhouse02] 2023.08.04 16:37:17.759447 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Reading a
pprox. 57 rows with 5 streams
[clickhouse02] 2023.08.04 16:37:17.760050 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> JoiningTransform: Before join block: 'camp_date_id Int64 Int64(size = 0), class_id Nullable(Int64) Nullab
le(size = 0, Int64(size = 0), UInt8(size = 0)), camp_date_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), class_name Nullable(String) Nullable(size = 0, String(size = 0), UInt
8(size = 0)), course_id Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_start_time Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), order_number Nullable(In
t32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_channel Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_type Nullable(Int32) Nullable(size = 0, Int32(size = 0),
UInt8(size = 0)), flow_source Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), channel_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), sub_channel_name Nu
llable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), apply_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), join_count Nullable(Int64) Nullable(size = 0, Int64(
size = 0), UInt8(size = 0)), recall_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_conver
t_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), effective_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), thirty_count Nullable(Int64) Nullable(si
ze = 0, Int64(size = 0), UInt8(size = 0)), sixty_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), two_hour_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size =
 0)), base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high
_order_amount Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_amount Nulla
ble(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), unpaid_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), new_unpaid_count Nullable(Int64) Nullable(size = 0
, Int64(size = 0), UInt8(size = 0)), created_time DateTime UInt32(size = 0), refund_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), refund_high_order_amount Nullable
(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), version Int64 Int64(size = 0), popup_base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0))'
[clickhouse02] 2023.08.04 16:37:17.760123 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> JoiningTransform: After join block: 'camp_date_id Int64 Int64(size = 0), class_id Nullable(Int64) Nullabl
e(size = 0, Int64(size = 0), UInt8(size = 0)), camp_date_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), class_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8
(size = 0)), course_id Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_start_time Nullable(Date) Nullable(size = 0, UInt16(size = 0), UInt8(size = 0)), order_number Nullable(Int
32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_channel Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), flow_type Nullable(Int32) Nullable(size = 0, Int32(size = 0), U
Int8(size = 0)), flow_source Nullable(Int32) Nullable(size = 0, Int32(size = 0), UInt8(size = 0)), channel_name Nullable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), sub_channel_name Nul
lable(String) Nullable(size = 0, String(size = 0), UInt8(size = 0)), apply_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), join_count Nullable(Int64) Nullable(size = 0, Int64(s
ize = 0), UInt8(size = 0)), recall_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), come_convert
_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), effective_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), thirty_count Nullable(Int64) Nullable(siz
e = 0, Int64(size = 0), UInt8(size = 0)), sixty_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), two_hour_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size =
0)), base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), live_high_
order_amount Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), sale_high_order_amount Nullab
le(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), unpaid_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), new_unpaid_count Nullable(Int64) Nullable(size = 0,
 Int64(size = 0), UInt8(size = 0)), created_time DateTime UInt32(size = 0), refund_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), refund_high_order_amount Nullable(
Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)), version Int64 Int64(size = 0), popup_base_live_high_order_count Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0))'
[clickhouse02] 2023.08.04 16:37:17.773066 [ 2978 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:17.773076 [ 3035 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:17.773087 [ 2978 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse02] 2023.08.04 16:37:17.773098 [ 3035 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse02] 2023.08.04 16:37:17.773122 [ 5710 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:17.773145 [ 3035 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 14 to 14 rows (from 490.00 B) in 0.013386111 sec. (1045.860 rows/sec., 3
5.75 KiB/sec.)
[clickhouse02] 2023.08.04 16:37:17.773141 [ 2978 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 2 to 2 rows (from 70.00 B) in 0.013388412 sec. (149.383 rows/sec., 5.11
KiB/sec.)
[clickhouse02] 2023.08.04 16:37:17.773144 [ 5710 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse02] 2023.08.04 16:37:17.773178 [ 5710 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 13 to 13 rows (from 455.00 B) in 0.013410264 sec. (969.407 rows/sec., 33
.13 KiB/sec.)
[clickhouse02] 2023.08.04 16:37:17.773202 [ 5879 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:17.773223 [ 5879 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse02] 2023.08.04 16:37:17.773261 [ 5879 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 14 to 14 rows (from 490.00 B) in 0.013478731 sec. (1038.673 rows/sec., 3
5.50 KiB/sec.)
[clickhouse02] 2023.08.04 16:37:17.773394 [ 6711 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:17.773417 [ 6711 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys128
[clickhouse02] 2023.08.04 16:37:17.773469 [ 6711 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 14 to 14 rows (from 490.00 B) in 0.013690962 sec. (1022.572 rows/sec., 3
4.95 KiB/sec.)
[clickhouse02] 2023.08.04 16:37:17.773486 [ 6711 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Merging aggregated data
[clickhouse02] 2023.08.04 16:37:36.871222 [ 6711 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:36.871275 [ 6711 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=629, median_size=37
[clickhouse02] 2023.08.04 16:37:36.871302 [ 6711 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse02] 2023.08.04 16:37:36.964058 [ 5710 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:36.964116 [ 5710 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=629, median_size=37
[clickhouse02] 2023.08.04 16:37:36.964148 [ 5710 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse02] 2023.08.04 16:37:37.616429 [ 6847 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:37.616481 [ 6847 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=629, median_size=37
[clickhouse02] 2023.08.04 16:37:37.616501 [ 6847 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse02] 2023.08.04 16:37:37.622105 [ 4184 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:37.622160 [ 4184 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=629, median_size=37
[clickhouse02] 2023.08.04 16:37:37.622187 [ 4184 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse02] 2023.08.04 16:37:37.682219 [ 3035 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:37.682276 [ 3035 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=629, median_size=37
[clickhouse02] 2023.08.04 16:37:37.682298 [ 3035 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse02] 2023.08.04 16:37:37.690007 [ 2978 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.926566845 sec. (0.000 rows/sec., 0.00 B/
sec.)
[clickhouse02] 2023.08.04 16:37:37.691781 [ 4081 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.928311312 sec. (0.000 rows/sec., 0.00 B/
sec.)
[clickhouse02] 2023.08.04 16:37:37.693018 [ 4184 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 73728 to 123 rows (from 24.17 MiB) in 19.929391417 sec. (3699.461 rows/s
ec., 1.21 MiB/sec.)
[clickhouse02] 2023.08.04 16:37:37.695200 [ 3473 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.931327756 sec. (0.000 rows/sec., 0.00 B/
sec.)
[clickhouse02] 2023.08.04 16:37:37.697772 [ 6711 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 31404 to 58 rows (from 10.20 MiB) in 19.934524404 sec. (1575.357 rows/se
c., 523.94 KiB/sec.)
[clickhouse02] 2023.08.04 16:37:37.703219 [ 3010 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.939636475 sec. (0.000 rows/sec., 0.00 B/
sec.)
[clickhouse02] 2023.08.04 16:37:37.704903 [ 3975 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:37.704951 [ 3975 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=629, median_size=37
[clickhouse02] 2023.08.04 16:37:37.704971 [ 3975 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse02] 2023.08.04 16:37:37.707790 [ 6847 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 133122 to 201 rows (from 43.90 MiB) in 19.943861515 sec. (6674.836 rows/
sec., 2.20 MiB/sec.)
[clickhouse02] 2023.08.04 16:37:37.712668 [ 6775 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.94898766 sec. (0.000 rows/sec., 0.00 B/s
ec.)
[clickhouse02] 2023.08.04 16:37:37.721838 [ 3035 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 65536 to 110 rows (from 21.46 MiB) in 19.958441061 sec. (3283.623 rows/s
ec., 1.08 MiB/sec.)
[clickhouse02] 2023.08.04 16:37:37.727530 [ 6547 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregating
[clickhouse02] 2023.08.04 16:37:37.727587 [ 6547 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: An entry for key=3946498890820961257 found in cache: sum_of_sizes=629, median_size=37
[clickhouse02] 2023.08.04 16:37:37.727622 [ 6547 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Aggregation method: nullable_keys256
[clickhouse02] 2023.08.04 16:37:37.730563 [ 7815 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.967041731 sec. (0.000 rows/sec., 0.00 B/
sec.)
[clickhouse02] 2023.08.04 16:37:37.730918 [ 3975 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 49152 to 86 rows (from 16.05 MiB) in 19.967149296 sec. (2461.643 rows/se
c., 823.27 KiB/sec.)
[clickhouse02] 2023.08.04 16:37:37.731827 [ 5879 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.968539682 sec. (0.000 rows/sec., 0.00 B/
sec.)
[clickhouse02] 2023.08.04 16:37:37.734508 [ 6547 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 15328 to 25 rows (from 5.06 MiB) in 19.970547516 sec. (767.530 rows/sec.
, 259.43 KiB/sec.)
[clickhouse02] 2023.08.04 16:37:37.739552 [ 5710 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 15680 to 28 rows (from 5.10 MiB) in 19.976216553 sec. (784.933 rows/sec.
, 261.45 KiB/sec.)
[clickhouse02] 2023.08.04 16:37:37.739868 [ 3607 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.976153169 sec. (0.000 rows/sec., 0.00 B/
sec.)
[clickhouse02] 2023.08.04 16:37:37.741827 [ 4323 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 19.978013052 sec. (0.000 rows/sec., 0.00 B/
sec.)
[clickhouse02] 2023.08.04 16:37:37.741844 [ 4323 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Merging aggregated data
[clickhouse02] 2023.08.04 16:37:37.741866 [ 4323 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> Aggregator: Statistics updated for key=3946498890820961257: new sum_of_sizes=631, median_size=0
[clickhouse02] 2023.08.04 16:37:37.743043 [ 4081 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> MergingSortedTransform: Merge sorted 2 blocks, 616 rows in 19.97522896 sec., 30.838194707731652 rows/sec.,
 11.64 KiB/sec
[clickhouse02] 2023.08.04 16:37:37.745343 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> executeQuery: Read 518898683 rows, 165.30 GiB in 20.59756 sec., 25192240.391580362 rows/sec., 8.03 GiB/se
c.
[clickhouse02] 2023.08.04 16:37:37.749753 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Trace> HashJoin: (0x7f6354bf8398) Join data is being destroyed, 21175 bytes and 56 rows in hash table
[clickhouse02] 2023.08.04 16:37:37.780176 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> MemoryTracker: Peak memory usage (for query): 534.10 MiB.
[clickhouse02] 2023.08.04 16:37:37.780193 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} <Debug> TCPHandler: Processed in 20.633745173 sec.

@yangshike
Copy link
Author

on 23.7 ，like not found key， run Full Table Scan
click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): Key condi
tion: unknown
[clickhouse02] 2023.08.04 16:37:17.429751 [ 24152 ] {d50cc16b-83e5-424b-9ae8-922e0118e494} click_kuaicaibi.t_bi_live_watcher_stat (0e902555-603b-4302-b2e6-fa72b38c4429) (SelectExecutor): MinMax in
dex condition: unknown

@yangshike
Copy link
Author

tihis is my table engine :
ENGINE = ReplicatedMergeTree()
PARTITION BY camp_date_id
ORDER BY camp_date_id

@davenger
Copy link
Member

davenger commented Aug 4, 2023

@yangshike could you please provide table schema (SHOW CREATE TABLE click_kuaicaibi.t_bi_live_watcher_stat ) and EXPLAIN indexes = 1 SELECT live_start_time ... for the query on both 23.5 and 23.7?

@alexey-milovidov alexey-milovidov added st-need-info We need extra data to continue (waiting for response) performance and removed potential bug To be reviewed by developers and confirmed/rejected. labels Aug 4, 2023
@yangshike
Copy link
Author

yangshike commented Aug 4, 2023 •
edited by novikd 

CREATE TABLE click_kuaicaibi.t_bi_live_watcher_stat
(
    `id` Int64,
    `camp_date_id` Int64,
    `class_id` Nullable(Int64),
    `camp_date_name` Nullable(String),
    `class_name` Nullable(String),
    `add_status` Nullable(Int32),
    `course_id` Nullable(Int64),
    `come_from` Nullable(Int32),
    `live_start_time` Nullable(Date),
    `order_number` Nullable(Int32),
    `flow_channel` Nullable(Int32),
    `flow_type` Nullable(Int32),
    `flow_source` Nullable(Int32),
    `channel` Nullable(Int64),
    `sub_channel` Nullable(Int64),
    `agent_merchant` Nullable(String),
    `channel_name` Nullable(String),
    `sub_channel_name` Nullable(String),
    `apply_count` Nullable(Int64),
    `join_count` Nullable(Int64),
    `recall_count` Nullable(Int64) COMMENT '召回人数',
    `come_count` Nullable(Int64),
    `come_convert_count` Nullable(Int64) DEFAULT 0,
    `effective_count` Nullable(Int64),
    `thirty_count` Nullable(Int64),
    `sixty_count` Nullable(Int64),
    `two_hour_count` Nullable(Int64),
    `base_live_high_order_count` Nullable(Int64),
    `live_high_order_count` Nullable(Int64),
    `live_high_order_amount` Nullable(Int64),
    `sale_high_order_count` Nullable(Int64),
    `sale_high_order_amount` Nullable(Int64),
    `unpaid_order_count` Nullable(Int64),
    `new_unpaid_count` Nullable(Int64),
    `created_time` DateTime,
    `modified_time` Nullable(DateTime),
    `base_refund_high_order_count` Nullable(Int64),
    `refund_live_high_order_count` Int64 DEFAULT 0 ,
    `refund_live_high_order_amount` Int64 DEFAULT 0,
    `refund_sale_high_order_count` Int64 DEFAULT 0 ,
    `refund_sale_high_order_amount` Int64 DEFAULT 0,
    `high_order_count` Int64 DEFAULT 0  ,
    `high_order_amount` Int64 DEFAULT 0,
    `refund_high_order_count` Nullable(Int64),
    `refund_high_order_amount` Nullable(Int64),
    `version` Int64 DEFAULT 1,
    `popup_base_live_high_order_count` Nullable(Int64) DEFAULT 0,
    `popup_base_refund_high_order_count` Nullable(Int64) DEFAULT 0 ,
    PROJECTION p1
    (
        SELECT
            live_start_time,
            camp_date_id,
            course_id,
            min(camp_date_name) AS camp_date_name,
            min(class_name) AS class_name,
            min(order_number) AS order_number,
            min(flow_channel) AS flow_channel,
            min(flow_type) AS flow_type,
            min(flow_source) AS flow_source,
            min(channel_name) AS channel_name,
            min(sub_channel_name) AS sub_channel_name,
            sum(apply_count) AS apply_count,
            sum(join_count) AS join_count,
            sum(come_count) AS come_count,
            sum(come_convert_count) AS come_convert_count,
            sum(effective_count) AS effective_count,
            sum(thirty_count) AS thirty_count,
            sum(sixty_count) AS sixty_count,
            sum(two_hour_count) AS two_hour_count,
            sum(base_live_high_order_count) AS base_live_high_order_count,
            sum(live_high_order_count) AS live_high_order_count,
            sum(live_high_order_amount) AS live_high_order_amount,
            sum(sale_high_order_count) AS sale_high_order_count,
            sum(sale_high_order_amount) AS sale_high_order_amount,
            sum(refund_high_order_count) AS refund_high_order_count,
            sum(refund_high_order_amount) AS refund_high_order_amount,
            sum(unpaid_order_count) AS unpaid_order_count,
            min(created_time) AS created_time,
            max(version) AS max_version
        GROUP BY
            camp_date_id,
            live_start_time,
            course_id
    )
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{layer}-{shard}/click_kuaicaibi/t_bi_live_watcher_stat2', '{replica}')
PARTITION BY camp_date_id
ORDER BY camp_date_id
SETTINGS storage_policy = 'all_jbod', index_granularity = 8192

@yangshike
Copy link
Author

yangshike commented Aug 4, 2023 •
edited by novikd

version 23.7.2

explain
String

Expression ((Projection + Before ORDER BY [lifted up part]))
  Sorting (Sorting for ORDER BY)
    Expression (Before ORDER BY)
      Aggregating
        Expression (Before GROUP BY)
          Filter (WHERE)
            Join (JOIN FillRightFirst)
              Expression (Before JOIN)
                ReadFromMergeTree (click_kuaicaibi.t_bi_live_watcher_stat)
                Indexes:
                  MinMax
                    Condition: true
                    Parts: 4251/4251
                    Granules: 70263/70263
                  Partition
                    Condition: true
                    Parts: 4251/4251
                    Granules: 70263/70263
                  PrimaryKey
                    Condition: true
                    Parts: 4251/4251
                    Granules: 70263/70263
              Expression ((Joined actions + (Rename joined columns + (Projection + Before ORDER BY))))
                Aggregating
                  Filter
                    ReadFromMergeTree (p1)
                    Indexes:
                      PrimaryKey
                        Keys:
                          camp_date_id
                        Condition: (camp_date_id in 4-element set)
                        Parts: 13/4251
                        Granules: 13/70263

33 rows in set. Elapsed: 0.139 sec.

@yangshike
Copy link
Author

yangshike commented Aug 4, 2023 •
edited by novikd

version 23.5.2
explain
String

Expression ((Projection + Before ORDER BY [lifted up part]))
  Sorting (Sorting for ORDER BY)
    Expression (Before ORDER BY)
      Aggregating
        Expression (Before GROUP BY)
          Filter (WHERE)
            Join (JOIN FillRightFirst)
              Expression (Before JOIN)
                ReadFromMergeTree (click_kuaicaibi.t_bi_live_watcher_stat)
                Indexes:
                  MinMax
                    Keys:
                      camp_date_id
                    Condition: (camp_date_id in 4-element set)
                    Parts: 13/4251
                    Granules: 218/70263
                  Partition
                    Keys:
                      camp_date_id
                    Condition: (camp_date_id in 4-element set)
                    Parts: 13/13
                    Granules: 218/218
                  PrimaryKey
                    Condition: true
                    Parts: 13/13
                    Granules: 218/218
              Expression ((Joined actions + (Rename joined columns + (Projection + Before ORDER BY))))
                Aggregating
                  Filter
                    ReadFromMergeTree (p1)
                    Indexes:
                      MinMax
                        Keys:
                          camp_date_id
                        Condition: (camp_date_id in 4-element set)
                        Parts: 13/4251
                        Granules: 218/70263
                      Partition
                        Keys:
                          camp_date_id
                        Condition: (camp_date_id in 4-element set)
                        Parts: 13/13
                        Granules: 218/218
                      PrimaryKey
                        Keys:
                          camp_date_id
                        Condition: (camp_date_id in 4-element set)
                        Parts: 13/13
                        Granules: 13/218

@davenger
Copy link
Member

davenger commented Aug 4, 2023

@yangshike From explains it looks like this could be #52308
Please try disabling settings query_plan_optimize_primary_key, query_plan_optimize_projection and explaining the query on 23.7

@yangshike
Copy link
Author

yangshike commented Aug 4, 2023 •
edited by novikd 

select * from system.settings where name ='query_plan_optimize_projection';

SELECT *
FROM system.settings
WHERE name = 'query_plan_optimize_projection'

Query id: 813dc010-72fc-4d83-be16-4c3050d5118b

┌─name───────────────────────────┬─value─┬─changed─┬─description──────────────────────────────────────────┬─min──┬─max──┬─readonly─┬─type─┬─default─┬─alias_for─┬─is_obsolete─┐
│ query_plan_optimize_projection │ 0     │       1 │ Use query plan for aggregation-in-order optimisation │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │        0 │ Bool │ 1       │           │           0 │
└────────────────────────────────┴───────┴─────────┴──────────────────────────────────────────────────────┴──────┴──────┴──────────┴──────┴─────────┴───────────┴─────────────┘

1 row in set. Elapsed: 0.003 sec.

clickhouse02 :) select * from system.settings where name ='query_plan_optimize_primary_key';

SELECT *
FROM system.settings
WHERE name = 'query_plan_optimize_primary_key'

Query id: 532db623-8b32-4256-8ac1-2ef7190eb3f4

┌─name────────────────────────────┬─value─┬─changed─┬─description───────────────────────────────────────────┬─min──┬─max──┬─readonly─┬─type─┬─default─┬─alias_for─┬─is_obsolete─┐
│ query_plan_optimize_primary_key │ 0     │       1 │ Analyze primary key using query plan (instead of AST) │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │        0 │ Bool │ 1       │           │           0 │
└─────────────────────────────────┴───────┴─────────┴───────────────────────────────────────────────────────┴──────┴──────┴──────────┴──────┴─────────┴───────────┴─────────────┘

1 row in set. Elapsed: 0.004 sec.

clickhouse02 :) EXPLAIN indexes = 1  SELECT live_start_time, camp_date_id, course_id , min(`camp_date_name`) camp_date_name, min(`class_name`) class_name, min(`order_number`) `order_number`, min(`flow_channel`) `flow_channel`, min(`flow_type`) `flow_type`, min(`flow_source`) `flow_source`, min(`channel_name`)`channel_name` , min(`sub_channel_name`) `sub_channel_name`, sum(apply_count) apply_count, sum(join_count) join_count, sum(recall_count) recall_count, sum(come_count) come_count, sum(come_convert_count) come_convert_count, sum(effective_count) effective_count, sum(thirty_count) thirty_count, sum(sixty_count) sixty_count, sum(two_hour_count) two_hour_count, sum(base_live_high_order_count) base_live_high_order_count, sum(popup_base_live_high_order_count) popup_base_live_high_order_count, sum(live_high_order_count) live_high_order_count, sum(live_high_order_amount) live_high_order_amount, sum(sale_high_order_count) sale_high_order_count, sum(sale_high_order_amount) sale_high_order_amount, sum(refund_high_order_count) refund_high_order_count, sum(refund_high_order_amount) refund_high_order_amount, sum(unpaid_order_count) unpaid_order_count, ifnull(sum(new_unpaid_count),0) new_unpaid_count, min(`created_time`) `created_time` FROM t_bi_live_watcher_stat right join ( select camp_date_id camp_date_id_version , live_start_time live_start_time_version, max(version) max_version from t_bi_live_watcher_stat where camp_date_id in (735200065002975412,734147951933796413,734147924259809357,734147896665489892 ) group by camp_date_id, live_start_time ) t_version_one on t_bi_live_watcher_stat.camp_date_id = t_version_one.camp_date_id_version and t_bi_live_watcher_stat.live_start_time = t_version_one.live_start_time_version and t_bi_live_watcher_stat.version = t_version_one.max_version where camp_date_id in (735200065002975412,734147951933796413,734147924259809357,734147896665489892 )group by live_start_time, camp_date_id, course_id order by order_number asc, live_start_time asc FORMAT TabSeparatedWithNamesAndTypes;

EXPLAIN indexes = 1
SELECT
    live_start_time,
    camp_date_id,
    course_id,
    min(camp_date_name) AS camp_date_name,
    min(class_name) AS class_name,
    min(order_number) AS order_number,
    min(flow_channel) AS flow_channel,
    min(flow_type) AS flow_type,
    min(flow_source) AS flow_source,
    min(channel_name) AS channel_name,
    min(sub_channel_name) AS sub_channel_name,
    sum(apply_count) AS apply_count,
    sum(join_count) AS join_count,
    sum(recall_count) AS recall_count,
    sum(come_count) AS come_count,
    sum(come_convert_count) AS come_convert_count,
    sum(effective_count) AS effective_count,
    sum(thirty_count) AS thirty_count,
    sum(sixty_count) AS sixty_count,
    sum(two_hour_count) AS two_hour_count,
    sum(base_live_high_order_count) AS base_live_high_order_count,
    sum(popup_base_live_high_order_count) AS popup_base_live_high_order_count,
    sum(live_high_order_count) AS live_high_order_count,
    sum(live_high_order_amount) AS live_high_order_amount,
    sum(sale_high_order_count) AS sale_high_order_count,
    sum(sale_high_order_amount) AS sale_high_order_amount,
    sum(refund_high_order_count) AS refund_high_order_count,
    sum(refund_high_order_amount) AS refund_high_order_amount,
    sum(unpaid_order_count) AS unpaid_order_count,
    ifnull(sum(new_unpaid_count), 0) AS new_unpaid_count,
    min(created_time) AS created_time
FROM t_bi_live_watcher_stat
RIGHT JOIN
(
    SELECT
        camp_date_id AS camp_date_id_version,
        live_start_time AS live_start_time_version,
        max(version) AS max_version
    FROM t_bi_live_watcher_stat
    WHERE camp_date_id IN (735200065002975412, 734147951933796413, 734147924259809357, 734147896665489892)
    GROUP BY
        camp_date_id,
        live_start_time
) AS t_version_one ON (t_bi_live_watcher_stat.camp_date_id = t_version_one.camp_date_id_version) AND (t_bi_live_watcher_stat.live_start_time = t_version_one.live_start_time_version) AND (t_bi_live_watcher_stat.version = t_version_one.max_version)
WHERE camp_date_id IN (735200065002975412, 734147951933796413, 734147924259809357, 734147896665489892)
GROUP BY
    live_start_time,
    camp_date_id,
    course_id
ORDER BY
    order_number ASC,
    live_start_time ASC
FORMAT TabSeparatedWithNamesAndTypes

Query id: 5ff99457-9188-45ab-b421-040fa97cdc19

explain
String
Expression ((Projection + Before ORDER BY [lifted up part]))
  Sorting (Sorting for ORDER BY)
    Expression (Before ORDER BY)
      Aggregating
        Expression (Before GROUP BY)
          Filter (WHERE)
            Join (JOIN FillRightFirst)
              Expression (Before JOIN)
                ReadFromMergeTree (click_kuaicaibi.t_bi_live_watcher_stat)
                Indexes:
                  MinMax
                    Condition: true
                    Parts: 4262/4262
                    Granules: 70364/70364
                  Partition
                    Condition: true
                    Parts: 4262/4262
                    Granules: 70364/70364
                  PrimaryKey
                    Keys:
                      camp_date_id
                    Condition: (camp_date_id in 4-element set)
                    Parts: 13/4262
                    Granules: 237/70364
              Expression ((Joined actions + (Rename joined columns + (Projection + Before ORDER BY))))
                ReadFromStorage (MergeTree(with Aggregate projection p1))

@yangshike
Copy link
Author

yangshike commented Aug 4, 2023 •
edited by novikd

After disabling, the query speed is the same as 23.5。
But I see it's also turned on by default on 23.5
on config of 23.5:

ClickHouse client version 23.5.2.7 (official build).
Connecting to database bigdata_knowledge at 127.0.0.1:9000 as user default.
Connected to ClickHouse server version 23.5.2 revision 54462.

Warnings:
 * Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
 * Some obsolete setting is changed. Check 'select * from system.settings where changed' and read the changelog.

clickhouse01 :) select * from system.settings where name ='query_plan_optimize_projection';

SELECT *
FROM system.settings
WHERE name = 'query_plan_optimize_projection'

Query id: f067f014-2d84-4027-9000-5f27a1846115

┌─name───────────────────────────┬─value─┬─changed─┬─description──────────────────────────────────────────┬─min──┬─max──┬─readonly─┬─type─┬─default─┬─alias_for─┐
│ query_plan_optimize_projection │ 1     │       0 │ Use query plan for aggregation-in-order optimisation │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │        0 │ Bool │ 1       │           │
└────────────────────────────────┴───────┴─────────┴──────────────────────────────────────────────────────┴──────┴──────┴──────────┴──────┴─────────┴───────────┘

1 row in set. Elapsed: 0.005 sec.

clickhouse01 :)  select * from system.settings where name ='query_plan_optimize_primary_key';

SELECT *
FROM system.settings
WHERE name = 'query_plan_optimize_primary_key'

Query id: a5f81b8b-5aa5-4665-b02b-ba85c458c7b8

┌─name────────────────────────────┬─value─┬─changed─┬─description───────────────────────────────────────────┬─min──┬─max──┬─readonly─┬─type─┬─default─┬─alias_for─┐
│ query_plan_optimize_primary_key │ 1     │       0 │ Analyze primary key using query plan (instead of AST) │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │        0 │ Bool │ 1       │           │
└─────────────────────────────────┴───────┴─────────┴───────────────────────────────────────────────────────┴──────┴──────┴──────────┴──────┴─────────┴───────────┘

1 row in set. Elapsed: 0.005 sec.

@yangshike
Copy link
Author

yangshike commented Aug 4, 2023 •
edited by novikd

this only disable setting for query_plan_optimize_primary_key

clickhouse02 :) select * from system.settings where name ='query_plan_optimize_projection';

SELECT *
FROM system.settings
WHERE name = 'query_plan_optimize_projection'

Query id: c7d48dc6-75c1-4089-b193-692618f97d30

┌─name───────────────────────────┬─value─┬─changed─┬─description──────────────────────────────────────────┬─min──┬─max──┬─readonly─┬─type─┬─default─┬─alias_for─┬─is_obsolete─┐
│ query_plan_optimize_projection │ 1     │       1 │ Use query plan for aggregation-in-order optimisation │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │        0 │ Bool │ 1       │           │           0 │
└────────────────────────────────┴───────┴─────────┴──────────────────────────────────────────────────────┴──────┴──────┴──────────┴──────┴─────────┴───────────┴─────────────┘

1 row in set. Elapsed: 0.003 sec.

clickhouse02 :) select * from system.settings where name ='query_plan_optimize_primary_key';

SELECT *
FROM system.settings
WHERE name = 'query_plan_optimize_primary_key'

Query id: 8e61e3e5-7099-4b09-9017-ea73798dee10

┌─name────────────────────────────┬─value─┬─changed─┬─description───────────────────────────────────────────┬─min──┬─max──┬─readonly─┬─type─┬─default─┬─alias_for─┬─is_obsolete─┐
│ query_plan_optimize_primary_key │ 0     │       1 │ Analyze primary key using query plan (instead of AST) │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │        0 │ Bool │ 1       │           │           0 │
└─────────────────────────────────┴───────┴─────────┴───────────────────────────────────────────────────────┴──────┴──────┴──────────┴──────┴─────────┴───────────┴─────────────┘

1 row in set. Elapsed: 0.004 sec.

clickhouse02 :) EXPLAIN indexes = 1  SELECT live_start_time, camp_date_id, course_id , min(`camp_date_name`) camp_date_name, min(`class_name`) class_name, min(`order_number`) `order_number`, min(`flow_channel`) `flow_channel`, min(`flow_type`) `flow_type`, min(`flow_source`) `flow_source`, min(`channel_name`)`channel_name` , min(`sub_channel_name`) `sub_channel_name`, sum(apply_count) apply_count, sum(join_count) join_count, sum(recall_count) recall_count, sum(come_count) come_count, sum(come_convert_count) come_convert_count, sum(effective_count) effective_count, sum(thirty_count) thirty_count, sum(sixty_count) sixty_count, sum(two_hour_count) two_hour_count, sum(base_live_high_order_count) base_live_high_order_count, sum(popup_base_live_high_order_count) popup_base_live_high_order_count, sum(live_high_order_count) live_high_order_count, sum(live_high_order_amount) live_high_order_amount, sum(sale_high_order_count) sale_high_order_count, sum(sale_high_order_amount) sale_high_order_amount, sum(refund_high_order_count) refund_high_order_count, sum(refund_high_order_amount) refund_high_order_amount, sum(unpaid_order_count) unpaid_order_count, ifnull(sum(new_unpaid_count),0) new_unpaid_count, min(`created_time`) `created_time` FROM t_bi_live_watcher_stat right join ( select camp_date_id camp_date_id_version , live_start_time live_start_time_version, max(version) max_version from t_bi_live_watcher_stat where camp_date_id in (735200065002975412,734147951933796413,734147924259809357,734147896665489892 ) group by camp_date_id, live_start_time ) t_version_one on t_bi_live_watcher_stat.camp_date_id = t_version_one.camp_date_id_version and t_bi_live_watcher_stat.live_start_time = t_version_one.live_start_time_version and t_bi_live_watcher_stat.version = t_version_one.max_version where camp_date_id in (735200065002975412,734147951933796413,734147924259809357,734147896665489892 )group by live_start_time, camp_date_id, course_id order by order_number asc, live_start_time asc FORMAT TabSeparatedWithNamesAndTypes;

EXPLAIN indexes = 1
SELECT
    live_start_time,
    camp_date_id,
    course_id,
    min(camp_date_name) AS camp_date_name,
    min(class_name) AS class_name,
    min(order_number) AS order_number,
    min(flow_channel) AS flow_channel,
    min(flow_type) AS flow_type,
    min(flow_source) AS flow_source,
    min(channel_name) AS channel_name,
    min(sub_channel_name) AS sub_channel_name,
    sum(apply_count) AS apply_count,
    sum(join_count) AS join_count,
    sum(recall_count) AS recall_count,
    sum(come_count) AS come_count,
    sum(come_convert_count) AS come_convert_count,
    sum(effective_count) AS effective_count,
    sum(thirty_count) AS thirty_count,
    sum(sixty_count) AS sixty_count,
    sum(two_hour_count) AS two_hour_count,
    sum(base_live_high_order_count) AS base_live_high_order_count,
    sum(popup_base_live_high_order_count) AS popup_base_live_high_order_count,
    sum(live_high_order_count) AS live_high_order_count,
    sum(live_high_order_amount) AS live_high_order_amount,
    sum(sale_high_order_count) AS sale_high_order_count,
    sum(sale_high_order_amount) AS sale_high_order_amount,
    sum(refund_high_order_count) AS refund_high_order_count,
    sum(refund_high_order_amount) AS refund_high_order_amount,
    sum(unpaid_order_count) AS unpaid_order_count,
    ifnull(sum(new_unpaid_count), 0) AS new_unpaid_count,
    min(created_time) AS created_time
FROM t_bi_live_watcher_stat
RIGHT JOIN
(
    SELECT
        camp_date_id AS camp_date_id_version,
        live_start_time AS live_start_time_version,
        max(version) AS max_version
    FROM t_bi_live_watcher_stat
    WHERE camp_date_id IN (735200065002975412, 734147951933796413, 734147924259809357, 734147896665489892)
    GROUP BY
        camp_date_id,
        live_start_time
) AS t_version_one ON (t_bi_live_watcher_stat.camp_date_id = t_version_one.camp_date_id_version) AND (t_bi_live_watcher_stat.live_start_time = t_version_one.live_start_time_version) AND (t_bi_live_watcher_stat.version = t_version_one.max_version)
WHERE camp_date_id IN (735200065002975412, 734147951933796413, 734147924259809357, 734147896665489892)
GROUP BY
    live_start_time,
    camp_date_id,
    course_id
ORDER BY
    order_number ASC,
    live_start_time ASC
FORMAT TabSeparatedWithNamesAndTypes

Query id: 0bece5fa-f3da-4a7d-bf3e-e22c03695d13

explain
String
Expression ((Projection + Before ORDER BY [lifted up part]))
  Sorting (Sorting for ORDER BY)
    Expression (Before ORDER BY)
      Aggregating
        Expression (Before GROUP BY)
          Filter (WHERE)
            Join (JOIN FillRightFirst)
              Expression (Before JOIN)
                ReadFromMergeTree (click_kuaicaibi.t_bi_live_watcher_stat)
                Indexes:
                  MinMax
                    Condition: true
                    Parts: 4300/4300
                    Granules: 70428/70428
                  Partition
                    Condition: true
                    Parts: 4300/4300
                    Granules: 70428/70428
                  PrimaryKey
                    Keys:
                      camp_date_id
                    Condition: (camp_date_id in 4-element set)
                    Parts: 21/4300
                    Granules: 249/70428
              Expression ((Joined actions + (Rename joined columns + (Projection + Before ORDER BY))))
                Aggregating
                  Filter
                    ReadFromMergeTree (p1)
                    Indexes:
                      PrimaryKey
                        Keys:
                          camp_date_id
                        Condition: (camp_date_id in 4-element set)
                        Parts: 21/4300
                        Granules: 21/70428

35 rows in set. Elapsed: 0.139 sec.

@alexey-milovidov alexey-milovidov changed the title query performance（version 23.7.2） query performance (version 23.7.2) Aug 6, 2023
@UnamedRus
Copy link
Contributor

Probably related #52308 ?

@yangshike
Copy link
Author

I want to know if this is a problem or not.

On 23.5, both parameter values are 1, but on 23.7, I need to set one of them to 0 (set query_plan_optimize_primary_key=0)

@novikd novikd removed the st-need-info We need extra data to continue (waiting for response) label Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@davenger @UnamedRus @novikd @yangshike @alexey-milovidov