[improvement](mtmv) Support union rewrite when the materialized view is not enough to provide all the data for the query #33800

seawinde · 2024-04-17T12:30:10Z

Proposed changes

When the materialized view is not enough to provide all the data for the query, if the materialized view is increment update by partition. we can union materialized view and origin query to reponse the query.

this depends on #33362

such as materialized view def is as following:

    CREATE MATERIALIZED VIEW mv_10086
    BUILD IMMEDIATE REFRESH AUTO ON MANUAL
    partition by(l_shipdate)
    DISTRIBUTED BY RANDOM BUCKETS 2
    PROPERTIES ('replication_num' = '1') 
    AS 
select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total
from lineitem
left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate
group by
l_shipdate,
o_orderdate,
l_partkey,
l_suppkey;

the materialized view data is as following:
+------------+-------------+-----------+-----------+-----------+
| l_shipdate | o_orderdate | l_partkey | l_suppkey | sum_total |
+------------+-------------+-----------+-----------+-----------+
| 2023-10-18 | 2023-10-18 | 2 | 3 | 109.20 |
| 2023-10-17 | 2023-10-17 | 2 | 3 | 99.50 |
| 2023-10-19 | 2023-10-19 | 2 | 3 | 99.50 |
+------------+-------------+-----------+-----------+-----------+

when we insert data to partition 2023-10-17, if we run query as following

    select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total
    from lineitem
    left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate
    group by
    l_shipdate,
    o_orderdate,
    l_partkey,
    l_suppkey;

query rewrite by materialzied view will fail with message Check partition query used validation fail
if we turn on the switch SET enable_materialized_view_union_rewrite = true; default true
we run the query above again, it will success and will use union all materialized view and origin query to response the query correctly. the plan is as following:

| Explain String(Nereids Planner)                                                                                                                                                                    |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                                                                                                    |
|   OUTPUT EXPRS:                                                                                                                                                                                    |
|     l_shipdate[#52]                                                                                                                                                                                |
|     o_orderdate[#53]                                                                                                                                                                               |
|     l_partkey[#54]                                                                                                                                                                                 |
|     l_suppkey[#55]                                                                                                                                                                                 |
|     sum_total[#56]                                                                                                                                                                                 |
|   PARTITION: UNPARTITIONED                                                                                                                                                                         |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   VRESULT SINK                                                                                                                                                                                     |
|      MYSQL_PROTOCAL                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   11:VEXCHANGE                                                                                                                                                                                     |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 1                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                   |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 11                                                                                                                                                                                |
|     UNPARTITIONED                                                                                                                                                                                  |
|                                                                                                                                                                                                    |
|   10:VUNION(756)                                                                                                                                                                                   |
|   |                                                                                                                                                                                                |
|   |----9:VAGGREGATE (merge finalize)(753)                                                                                                                                                          |
|   |    |  output: sum(partial_sum(o_totalprice)[#46])[#51]                                                                                                                                         |
|   |    |  group by: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                              |
|   |    |  cardinality=2                                                                                                                                                                            |
|   |    |  distribute expr lists: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                 |
|   |    |                                                                                                                                                                                           |
|   |    8:VEXCHANGE                                                                                                                                                                                 |
|   |       offset: 0                                                                                                                                                                                |
|   |       distribute expr lists: l_shipdate[#42]                                                                                                                                                   |
|   |                                                                                                                                                                                                |
|   1:VEXCHANGE                                                                                                                                                                                      |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 2                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25]                                                                                                                                   |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 08                                                                                                                                                                                |
|     HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                            |
|                                                                                                                                                                                                    |
|   7:VAGGREGATE (update serialize)(747)                                                                                                                                                             |
|   |  STREAMING                                                                                                                                                                                     |
|   |  output: partial_sum(o_totalprice[#41])[#46]                                                                                                                                                   |
|   |  group by: l_shipdate[#37], o_orderdate[#38], l_partkey[#39], l_suppkey[#40]                                                                                                                   |
|   |  cardinality=2                                                                                                                                                                                 |
|   |  distribute expr lists: l_shipdate[#37]                                                                                                                                                        |
|   |                                                                                                                                                                                                |
|   6:VHASH JOIN(741)                                                                                                                                                                                |
|   |  join op: RIGHT OUTER JOIN(PARTITIONED)[]                                                                                                                                                      |
|   |  equal join conjunct: (o_orderkey[#21] = l_orderkey[#5])                                                                                                                                       |
|   |  equal join conjunct: (o_orderdate[#25] = l_shipdate[#15])                                                                                                                                     |
|   |  runtime filters: RF000[min_max] <- l_orderkey[#5](2/2/2048), RF001[bloom] <- l_orderkey[#5](2/2/2048), RF002[min_max] <- l_shipdate[#15](1/1/2048), RF003[bloom] <- l_shipdate[#15](1/1/2048) |
|   |  cardinality=2                                                                                                                                                                                 |
|   |  vec output tuple id: 4                                                                                                                                                                        |
|   |  output tuple id: 4                                                                                                                                                                            |
|   |  vIntermediate tuple ids: 3                                                                                                                                                                    |
|   |  hash output slot ids: 6 7 24 25 15                                                                                                                                                            |
|   |  final projections: l_shipdate[#36], o_orderdate[#32], l_partkey[#34], l_suppkey[#35], o_totalprice[#31]                                                                                       |
|   |  final project output tuple id: 4                                                                                                                                                              |
|   |  distribute expr lists: o_orderkey[#21], o_orderdate[#25]                                                                                                                                      |
|   |  distribute expr lists: l_orderkey[#5], l_shipdate[#15]                                                                                                                                        |
|   |                                                                                                                                                                                                |
|   |----3:VEXCHANGE                                                                                                                                                                                 |
|   |       offset: 0                                                                                                                                                                                |
|   |       distribute expr lists: l_orderkey[#5]                                                                                                                                                    |
|   |                                                                                                                                                                                                |
|   5:VEXCHANGE                                                                                                                                                                                      |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 3                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: RANDOM                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 05                                                                                                                                                                                |
|     HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25]                                                                                                                                            |
|                                                                                                                                                                                                    |
|   4:VOlapScanNode(722)                                                                                                                                                                             |
|      TABLE: union_db.orders(orders), PREAGGREGATION: ON                                                                                                                                            |
|      runtime filters: RF000[min_max] -> o_orderkey[#21], RF001[bloom] -> o_orderkey[#21], RF002[min_max] -> o_orderdate[#25], RF003[bloom] -> o_orderdate[#25]                                     |
|      partitions=3/3 (p_20231017,p_20231018,p_20231019), tablets=9/9, tabletList=161188,161190,161192 ...                                                                                           |
|      cardinality=3, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 4                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: l_orderkey[#5]                                                                                                                                                      |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 03                                                                                                                                                                                |
|     HASH_PARTITIONED: l_orderkey[#5], l_shipdate[#15]                                                                                                                                              |
|                                                                                                                                                                                                    |
|   2:VOlapScanNode(729)                                                                                                                                                                             |
|      TABLE: union_db.lineitem(lineitem), PREAGGREGATION: ON                                                                                                                                        |
|      PREDICATES: (l_shipdate[#15] >= '2023-10-17') AND (l_shipdate[#15] < '2023-10-18')                                                                                                            |
|      partitions=1/3 (p_20231017), tablets=3/3, tabletList=161223,161225,161227                                                                                                                     |
|      cardinality=2, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 5                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: RANDOM                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 01                                                                                                                                                                                |
|     RANDOM                                                                                                                                                                                         |
|                                                                                                                                                                                                    |
|   0:VOlapScanNode(718)                                                                                                                                                                             |
|      TABLE: union_db.mv_10086(mv_10086), PREAGGREGATION: ON                                                                                                                                        |
|      partitions=2/3 (p_20231018_20231019,p_20231019_20231020), tablets=4/4, tabletList=161251,161253,161265 ...                                                                                    |
|      cardinality=2, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| MaterializedView                                                                                                                                                                                   |
| MaterializedViewRewriteSuccessAndChose:                                                                                                                                                            |
|   Names: mv_10086                                                                                                                                                                                  |
| MaterializedViewRewriteSuccessButNotChose:                                                                                                                                                         |
|                                                                                                                                                                                                    |
| MaterializedViewRewriteFail:                                                                                                                                                                       |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-17T12:30:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

seawinde · 2024-04-17T12:31:28Z

run buildall

doris-robot · 2024-04-17T13:07:58Z

TPC-H: Total hot run time: 38414 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9de6c83087f16c53e84564caa4a4d231203c62e6, data reload: false

------ Round 1 ----------------------------------
q1	17605	4318	4242	4242
q2	2024	181	179	179
q3	10473	1175	1170	1170
q4	10213	754	778	754
q5	7511	2650	2599	2599
q6	217	132	134	132
q7	1003	646	598	598
q8	9253	2053	2011	2011
q9	7223	6521	6511	6511
q10	8660	3554	3521	3521
q11	484	225	232	225
q12	441	215	215	215
q13	17770	2958	2955	2955
q14	275	228	246	228
q15	514	467	463	463
q16	511	383	380	380
q17	960	675	671	671
q18	7331	6749	6712	6712
q19	8478	1513	1492	1492
q20	687	319	308	308
q21	3409	2747	2814	2747
q22	371	301	315	301
Total cold run time: 115413 ms
Total hot run time: 38414 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4209	4201	4201
q2	375	264	268	264
q3	3008	2710	2743	2710
q4	1859	1655	1567	1567
q5	5291	5287	5291	5287
q6	209	121	122	121
q7	2234	1853	1867	1853
q8	3161	3325	3345	3325
q9	8537	8507	8648	8507
q10	4079	3923	4022	3923
q11	634	506	497	497
q12	802	652	642	642
q13	15956	3284	3133	3133
q14	327	281	283	281
q15	507	468	472	468
q16	521	437	438	437
q17	1835	1532	1504	1504
q18	8041	8003	7761	7761
q19	1695	1605	1574	1574
q20	1958	1852	1856	1852
q21	7411	5004	4896	4896
q22	534	451	450	450
Total cold run time: 73299 ms
Total hot run time: 55253 ms

doris-robot · 2024-04-17T13:19:34Z

TPC-DS: Total hot run time: 184675 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9de6c83087f16c53e84564caa4a4d231203c62e6, data reload: false

query1	907	371	364	364
query2	6195	2862	2364	2364
query3	6657	210	204	204
query4	23404	21343	21299	21299
query5	4156	417	410	410
query6	276	183	179	179
query7	4591	285	280	280
query8	222	176	172	172
query9	8510	2299	2301	2299
query10	416	233	250	233
query11	14810	14224	14099	14099
query12	135	85	83	83
query13	1629	359	352	352
query14	8629	8391	7314	7314
query15	258	170	180	170
query16	8184	268	261	261
query17	1988	586	562	562
query18	2115	286	288	286
query19	325	153	155	153
query20	97	89	86	86
query21	192	126	122	122
query22	5003	4743	4755	4743
query23	33800	33125	33442	33125
query24	7701	3108	3081	3081
query25	601	412	408	408
query26	717	155	161	155
query27	2218	365	355	355
query28	5476	2080	2064	2064
query29	869	683	628	628
query30	316	178	185	178
query31	989	765	748	748
query32	90	53	51	51
query33	540	268	237	237
query34	907	496	494	494
query35	821	697	691	691
query36	1062	899	949	899
query37	110	68	67	67
query38	3512	3497	3313	3313
query39	1634	1593	1568	1568
query40	173	139	135	135
query41	49	43	44	43
query42	107	98	102	98
query43	606	560	561	560
query44	1114	749	755	749
query45	283	282	273	273
query46	1113	736	746	736
query47	2014	1899	1945	1899
query48	364	299	297	297
query49	815	375	380	375
query50	787	400	400	400
query51	6846	6808	6747	6747
query52	99	98	90	90
query53	348	277	275	275
query54	267	250	234	234
query55	75	75	73	73
query56	238	232	228	228
query57	1290	1200	1212	1200
query58	214	202	200	200
query59	3734	3202	3268	3202
query60	254	229	233	229
query61	92	86	87	86
query62	586	429	443	429
query63	306	275	272	272
query64	4754	3987	3757	3757
query65	3054	3031	3019	3019
query66	740	333	323	323
query67	15321	15311	14743	14743
query68	7792	531	534	531
query69	530	306	309	306
query70	1216	1201	1169	1169
query71	1441	1265	1262	1262
query72	6437	2621	2451	2451
query73	731	315	313	313
query74	6882	6421	6336	6336
query75	3809	2630	2604	2604
query76	4424	983	969	969
query77	609	257	286	257
query78	11096	10122	10128	10122
query79	7957	526	509	509
query80	1298	436	431	431
query81	496	251	247	247
query82	886	96	93	93
query83	190	165	163	163
query84	268	82	84	82
query85	1217	269	268	268
query86	448	289	307	289
query87	3541	3283	3255	3255
query88	5025	2392	2332	2332
query89	490	374	378	374
query90	1853	187	182	182
query91	125	97	96	96
query92	53	48	52	48
query93	6562	500	501	500
query94	1052	187	177	177
query95	389	299	295	295
query96	606	277	264	264
query97	3115	2920	2933	2920
query98	231	228	234	228
query99	1249	851	833	833
Total cold run time: 287723 ms
Total hot run time: 184675 ms

doris-robot · 2024-04-17T13:25:05Z

ClickBench: Total hot run time: 30.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9de6c83087f16c53e84564caa4a4d231203c62e6, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.05	0.06
query4	1.66	0.07	0.08
query5	0.51	0.50	0.49
query6	1.47	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.05
query9	0.55	0.50	0.49
query10	0.56	0.56	0.55
query11	0.16	0.11	0.12
query12	0.15	0.11	0.12
query13	0.60	0.59	0.59
query14	0.76	0.77	0.81
query15	0.82	0.81	0.80
query16	0.36	0.37	0.37
query17	0.93	0.93	1.01
query18	0.23	0.22	0.24
query19	1.78	1.78	1.68
query20	0.02	0.01	0.01
query21	15.40	0.64	0.64
query22	4.69	6.28	2.17
query23	18.29	1.38	1.24
query24	1.40	0.29	0.29
query25	0.14	0.08	0.08
query26	0.25	0.16	0.17
query27	0.07	0.08	0.08
query28	13.40	1.00	0.98
query29	12.57	3.23	3.24
query30	0.26	0.06	0.06
query31	2.84	0.37	0.38
query32	3.32	0.46	0.46
query33	2.83	2.82	2.84
query34	17.17	4.46	4.39
query35	4.50	4.48	4.57
query36	0.66	0.46	0.48
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.17	0.14	0.15
query41	0.09	0.04	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.48 s
Total hot run time: 30.56 s

doris-robot · 2024-04-17T13:28:11Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9de6c83087f16c53e84564caa4a4d231203c62e6 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

seawinde · 2024-04-19T14:05:03Z

run buildall

doris-robot · 2024-04-19T14:22:37Z

TPC-H: Total hot run time: 38823 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad4e5f8ac80ebcd71817f2aebb3807bee20ebcfc, data reload: false

------ Round 1 ----------------------------------
q1	17748	4512	4461	4461
q2	2636	194	191	191
q3	12212	1156	1226	1156
q4	10795	794	825	794
q5	7480	2675	2682	2675
q6	217	133	133	133
q7	1008	601	609	601
q8	9224	2091	2064	2064
q9	7410	6630	6583	6583
q10	8434	3538	3507	3507
q11	434	235	221	221
q12	402	221	212	212
q13	17757	2950	2974	2950
q14	264	220	223	220
q15	539	490	476	476
q16	533	382	369	369
q17	995	754	691	691
q18	7253	6686	6744	6686
q19	1621	1541	1464	1464
q20	641	324	312	312
q21	3466	2745	2819	2745
q22	366	312	314	312
Total cold run time: 111435 ms
Total hot run time: 38823 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4255	4196	4209	4196
q2	386	264	274	264
q3	2992	2704	2762	2704
q4	1903	1521	1562	1521
q5	5333	5328	5349	5328
q6	210	121	125	121
q7	2241	1876	1859	1859
q8	3201	3345	3361	3345
q9	8671	8578	8625	8578
q10	3909	3704	3659	3659
q11	591	478	488	478
q12	733	552	594	552
q13	16522	2912	2914	2912
q14	307	264	278	264
q15	527	476	470	470
q16	483	412	419	412
q17	1787	1483	1455	1455
q18	7537	7451	7376	7376
q19	1628	1553	1507	1507
q20	1926	1742	1744	1742
q21	4938	4694	4672	4672
q22	508	451	466	451
Total cold run time: 70588 ms
Total hot run time: 53866 ms

doris-robot · 2024-04-19T14:34:16Z

TPC-DS: Total hot run time: 184448 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ad4e5f8ac80ebcd71817f2aebb3807bee20ebcfc, data reload: false

query1	914	387	376	376
query2	6478	2520	2375	2375
query3	6646	198	208	198
query4	24795	21324	21296	21296
query5	4216	403	420	403
query6	277	186	176	176
query7	4599	287	286	286
query8	236	184	200	184
query9	8365	2318	2322	2318
query10	593	251	247	247
query11	14804	14220	14323	14220
query12	140	90	83	83
query13	1645	364	355	355
query14	9910	7424	7419	7419
query15	257	180	178	178
query16	8144	264	259	259
query17	1917	604	554	554
query18	2103	277	279	277
query19	197	153	155	153
query20	90	87	86	86
query21	204	127	124	124
query22	5019	4848	4871	4848
query23	33923	33329	33317	33317
query24	11854	3019	3006	3006
query25	643	359	364	359
query26	1731	145	147	145
query27	3085	305	302	302
query28	7399	1995	1992	1992
query29	1040	599	593	593
query30	279	174	174	174
query31	954	732	720	720
query32	84	51	59	51
query33	756	239	234	234
query34	1105	473	465	465
query35	842	688	683	683
query36	1070	911	934	911
query37	280	71	77	71
query38	3297	3159	3227	3159
query39	1578	1509	1533	1509
query40	273	124	122	122
query41	45	43	43	43
query42	102	94	93	93
query43	586	561	542	542
query44	1253	751	724	724
query45	279	262	263	262
query46	1070	763	706	706
query47	1921	1841	1846	1841
query48	351	283	292	283
query49	1184	384	377	377
query50	753	370	375	370
query51	6731	6753	6811	6753
query52	110	90	86	86
query53	354	276	280	276
query54	300	224	221	221
query55	73	72	71	71
query56	242	227	217	217
query57	1227	1118	1139	1118
query58	222	189	198	189
query59	3271	3385	3042	3042
query60	263	225	229	225
query61	86	87	90	87
query62	650	440	422	422
query63	300	276	274	274
query64	6318	4016	3857	3857
query65	3147	3042	3031	3031
query66	1365	330	335	330
query67	15403	14886	15151	14886
query68	5198	522	523	522
query69	463	296	290	290
query70	1215	1187	1127	1127
query71	1397	1264	1261	1261
query72	6509	2633	2429	2429
query73	716	318	315	315
query74	6870	6444	6489	6444
query75	3409	2700	2587	2587
query76	3379	956	972	956
query77	400	258	254	254
query78	11045	10241	10177	10177
query79	4131	514	509	509
query80	1770	423	429	423
query81	528	255	243	243
query82	896	95	98	95
query83	278	171	169	169
query84	262	87	84	84
query85	2149	267	257	257
query86	514	288	296	288
query87	3423	3260	3349	3260
query88	4645	2307	2310	2307
query89	492	373	372	372
query90	2013	182	181	181
query91	124	96	96	96
query92	63	48	46	46
query93	4961	500	492	492
query94	1282	178	179	178
query95	387	313	297	297
query96	606	267	260	260
query97	3156	2923	2920	2920
query98	236	221	213	213
query99	1206	842	860	842
Total cold run time: 293071 ms
Total hot run time: 184448 ms

doris-robot · 2024-04-19T14:39:45Z

ClickBench: Total hot run time: 30.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ad4e5f8ac80ebcd71817f2aebb3807bee20ebcfc, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.07	0.07
query5	0.49	0.51	0.48
query6	1.47	0.71	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.56	0.49	0.49
query10	0.55	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.58	0.58
query14	0.77	0.75	0.78
query15	0.82	0.79	0.81
query16	0.36	0.36	0.36
query17	0.94	0.97	0.95
query18	0.22	0.22	0.24
query19	1.89	1.68	1.72
query20	0.02	0.00	0.00
query21	15.43	0.66	0.65
query22	4.59	6.80	1.97
query23	18.30	1.38	1.30
query24	1.64	0.37	0.21
query25	0.16	0.08	0.08
query26	0.26	0.16	0.16
query27	0.07	0.06	0.08
query28	13.21	0.99	0.98
query29	12.58	3.27	3.25
query30	0.27	0.08	0.07
query31	2.83	0.38	0.38
query32	3.30	0.47	0.47
query33	2.84	2.77	2.80
query34	16.98	4.40	4.45
query35	4.45	4.44	4.42
query36	0.65	0.47	0.48
query37	0.17	0.16	0.16
query38	0.15	0.13	0.14
query39	0.04	0.03	0.04
query40	0.19	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.35 s
Total hot run time: 30.25 s

doris-robot · 2024-04-19T14:42:51Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ad4e5f8ac80ebcd71817f2aebb3807bee20ebcfc with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

seawinde · 2024-04-20T03:46:01Z

run buildall

seawinde · 2024-04-20T04:22:45Z

run buildall

doris-robot · 2024-04-20T04:40:53Z

TPC-H: Total hot run time: 38429 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45fb21a6f91f8d4c1bb5f85cfca594159d7cddaf, data reload: false

------ Round 1 ----------------------------------
q1	17625	4264	4331	4264
q2	2011	200	187	187
q3	10436	1128	1180	1128
q4	10183	779	711	711
q5	7509	2706	2642	2642
q6	215	131	133	131
q7	1014	597	584	584
q8	9411	2032	2022	2022
q9	7631	6590	6528	6528
q10	8505	3512	3570	3512
q11	439	233	230	230
q12	518	215	214	214
q13	18725	2936	3009	2936
q14	285	227	229	227
q15	526	484	481	481
q16	507	378	376	376
q17	943	687	694	687
q18	7385	6693	6768	6693
q19	8118	1526	1460	1460
q20	679	315	294	294
q21	3513	2834	2818	2818
q22	366	308	304	304
Total cold run time: 116544 ms
Total hot run time: 38429 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4338	4243	4226	4226
q2	367	269	263	263
q3	2942	2750	2754	2750
q4	1838	1616	1576	1576
q5	5317	5333	5304	5304
q6	208	122	122	122
q7	2215	1847	1887	1847
q8	3200	3350	3312	3312
q9	8585	8545	8772	8545
q10	4109	3928	3938	3928
q11	631	501	499	499
q12	805	645	638	638
q13	17216	3248	3168	3168
q14	333	298	286	286
q15	529	475	492	475
q16	509	437	450	437
q17	1844	1545	1489	1489
q18	8067	7955	7857	7857
q19	1653	1572	1586	1572
q20	2053	1908	1803	1803
q21	5129	5068	4926	4926
q22	534	485	472	472
Total cold run time: 72422 ms
Total hot run time: 55495 ms

doris-robot · 2024-04-20T04:52:33Z

TPC-DS: Total hot run time: 185388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45fb21a6f91f8d4c1bb5f85cfca594159d7cddaf, data reload: false

query1	917	372	385	372
query2	6186	2694	2405	2405
query3	6644	203	200	200
query4	23402	21288	21178	21178
query5	4188	405	410	405
query6	278	190	179	179
query7	4586	299	288	288
query8	248	202	192	192
query9	8488	2335	2339	2335
query10	419	247	254	247
query11	14743	14101	14216	14101
query12	133	88	88	88
query13	1634	357	356	356
query14	9745	7913	7456	7456
query15	254	185	180	180
query16	8194	260	260	260
query17	1911	572	537	537
query18	2094	268	269	268
query19	238	150	143	143
query20	91	84	83	83
query21	196	132	129	129
query22	5047	4822	4842	4822
query23	33773	33269	33161	33161
query24	10842	3189	3045	3045
query25	585	387	392	387
query26	702	159	167	159
query27	2322	366	373	366
query28	5863	2074	2155	2074
query29	863	618	614	614
query30	280	187	181	181
query31	1025	753	746	746
query32	97	55	52	52
query33	657	249	262	249
query34	926	490	495	490
query35	861	690	707	690
query36	1062	937	915	915
query37	114	71	71	71
query38	3427	3368	3274	3274
query39	1634	1596	1589	1589
query40	179	131	129	129
query41	47	44	48	44
query42	104	99	93	93
query43	597	566	538	538
query44	1082	735	758	735
query45	281	274	281	274
query46	1133	771	745	745
query47	2056	1985	1982	1982
query48	380	296	316	296
query49	895	420	395	395
query50	786	381	384	381
query51	6923	6804	6738	6738
query52	102	88	88	88
query53	348	272	280	272
query54	296	229	229	229
query55	78	73	71	71
query56	237	220	219	219
query57	1197	1127	1116	1116
query58	217	203	207	203
query59	3364	3090	3098	3090
query60	261	229	237	229
query61	89	88	86	86
query62	604	441	443	441
query63	302	274	273	273
query64	4589	3897	4055	3897
query65	3067	3000	3062	3000
query66	770	340	340	340
query67	15339	15179	15039	15039
query68	5145	535	533	533
query69	496	301	299	299
query70	1227	1190	1180	1180
query71	1393	1271	1265	1265
query72	6482	2757	2557	2557
query73	714	317	319	317
query74	6853	6399	6421	6399
query75	3369	2707	2581	2581
query76	3244	968	920	920
query77	438	269	266	266
query78	11027	10130	10244	10130
query79	8822	520	514	514
query80	1966	459	458	458
query81	542	248	253	248
query82	1428	97	96	96
query83	318	180	168	168
query84	269	89	88	88
query85	1701	321	319	319
query86	455	303	309	303
query87	3451	3316	3290	3290
query88	5182	2322	2330	2322
query89	581	377	368	368
query90	1920	183	180	180
query91	136	108	107	107
query92	54	47	49	47
query93	6706	502	492	492
query94	1097	186	182	182
query95	392	305	297	297
query96	617	265	262	262
query97	3109	2894	2928	2894
query98	239	214	216	214
query99	1222	911	862	862
Total cold run time: 290587 ms
Total hot run time: 185388 ms

doris-robot · 2024-04-20T04:58:04Z

ClickBench: Total hot run time: 31.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45fb21a6f91f8d4c1bb5f85cfca594159d7cddaf, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.09
query5	0.50	0.50	0.50
query6	1.48	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.55	0.56	0.54
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.60	0.58	0.57
query14	0.74	0.78	0.76
query15	0.82	0.81	0.81
query16	0.36	0.36	0.36
query17	1.01	1.00	1.00
query18	0.22	0.24	0.24
query19	1.87	1.80	1.79
query20	0.01	0.00	0.01
query21	15.40	0.65	0.65
query22	4.08	6.20	2.48
query23	18.29	1.33	1.24
query24	1.53	0.34	0.21
query25	0.15	0.08	0.08
query26	0.25	0.16	0.15
query27	0.08	0.08	0.07
query28	13.32	0.98	0.97
query29	12.58	3.42	3.40
query30	0.26	0.08	0.06
query31	2.85	0.38	0.37
query32	3.30	0.45	0.45
query33	2.77	2.90	2.86
query34	17.15	4.43	4.49
query35	4.48	4.53	4.60
query36	0.66	0.46	0.46
query37	0.19	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.04	0.05
query42	0.05	0.04	0.05
query43	0.03	0.04	0.03
Total cold run time: 108.96 s
Total hot run time: 31.22 s

seawinde · 2024-04-20T08:25:35Z

run buildall

doris-robot · 2024-04-20T09:02:10Z

TPC-H: Total hot run time: 38248 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 538a57558675d3fd883797018a8938dff477f227, data reload: false

------ Round 1 ----------------------------------
q1	17616	4277	4212	4212
q2	2010	189	198	189
q3	10450	1175	1121	1121
q4	10188	802	750	750
q5	7479	2676	2602	2602
q6	218	132	129	129
q7	1015	617	586	586
q8	9231	2065	2038	2038
q9	7414	6592	6526	6526
q10	8584	3498	3485	3485
q11	434	224	223	223
q12	439	214	204	204
q13	17889	2921	2910	2910
q14	269	224	239	224
q15	518	476	482	476
q16	527	372	369	369
q17	967	666	729	666
q18	7381	6849	6666	6666
q19	4854	1515	1510	1510
q20	641	308	297	297
q21	3554	2772	2839	2772
q22	356	312	293	293
Total cold run time: 112034 ms
Total hot run time: 38248 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4336	4199	4214	4199
q2	381	271	259	259
q3	2986	2720	2711	2711
q4	1862	1659	1595	1595
q5	5314	5306	5283	5283
q6	206	122	123	122
q7	2184	1883	1859	1859
q8	3209	3315	3349	3315
q9	8597	8588	8664	8588
q10	4148	3885	3932	3885
q11	626	498	482	482
q12	797	638	614	614
q13	16342	3197	3344	3197
q14	324	295	302	295
q15	527	476	471	471
q16	500	451	456	451
q17	1814	1516	1512	1512
q18	8260	7933	7992	7933
q19	1719	1558	1596	1558
q20	1976	1758	1818	1758
q21	5181	5095	4982	4982
q22	549	467	528	467
Total cold run time: 71838 ms
Total hot run time: 55536 ms

doris-robot · 2024-04-20T09:14:03Z

TPC-DS: Total hot run time: 184213 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 538a57558675d3fd883797018a8938dff477f227, data reload: false

query1	3160	397	390	390
query2	7338	2807	2715	2715
query3	7959	247	216	216
query4	23217	21310	21387	21310
query5	4140	411	404	404
query6	277	188	184	184
query7	4593	283	285	283
query8	232	186	185	185
query9	8493	2353	2304	2304
query10	417	232	238	232
query11	14771	14128	14109	14109
query12	140	86	83	83
query13	1643	360	350	350
query14	9927	7594	7906	7594
query15	252	173	178	173
query16	8268	283	265	265
query17	2119	575	559	559
query18	2126	278	276	276
query19	347	155	158	155
query20	95	87	84	84
query21	203	122	123	122
query22	5031	4841	4840	4840
query23	33955	33175	33248	33175
query24	11067	3057	3044	3044
query25	625	403	408	403
query26	1037	154	168	154
query27	2475	380	371	371
query28	7138	2073	2060	2060
query29	896	649	635	635
query30	317	180	184	180
query31	997	776	766	766
query32	103	53	55	53
query33	745	247	249	247
query34	1209	491	502	491
query35	886	697	700	697
query36	1048	1004	910	910
query37	128	71	73	71
query38	3455	3408	3320	3320
query39	1063	1032	1029	1029
query40	174	127	127	127
query41	44	44	45	44
query42	110	101	105	101
query43	595	556	540	540
query44	1147	772	764	764
query45	275	274	264	264
query46	1108	760	734	734
query47	2031	1977	1904	1904
query48	363	299	298	298
query49	851	408	398	398
query50	777	406	394	394
query51	6830	6883	6761	6761
query52	103	92	92	92
query53	355	289	282	282
query54	303	247	235	235
query55	77	79	72	72
query56	237	227	232	227
query57	1315	1214	1192	1192
query58	230	212	203	203
query59	3581	3182	3012	3012
query60	251	233	225	225
query61	88	87	85	85
query62	597	437	428	428
query63	308	285	279	279
query64	5066	3206	4079	3206
query65	3033	3009	3021	3009
query66	766	324	320	320
query67	15313	15321	14758	14758
query68	9031	530	567	530
query69	542	298	300	298
query70	1309	1112	1130	1112
query71	1484	1264	1274	1264
query72	6410	2632	2430	2430
query73	1063	317	317	317
query74	6845	6513	6300	6300
query75	4159	2585	2596	2585
query76	5388	1050	986	986
query77	608	256	254	254
query78	11032	10273	10182	10182
query79	9799	520	523	520
query80	1680	445	440	440
query81	512	238	239	238
query82	246	95	96	95
query83	194	161	163	161
query84	266	83	86	83
query85	880	285	266	266
query86	339	281	301	281
query87	3536	3237	3258	3237
query88	4685	2316	2321	2316
query89	509	380	383	380
query90	1997	186	182	182
query91	123	97	106	97
query92	59	57	47	47
query93	6134	509	502	502
query94	1218	175	180	175
query95	400	299	294	294
query96	602	266	264	264
query97	3122	2933	2935	2933
query98	238	231	220	220
query99	1063	849	831	831
Total cold run time: 303293 ms
Total hot run time: 184213 ms

doris-robot · 2024-04-20T09:19:32Z

ClickBench: Total hot run time: 30.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 538a57558675d3fd883797018a8938dff477f227, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.06	0.06
query4	1.66	0.10	0.08
query5	0.50	0.51	0.50
query6	1.49	0.72	0.71
query7	0.03	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.54	0.54	0.53
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.58	0.58
query14	0.75	0.77	0.78
query15	0.83	0.80	0.80
query16	0.36	0.35	0.36
query17	1.01	0.92	0.98
query18	0.21	0.26	0.24
query19	1.73	1.66	1.76
query20	0.01	0.01	0.01
query21	15.40	0.66	0.65
query22	4.24	7.60	1.89
query23	18.21	1.32	1.28
query24	1.82	0.19	0.22
query25	0.14	0.09	0.08
query26	0.25	0.16	0.17
query27	0.08	0.08	0.07
query28	13.46	0.99	0.97
query29	12.60	3.29	3.24
query30	0.26	0.06	0.06
query31	2.91	0.38	0.37
query32	3.27	0.47	0.47
query33	2.79	2.84	2.76
query34	17.12	4.45	4.47
query35	4.56	4.44	4.49
query36	0.66	0.46	0.45
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.15	0.14
query41	0.10	0.04	0.05
query42	0.05	0.05	0.06
query43	0.04	0.03	0.04
Total cold run time: 109.48 s
Total hot run time: 30.16 s

seawinde · 2024-04-20T14:57:49Z

run buildall

doris-robot · 2024-04-20T15:17:01Z

TPC-H: Total hot run time: 38406 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a56de7068feaf354becff3eb9f92334a4e38cd2f, data reload: false

------ Round 1 ----------------------------------
q1	6501	4491	4269	4269
q2	657	190	184	184
q3	6388	1132	1142	1132
q4	3924	706	780	706
q5	2665	2567	2909	2567
q6	233	133	136	133
q7	1022	593	583	583
q8	1954	2041	2026	2026
q9	6724	6583	6536	6536
q10	3638	3532	3532	3532
q11	351	233	228	228
q12	374	224	218	218
q13	17652	2955	3001	2955
q14	267	228	229	228
q15	534	493	471	471
q16	500	391	382	382
q17	937	673	698	673
q18	7275	6843	6797	6797
q19	1549	1491	1457	1457
q20	499	321	309	309
q21	3460	2814	2716	2716
q22	367	304	306	304
Total cold run time: 67471 ms
Total hot run time: 38406 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4252	4291	4272	4272
q2	377	263	274	263
q3	3007	2774	2815	2774
q4	1862	1566	1586	1566
q5	5346	5306	5319	5306
q6	209	120	124	120
q7	2224	1823	1899	1823
q8	3176	3315	3304	3304
q9	8524	8539	8484	8484
q10	3900	3693	3773	3693
q11	582	480	480	480
q12	759	593	578	578
q13	13953	2943	2920	2920
q14	285	272	265	265
q15	508	469	479	469
q16	457	413	428	413
q17	1759	1477	1454	1454
q18	7716	7855	7756	7756
q19	1686	1595	1590	1590
q20	2067	1829	1871	1829
q21	5069	5026	4905	4905
q22	541	491	478	478
Total cold run time: 68259 ms
Total hot run time: 54742 ms

doris-robot · 2024-04-20T15:28:53Z

TPC-DS: Total hot run time: 186219 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a56de7068feaf354becff3eb9f92334a4e38cd2f, data reload: false

query1	897	374	378	374
query2	6481	2440	2408	2408
query3	6656	202	204	202
query4	24460	21302	21512	21302
query5	4281	423	422	422
query6	280	179	187	179
query7	5493	299	289	289
query8	238	194	193	193
query9	8746	2338	2343	2338
query10	562	247	269	247
query11	14788	14172	14181	14172
query12	138	92	86	86
query13	1633	360	352	352
query14	8859	8272	7565	7565
query15	252	184	183	183
query16	8182	263	256	256
query17	1983	598	543	543
query18	2105	273	268	268
query19	314	149	150	149
query20	91	85	84	84
query21	199	125	131	125
query22	4962	4916	4872	4872
query23	33799	33530	33955	33530
query24	10868	3060	3059	3059
query25	599	414	381	381
query26	697	153	154	153
query27	2289	364	373	364
query28	6090	2069	2039	2039
query29	838	625	606	606
query30	323	176	178	176
query31	981	788	748	748
query32	96	52	57	52
query33	771	250	253	250
query34	1299	491	526	491
query35	836	724	699	699
query36	1082	931	953	931
query37	112	74	74	74
query38	3483	3383	3320	3320
query39	1646	1604	1592	1592
query40	180	127	127	127
query41	46	44	49	44
query42	109	98	98	98
query43	600	530	537	530
query44	1186	776	762	762
query45	295	279	277	277
query46	1123	791	720	720
query47	2041	1974	1946	1946
query48	368	303	300	300
query49	859	396	401	396
query50	770	413	397	397
query51	6893	6857	6695	6695
query52	111	93	94	93
query53	349	276	282	276
query54	328	246	250	246
query55	82	75	74	74
query56	253	242	238	238
query57	1308	1206	1225	1206
query58	237	209	224	209
query59	3439	3099	3148	3099
query60	271	237	245	237
query61	108	106	104	104
query62	593	438	445	438
query63	304	281	274	274
query64	5176	4039	3912	3912
query65	3052	3033	3051	3033
query66	742	325	329	325
query67	15770	15063	15000	15000
query68	8823	543	540	540
query69	535	305	306	305
query70	1261	1123	1203	1123
query71	1491	1270	1267	1267
query72	6542	2611	2469	2469
query73	782	320	315	315
query74	6849	6531	6489	6489
query75	4033	2635	2666	2635
query76	4759	1082	967	967
query77	605	265	263	263
query78	10883	10367	10190	10190
query79	10608	508	514	508
query80	2175	427	433	427
query81	519	245	240	240
query82	646	100	93	93
query83	195	157	159	157
query84	265	79	83	79
query85	1132	277	269	269
query86	359	299	312	299
query87	3547	3230	3276	3230
query88	4865	2322	2326	2322
query89	526	375	362	362
query90	1898	187	181	181
query91	123	147	94	94
query92	64	49	46	46
query93	7018	482	489	482
query94	1106	182	173	173
query95	389	285	287	285
query96	611	265	262	262
query97	3142	2950	2966	2950
query98	236	227	220	220
query99	1218	847	858	847
Total cold run time: 301207 ms
Total hot run time: 186219 ms

doris-robot · 2024-04-20T15:34:23Z

ClickBench: Total hot run time: 30.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a56de7068feaf354becff3eb9f92334a4e38cd2f, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.26	0.05	0.05
query4	2.27	0.08	0.07
query5	0.51	0.51	0.49
query6	1.13	0.72	0.73
query7	0.01	0.01	0.01
query8	0.06	0.05	0.04
query9	0.53	0.49	0.49
query10	0.55	0.55	0.53
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.58	0.58
query14	0.75	0.76	0.78
query15	0.82	0.82	0.81
query16	0.37	0.38	0.37
query17	1.03	0.95	1.02
query18	0.22	0.25	0.22
query19	1.87	1.69	1.81
query20	0.02	0.01	0.01
query21	15.41	0.64	0.63
query22	4.23	7.09	2.20
query23	18.30	1.35	1.20
query24	1.79	0.22	0.22
query25	0.14	0.08	0.08
query26	0.27	0.16	0.16
query27	0.07	0.07	0.07
query28	13.38	1.00	0.97
query29	12.56	3.32	3.33
query30	0.25	0.07	0.07
query31	2.84	0.38	0.37
query32	3.29	0.46	0.46
query33	2.86	2.85	2.81
query34	17.16	4.42	4.42
query35	4.49	4.51	4.45
query36	0.65	0.47	0.47
query37	0.17	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.84 s
Total hot run time: 30.58 s

…is not enough to provide all the data for the query

seawinde · 2024-04-21T02:33:49Z

run buildall

doris-robot · 2024-04-21T02:54:00Z

TPC-H: Total hot run time: 38500 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b736ea3e617928c008a856367386b8a3953fcd7a, data reload: false

------ Round 1 ----------------------------------
q1	17673	4323	4216	4216
q2	2016	196	185	185
q3	10486	1185	1164	1164
q4	10189	861	732	732
q5	7541	2743	2665	2665
q6	227	136	140	136
q7	1023	624	617	617
q8	9232	2047	2049	2047
q9	7354	6564	6542	6542
q10	8575	3547	3508	3508
q11	443	246	233	233
q12	415	223	217	217
q13	17772	2949	3001	2949
q14	266	234	223	223
q15	528	488	490	488
q16	528	380	371	371
q17	950	697	657	657
q18	7421	6743	6636	6636
q19	5158	1529	1537	1529
q20	658	333	317	317
q21	3473	2861	2759	2759
q22	367	336	309	309
Total cold run time: 112295 ms
Total hot run time: 38500 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4490	4186	4232	4186
q2	383	263	268	263
q3	2991	2761	2710	2710
q4	1863	1575	1502	1502
q5	5355	5362	5288	5288
q6	212	126	128	126
q7	2214	1817	1853	1817
q8	3189	3342	3287	3287
q9	8550	8526	8525	8525
q10	3914	3882	3872	3872
q11	601	496	490	490
q12	816	630	658	630
q13	16518	3152	3139	3139
q14	308	284	278	278
q15	537	482	494	482
q16	504	441	452	441
q17	1789	1525	1493	1493
q18	8239	8098	7871	7871
q19	1688	1580	1570	1570
q20	2040	1861	1844	1844
q21	5206	4896	4708	4708
q22	583	473	466	466
Total cold run time: 71990 ms
Total hot run time: 54988 ms

seawinde · 2024-04-21T03:12:00Z

run buildall

github-actions · 2024-04-21T03:33:11Z

PR approved by anyone and no changes requested.

doris-robot · 2024-04-21T03:57:23Z

TPC-H: Total hot run time: 38023 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c9a6d246bd5b5635bb788409e2e53b1649e3cb80, data reload: false

------ Round 1 ----------------------------------
q1	17592	4296	4210	4210
q2	2006	194	188	188
q3	10462	1137	1104	1104
q4	10193	811	765	765
q5	7516	2694	2595	2595
q6	223	130	131	130
q7	1005	615	575	575
q8	9241	2065	2016	2016
q9	7361	6531	6480	6480
q10	8511	3502	3581	3502
q11	444	241	238	238
q12	465	217	211	211
q13	17735	2917	2929	2917
q14	259	226	227	226
q15	516	482	486	482
q16	544	374	377	374
q17	955	660	639	639
q18	7243	6703	6672	6672
q19	7259	1539	1475	1475
q20	670	317	304	304
q21	3565	2608	2720	2608
q22	353	312	327	312
Total cold run time: 114118 ms
Total hot run time: 38023 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4322	4191	4195	4191
q2	367	262	276	262
q3	2956	2750	2766	2750
q4	1888	1550	1547	1547
q5	5338	5326	5270	5270
q6	207	121	119	119
q7	2267	1883	1878	1878
q8	3186	3290	3289	3289
q9	8574	8562	8563	8562
q10	4153	3876	4001	3876
q11	592	500	506	500
q12	781	605	621	605
q13	17468	3201	3218	3201
q14	341	296	292	292
q15	528	483	483	483
q16	508	465	450	450
q17	1808	1509	1525	1509
q18	7987	7828	7891	7828
q19	1716	1602	1569	1569
q20	1961	1896	1859	1859
q21	9625	4848	5052	4848
q22	529	464	451	451
Total cold run time: 77102 ms
Total hot run time: 55339 ms

doris-robot · 2024-04-21T04:08:58Z

TPC-DS: Total hot run time: 185388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c9a6d246bd5b5635bb788409e2e53b1649e3cb80, data reload: false

query1	901	366	352	352
query2	6210	2574	2363	2363
query3	6672	203	198	198
query4	23046	21434	21275	21275
query5	4140	407	398	398
query6	270	198	182	182
query7	4590	283	279	279
query8	238	190	187	187
query9	8631	2358	2321	2321
query10	419	239	237	237
query11	14944	14192	14230	14192
query12	136	88	83	83
query13	1629	358	352	352
query14	8755	8288	7942	7942
query15	256	181	186	181
query16	8175	269	259	259
query17	1929	567	561	561
query18	2112	289	284	284
query19	333	156	171	156
query20	93	87	90	87
query21	197	130	121	121
query22	5067	4855	4807	4807
query23	33715	33037	33391	33037
query24	9591	3031	3057	3031
query25	585	385	396	385
query26	699	154	149	149
query27	2372	359	376	359
query28	6125	2099	2075	2075
query29	877	613	596	596
query30	278	186	182	182
query31	984	752	759	752
query32	96	50	54	50
query33	629	251	269	251
query34	914	490	493	490
query35	798	705	717	705
query36	1082	954	917	917
query37	101	72	68	68
query38	3503	3308	3317	3308
query39	1608	1575	1565	1565
query40	180	126	129	126
query41	45	44	41	41
query42	106	100	95	95
query43	598	562	557	557
query44	1114	758	748	748
query45	301	272	266	266
query46	1098	739	717	717
query47	2008	1948	2001	1948
query48	361	305	289	289
query49	874	388	400	388
query50	776	397	389	389
query51	6869	6803	6747	6747
query52	107	85	86	85
query53	342	273	269	269
query54	284	221	241	221
query55	81	72	75	72
query56	242	229	229	229
query57	1244	1117	1128	1117
query58	228	187	195	187
query59	3329	3078	3136	3078
query60	251	222	250	222
query61	85	83	82	82
query62	578	432	422	422
query63	301	279	278	278
query64	4522	3857	3813	3813
query65	3089	3030	3011	3011
query66	758	320	330	320
query67	15369	15028	15232	15028
query68	6077	533	525	525
query69	523	295	291	291
query70	1231	1193	1172	1172
query71	1435	1262	1261	1261
query72	6471	2663	2418	2418
query73	721	313	317	313
query74	6825	6434	6466	6434
query75	3476	2651	2594	2594
query76	3658	963	950	950
query77	583	257	257	257
query78	10993	10173	10173	10173
query79	6744	521	521	521
query80	2068	426	434	426
query81	508	235	239	235
query82	1615	97	92	92
query83	317	169	161	161
query84	254	84	79	79
query85	1308	274	258	258
query86	478	309	303	303
query87	3444	3268	3280	3268
query88	4838	2313	2323	2313
query89	485	370	368	368
query90	1871	178	177	177
query91	119	95	97	95
query92	53	45	46	45
query93	5878	509	497	497
query94	1105	175	174	174
query95	377	289	287	287
query96	598	257	261	257
query97	3148	2914	2952	2914
query98	229	221	213	213
query99	1228	839	835	835
Total cold run time: 286498 ms
Total hot run time: 185388 ms

doris-robot · 2024-04-21T04:14:30Z

ClickBench: Total hot run time: 30.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c9a6d246bd5b5635bb788409e2e53b1649e3cb80, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.66	0.10	0.10
query5	0.50	0.50	0.51
query6	1.47	0.72	0.72
query7	0.01	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.50	0.51
query10	0.55	0.57	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	0.76	0.77	0.81
query15	0.82	0.79	0.80
query16	0.37	0.36	0.35
query17	0.98	1.02	0.98
query18	0.21	0.22	0.24
query19	1.82	1.73	1.73
query20	0.01	0.01	0.02
query21	15.44	0.68	0.66
query22	4.02	7.17	2.33
query23	18.32	1.35	1.24
query24	1.80	0.26	0.21
query25	0.15	0.09	0.08
query26	0.26	0.16	0.15
query27	0.08	0.08	0.08
query28	13.35	1.01	0.99
query29	12.59	3.25	3.26
query30	0.27	0.06	0.06
query31	2.84	0.38	0.38
query32	3.28	0.46	0.47
query33	2.84	2.81	2.80
query34	16.84	4.48	4.38
query35	4.48	4.50	4.49
query36	0.65	0.47	0.45
query37	0.18	0.16	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.03	0.03
Total cold run time: 108.92 s
Total hot run time: 30.83 s

github-actions · 2024-04-21T05:15:57Z

PR approved by at least one committer and no changes requested.

…is not enough to provide all the data for the query (#33800) When the materialized view is not enough to provide all the data for the query, if the materialized view is increment update by partition. we can union materialized view and origin query to reponse the query. this depends on #33362 such as materialized view def is as following: > CREATE MATERIALIZED VIEW mv_10086 > BUILD IMMEDIATE REFRESH AUTO ON MANUAL > partition by(l_shipdate) > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ('replication_num' = '1') > AS > select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total > from lineitem > left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate > group by > l_shipdate, > o_orderdate, > l_partkey, > l_suppkey; the materialized view data is as following: +------------+-------------+-----------+-----------+-----------+ | l_shipdate | o_orderdate | l_partkey | l_suppkey | sum_total | +------------+-------------+-----------+-----------+-----------+ | 2023-10-18 | 2023-10-18 | 2 | 3 | 109.20 | | 2023-10-17 | 2023-10-17 | 2 | 3 | 99.50 | | 2023-10-19 | 2023-10-19 | 2 | 3 | 99.50 | +------------+-------------+-----------+-----------+-----------+ when we insert data to partition `2023-10-17`, if we run query as following ``` select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total from lineitem left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate group by l_shipdate, o_orderdate, l_partkey, l_suppkey; ``` query rewrite by materialzied view will fail with message `Check partition query used validation fail` if we turn on the switch `SET enable_materialized_view_union_rewrite = true;` default true we run the query above again, it will success and will use union all materialized view and origin query to response the query correctly. the plan is as following: ``` | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | l_shipdate[#52] | | o_orderdate[#53] | | l_partkey[#54] | | l_suppkey[#55] | | sum_total[#56] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 11:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 11 | | UNPARTITIONED | | | | 10:VUNION(756) | | | | | |----9:VAGGREGATE (merge finalize)(753) | | | | output: sum(partial_sum(o_totalprice)[#46])[#51] | | | | group by: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | cardinality=2 | | | | distribute expr lists: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | | | | 8:VEXCHANGE | | | offset: 0 | | | distribute expr lists: l_shipdate[#42] | | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 2 | | | | PARTITION: HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 08 | | HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | 7:VAGGREGATE (update serialize)(747) | | | STREAMING | | | output: partial_sum(o_totalprice[#41])[#46] | | | group by: l_shipdate[#37], o_orderdate[#38], l_partkey[#39], l_suppkey[#40] | | | cardinality=2 | | | distribute expr lists: l_shipdate[#37] | | | | | 6:VHASH JOIN(741) | | | join op: RIGHT OUTER JOIN(PARTITIONED)[] | | | equal join conjunct: (o_orderkey[#21] = l_orderkey[#5]) | | | equal join conjunct: (o_orderdate[#25] = l_shipdate[#15]) | | | runtime filters: RF000[min_max] <- l_orderkey[#5](2/2/2048), RF001[bloom] <- l_orderkey[#5](2/2/2048), RF002[min_max] <- l_shipdate[#15](1/1/2048), RF003[bloom] <- l_shipdate[#15](1/1/2048) | | | cardinality=2 | | | vec output tuple id: 4 | | | output tuple id: 4 | | | vIntermediate tuple ids: 3 | | | hash output slot ids: 6 7 24 25 15 | | | final projections: l_shipdate[#36], o_orderdate[#32], l_partkey[#34], l_suppkey[#35], o_totalprice[#31] | | | final project output tuple id: 4 | | | distribute expr lists: o_orderkey[#21], o_orderdate[#25] | | | distribute expr lists: l_orderkey[#5], l_shipdate[#15] | | | | | |----3:VEXCHANGE | | | offset: 0 | | | distribute expr lists: l_orderkey[#5] | | | | | 5:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 3 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 05 | | HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25] | | | | 4:VOlapScanNode(722) | | TABLE: union_db.orders(orders), PREAGGREGATION: ON | | runtime filters: RF000[min_max] -> o_orderkey[#21], RF001[bloom] -> o_orderkey[#21], RF002[min_max] -> o_orderdate[#25], RF003[bloom] -> o_orderdate[#25] | | partitions=3/3 (p_20231017,p_20231018,p_20231019), tablets=9/9, tabletList=161188,161190,161192 ... | | cardinality=3, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 4 | | | | PARTITION: HASH_PARTITIONED: l_orderkey[#5] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 03 | | HASH_PARTITIONED: l_orderkey[#5], l_shipdate[#15] | | | | 2:VOlapScanNode(729) | | TABLE: union_db.lineitem(lineitem), PREAGGREGATION: ON | | PREDICATES: (l_shipdate[#15] >= '2023-10-17') AND (l_shipdate[#15] < '2023-10-18') | | partitions=1/3 (p_20231017), tablets=3/3, tabletList=161223,161225,161227 | | cardinality=2, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 5 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | RANDOM | | | | 0:VOlapScanNode(718) | | TABLE: union_db.mv_10086(mv_10086), PREAGGREGATION: ON | | partitions=2/3 (p_20231018_20231019,p_20231019_20231020), tablets=4/4, tabletList=161251,161253,161265 ... | | cardinality=2, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | MaterializedView | | MaterializedViewRewriteSuccessAndChose: | | Names: mv_10086 | | MaterializedViewRewriteSuccessButNotChose: | | | | MaterializedViewRewriteFail: | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ```

this is brought by #33800 if mv is partitioned materialzied view, the data will be wrong by using the hited materialized view when the paritions in related base partiton table are deleted, created and so on. this fix the problem. if **SET enable_materialized_view_union_rewrite=true;** this will use the materializd view and make sure the data is corrent if **SET enable_materialized_view_union_rewrite=false;** this will query base table directly to make sure the data is right

seawinde added 6 commits April 21, 2024 10:31

[improvement](mtmv) Support union rewrite when the materialized view …

3b3d65a

…is not enough to provide all the data for the query

fix loop union rewrite

c18416d

fix common expression in query select problem

3d48666

fix self connection failed

a56549c

fix partition mv un find

3a62e48

fix mv output

52d643f

zfr9527 and others added 5 commits April 21, 2024 10:31

[test](mtmv)Add union rewrite mtmv test case

f0da6dd

fix nested loop

6cd014d

[test](mtmv)Add mv nested materialized view test case

0ba9b75

remove useless test

751a1f4

fix regression test

b736ea3

seawinde force-pushed the partition_union_rewrite branch from ae8f035 to b736ea3 Compare April 21, 2024 02:31

fix regression comment

c9a6d24

XieJiann approved these changes Apr 21, 2024

View reviewed changes

github-actions bot added the reviewed label Apr 21, 2024

morrySnow approved these changes Apr 21, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 21, 2024

morrySnow merged commit 06c9fb6 into apache:master Apr 21, 2024
26 of 28 checks passed

seawinde mentioned this pull request May 22, 2024

[fix](mtmv) Fix partition mv rewrite result wrong #35236

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improvement](mtmv) Support union rewrite when the materialized view is not enough to provide all the data for the query #33800

[improvement](mtmv) Support union rewrite when the materialized view is not enough to provide all the data for the query #33800

seawinde commented Apr 17, 2024 •

edited

Loading

doris-robot commented Apr 17, 2024

seawinde commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

seawinde commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

seawinde commented Apr 20, 2024

seawinde commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

seawinde commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

seawinde commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

seawinde commented Apr 21, 2024

doris-robot commented Apr 21, 2024

seawinde commented Apr 21, 2024

github-actions bot commented Apr 21, 2024

doris-robot commented Apr 21, 2024

doris-robot commented Apr 21, 2024

doris-robot commented Apr 21, 2024

github-actions bot commented Apr 21, 2024

[improvement](mtmv) Support union rewrite when the materialized view is not enough to provide all the data for the query #33800

[improvement](mtmv) Support union rewrite when the materialized view is not enough to provide all the data for the query #33800

Conversation

seawinde commented Apr 17, 2024 • edited Loading

Proposed changes

Further comments

doris-robot commented Apr 17, 2024

seawinde commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

seawinde commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

seawinde commented Apr 20, 2024

seawinde commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

seawinde commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

seawinde commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

doris-robot commented Apr 20, 2024

seawinde commented Apr 21, 2024

doris-robot commented Apr 21, 2024

seawinde commented Apr 21, 2024

github-actions bot commented Apr 21, 2024

doris-robot commented Apr 21, 2024

doris-robot commented Apr 21, 2024

doris-robot commented Apr 21, 2024

github-actions bot commented Apr 21, 2024

seawinde commented Apr 17, 2024 •

edited

Loading