[Feature](materialized-view) support count(1) on materialized view #28135

BiteTheDDDDt · 2023-12-07T10:46:17Z

Proposed changes

support count(1) on materialized view
fix match failed like select k1, sum(k1) from t group by k1

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

BiteTheDDDDt · 2023-12-07T10:46:36Z

run buildall

github-actions · 2023-12-07T10:54:10Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-12-07T11:50:10Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.92 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17212435846 Bytes

doris-robot · 2023-12-07T12:10:57Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 28cd4f37002655bfc133ad9cc78c8fe730cec1a4, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4704	4405	4469	4405
q2	367	146	158	146
q3	1455	1263	1268	1263
q4	1107	939	924	924
q5	3131	3159	3199	3159
q6	249	133	128	128
q7	1010	493	503	493
q8	2217	2233	2226	2226
q9	6679	6719	6678	6678
q10	3205	3276	3268	3268
q11	321	212	209	209
q12	353	205	207	205
q13	4546	3830	3774	3774
q14	237	208	216	208
q15	559	523	518	518
q16	439	388	395	388
q17	1023	626	627	626
q18	7525	7631	7498	7498
q19	1527	1394	1448	1394
q20	621	525	300	300
q21	3098	2804	2730	2730
q22	359	293	297	293
Total cold run time: 44732 ms
Total hot run time: 40833 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4395	4411	4410	4410
q2	269	164	175	164
q3	3515	3519	3500	3500
q4	2376	2370	2358	2358
q5	5730	5733	5769	5733
q6	241	123	122	122
q7	2389	1867	1890	1867
q8	3526	3518	3531	3518
q9	9043	8999	8966	8966
q10	3912	3990	3990	3990
q11	505	388	377	377
q12	764	602	589	589
q13	4297	3554	3516	3516
q14	282	247	257	247
q15	575	521	523	521
q16	512	454	471	454
q17	1859	1865	1872	1865
q18	8717	8227	8297	8227
q19	1735	1735	1787	1735
q20	2254	1937	1936	1936
q21	6506	6143	6142	6142
q22	502	416	422	416
Total cold run time: 63904 ms
Total hot run time: 60653 ms

BiteTheDDDDt · 2023-12-08T06:55:51Z

run buildall

github-actions · 2023-12-08T07:03:37Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-12-08T08:37:34Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17212080782 Bytes

BiteTheDDDDt · 2023-12-08T08:55:39Z

run buildall

github-actions · 2023-12-08T09:03:10Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-12-08T09:42:02Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.46 seconds
stream load tsv: 592 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17211961410 Bytes

github-actions · 2023-12-08T11:03:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-12-08T11:03:23Z

PR approved by anyone and no changes requested.

doris-robot · 2023-12-08T11:46:41Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit a536d0e656b95130afec8114d9fe088c7b2aad83, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4765	4478	4444	4444
q2	360	138	159	138
q3	1460	1242	1211	1211
q4	1114	950	904	904
q5	3077	3112	3114	3112
q6	244	131	128	128
q7	1004	478	481	478
q8	2194	2217	2168	2168
q9	6645	6616	6658	6616
q10	3207	3256	3252	3252
q11	334	208	203	203
q12	347	209	204	204
q13	4557	3776	3808	3776
q14	242	214	219	214
q15	570	527	528	527
q16	456	388	391	388
q17	1003	568	514	514
q18	7587	6984	7086	6984
q19	1510	1359	1430	1359
q20	509	330	312	312
q21	3030	2622	2610	2610
q22	349	281	281	281
Total cold run time: 44564 ms
Total hot run time: 39823 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4404	4417	4417	4417
q2	269	162	164	162
q3	3519	3529	3505	3505
q4	2373	2370	2370	2370
q5	5736	5719	5701	5701
q6	238	124	128	124
q7	2399	1877	1872	1872
q8	3497	3509	3496	3496
q9	8969	9032	9032	9032
q10	3911	3964	3958	3958
q11	501	404	401	401
q12	805	596	579	579
q13	4277	3604	3553	3553
q14	292	261	254	254
q15	576	525	528	525
q16	491	468	475	468
q17	1864	1861	1826	1826
q18	8695	8183	8278	8183
q19	1718	1718	1748	1718
q20	2246	1944	1942	1942
q21	6510	6183	6181	6181
q22	508	414	427	414
Total cold run time: 63798 ms
Total hot run time: 60681 ms

doris-robot · 2023-12-08T12:59:59Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit ed1b7216bc0b2e7a6e1aaae783049edbcb874a65, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4700	4451	4445	4445
q2	357	162	158	158
q3	1440	1256	1258	1256
q4	1105	930	953	930
q5	3088	3129	3156	3129
q6	253	130	125	125
q7	985	479	483	479
q8	2178	2228	2194	2194
q9	6634	6630	6642	6630
q10	3205	3245	3262	3245
q11	320	218	211	211
q12	356	216	212	212
q13	4550	3768	3808	3768
q14	248	208	216	208
q15	559	523	524	523
q16	446	392	397	392
q17	1004	610	570	570
q18	7388	6966	6846	6846
q19	1527	1388	1402	1388
q20	500	322	309	309
q21	3086	2670	2641	2641
q22	351	282	293	282
Total cold run time: 44280 ms
Total hot run time: 39941 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4410	4385	4364	4364
q2	268	166	174	166
q3	3518	3526	3517	3517
q4	2381	2355	2354	2354
q5	5724	5733	5725	5725
q6	238	119	120	119
q7	2369	1872	1896	1872
q8	3498	3503	3500	3500
q9	8975	9010	8980	8980
q10	3897	3986	4001	3986
q11	496	385	384	384
q12	770	593	593	593
q13	4277	3558	3572	3558
q14	297	262	262	262
q15	573	521	519	519
q16	501	452	456	452
q17	1871	1866	1845	1845
q18	8692	8100	8496	8100
q19	1719	1721	1713	1713
q20	2244	1958	1943	1943
q21	6487	6163	6148	6148
q22	515	427	419	419
Total cold run time: 63720 ms
Total hot run time: 60519 ms

yiguolei

LGTM

…pache#28135) support count(1) on materialized view fix match failed like select k1, sum(k1) from t group by k1

#28135 (#29235)

…28135 (#32839)

support count(1) on materialized view

28cd4f3

BiteTheDDDDt added 2 commits December 8, 2023 12:00

fix

250a899

fix

a536d0e

fix

ed1b721

starocean999 approved these changes Dec 8, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 8, 2023

github-actions bot added the reviewed label Dec 8, 2023

yiguolei approved these changes Dec 8, 2023

View reviewed changes

yiguolei merged commit 027b060 into apache:master Dec 8, 2023
27 of 29 checks passed

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[Feature](materialized-view) support count(1) on materialized view (a…

6ca32c4

…pache#28135) support count(1) on materialized view fix match failed like select k1, sum(k1) from t group by k1

BiteTheDDDDt mentioned this pull request Dec 28, 2023

[Feature](materialized-view) support visitBitmapUnion mv rewrite (#29… #29235

Merged

xiaokang pushed a commit that referenced this pull request Dec 29, 2023

[Feature](materialized-view) support visitBitmapUnion mv rewrite #29200

838371c

#28135 (#29235)

wm1581066 added the dev/2.0.4-merged label Dec 29, 2023

cambyzju mentioned this pull request Jan 5, 2024

[fix](regression-test) after enable check_overflow_for_decimal test_agg_keys_schema_change_decimalv3 failed #29577

Merged

BiteTheDDDDt mentioned this pull request Mar 26, 2024

[Bug](schame-change) fix wrong result after reorder mor table #32839

Merged

xiaokang pushed a commit that referenced this pull request Apr 2, 2024

[Bug](schame-change) fix wrong result after reorder mor table #29045 #…

bce3c22

…28135 (#32839)

cambyzju mentioned this pull request May 11, 2024

[fix](index) should not use light index change for bloom filter index #34732

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature](materialized-view) support count(1) on materialized view #28135

[Feature](materialized-view) support count(1) on materialized view #28135

BiteTheDDDDt commented Dec 7, 2023 •

edited

Loading

BiteTheDDDDt commented Dec 7, 2023

github-actions bot commented Dec 7, 2023

doris-robot commented Dec 7, 2023

doris-robot commented Dec 7, 2023

BiteTheDDDDt commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

doris-robot commented Dec 8, 2023

BiteTheDDDDt commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

doris-robot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

doris-robot commented Dec 8, 2023

doris-robot commented Dec 8, 2023

yiguolei left a comment

[Feature](materialized-view) support count(1) on materialized view #28135

[Feature](materialized-view) support count(1) on materialized view #28135

Conversation

BiteTheDDDDt commented Dec 7, 2023 • edited Loading

Proposed changes

Further comments

BiteTheDDDDt commented Dec 7, 2023

github-actions bot commented Dec 7, 2023

doris-robot commented Dec 7, 2023

doris-robot commented Dec 7, 2023

BiteTheDDDDt commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

doris-robot commented Dec 8, 2023

BiteTheDDDDt commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

doris-robot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

doris-robot commented Dec 8, 2023

doris-robot commented Dec 8, 2023

yiguolei left a comment

Choose a reason for hiding this comment

BiteTheDDDDt commented Dec 7, 2023 •

edited

Loading