Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](fe) Key column varchar length change should't light schema change #28235

Merged
merged 1 commit into from
Dec 12, 2023
Merged

[fix](fe) Key column varchar length change should't light schema change #28235

merged 1 commit into from
Dec 12, 2023

Conversation

SWJTU-ZhangLei
Copy link
Contributor

Proposed changes

Issue Number: close #28226

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

dataroaring
dataroaring previously approved these changes Dec 11, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 11, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit cacafbbeff0f4a47f108f8924f437e7e7f64d9dd, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4651	4481	4486	4481
q2	361	199	157	157
q3	1465	1257	1229	1229
q4	1121	910	888	888
q5	3159	3171	3166	3166
q6	251	127	126	126
q7	996	491	483	483
q8	2201	2202	2222	2202
q9	6765	6659	6693	6659
q10	3236	3235	3279	3235
q11	321	199	194	194
q12	349	208	204	204
q13	4549	3771	3770	3770
q14	244	218	214	214
q15	562	534	523	523
q16	444	381	393	381
q17	996	569	599	569
q18	7767	7394	6987	6987
q19	1532	1396	1434	1396
q20	691	613	300	300
q21	3088	2654	2694	2654
q22	348	279	284	279
Total cold run time: 45097 ms
Total hot run time: 40097 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4403	4384	4369	4369
q2	270	164	174	164
q3	3572	3548	3532	3532
q4	2384	2379	2379	2379
q5	5761	5740	5725	5725
q6	239	120	119	119
q7	2394	1885	1863	1863
q8	3527	3531	3517	3517
q9	9087	9019	9040	9019
q10	3887	3993	3983	3983
q11	494	372	374	372
q12	753	614	604	604
q13	4276	3590	3542	3542
q14	280	242	253	242
q15	575	532	513	513
q16	513	469	467	467
q17	1869	1850	1858	1850
q18	8732	8410	8489	8410
q19	1740	1743	1749	1743
q20	2268	1926	1940	1926
q21	6558	6204	6182	6182
q22	501	432	417	417
Total cold run time: 64083 ms
Total hot run time: 60938 ms

@wm1581066 wm1581066 added the usercase Important user case type label label Dec 11, 2023
@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.43 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17222677923 Bytes

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 11, 2023
@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit a2bce15064e2d26a0e5e9507b447adee8c99617d, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4668	4442	4477	4442
q2	366	149	159	149
q3	1449	1255	1197	1197
q4	1113	924	893	893
q5	3192	3125	3143	3125
q6	256	126	123	123
q7	973	490	472	472
q8	2235	2226	2186	2186
q9	6678	6680	6693	6680
q10	3575	3244	3243	3243
q11	333	205	196	196
q12	355	210	212	210
q13	4575	3804	3831	3804
q14	237	213	220	213
q15	569	524	505	505
q16	437	388	390	388
q17	1015	616	547	547
q18	9139	8008	7038	7038
q19	1517	1261	1470	1261
q20	545	310	1244	310
q21	3109	2685	2650	2650
q22	352	287	290	287
Total cold run time: 46688 ms
Total hot run time: 39919 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4409	4398	4419	4398
q2	265	166	164	164
q3	3543	3530	3539	3530
q4	2384	2364	2364	2364
q5	5724	5751	5737	5737
q6	238	122	121	121
q7	2436	1872	1866	1866
q8	3531	3521	3533	3521
q9	9064	8975	8979	8975
q10	3908	3982	3995	3982
q11	494	386	381	381
q12	767	612	609	609
q13	4303	3523	3562	3523
q14	279	264	247	247
q15	569	519	514	514
q16	501	458	461	458
q17	1882	1869	1853	1853
q18	8659	8221	8688	8221
q19	1728	1769	1760	1760
q20	2247	1945	1956	1945
q21	6532	6182	6165	6165
q22	494	427	426	426
Total cold run time: 63957 ms
Total hot run time: 60760 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.97 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17222659369 Bytes

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 087de8789802fc99ab90c641d7fa5ed2bb7e5aa4, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4687	4476	4460	4460
q2	375	149	162	149
q3	1452	1231	1252	1231
q4	1112	934	895	895
q5	3130	3128	3208	3128
q6	254	128	130	128
q7	1013	496	477	477
q8	2209	2224	2181	2181
q9	6707	6688	6748	6688
q10	3219	3242	3250	3242
q11	317	204	200	200
q12	350	212	215	212
q13	4577	3845	3838	3838
q14	240	215	213	213
q15	565	507	520	507
q16	438	395	391	391
q17	1012	569	590	569
q18	8480	7013	6995	6995
q19	1513	1275	1459	1275
q20	893	307	297	297
q21	3101	2681	2673	2673
q22	358	283	287	283
Total cold run time: 46002 ms
Total hot run time: 40032 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4478	4384	4402	4384
q2	267	165	175	165
q3	3567	3524	3539	3524
q4	2389	2385	2393	2385
q5	5759	5745	5761	5745
q6	239	122	120	120
q7	2437	1880	1895	1880
q8	3542	3524	3533	3524
q9	9087	9021	8998	8998
q10	3906	3972	4000	3972
q11	503	388	392	388
q12	782	615	606	606
q13	4327	3570	3545	3545
q14	287	251	256	251
q15	567	528	532	528
q16	498	451	472	451
q17	1894	1852	1852	1852
q18	8754	8365	8417	8365
q19	1703	1748	1757	1748
q20	2250	1949	1950	1949
q21	6547	6240	6143	6143
q22	520	441	418	418
Total cold run time: 64303 ms
Total hot run time: 60941 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.71 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17219968714 Bytes

yiguolei
yiguolei approved these changes Dec 12, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 12, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@yiguolei yiguolei merged commit 7f9cecb into apache:master Dec 12, 2023
27 of 28 checks passed
xzj7019 pushed a commit to xzj7019/doris that referenced this pull request Dec 13, 2023
@SWJTU-ZhangLei
[fix](fe) Key column varchar length change should't light schema chan…
481cee1 
…ge (apache#28235)
@xiaokang xiaokang added the p0_b label Dec 13, 2023
SWJTU-ZhangLei added a commit to SWJTU-ZhangLei/incubator-doris that referenced this pull request Dec 13, 2023
@SWJTU-ZhangLei
[fix](fe) Key column varchar length change should't light schema chan…
6514aeb 
…ge (apache#28235)
@SWJTU-ZhangLei SWJTU-ZhangLei mentioned this pull request Dec 13, 2023
Merged
xiaokang pushed a commit that referenced this pull request Dec 14, 2023
@SWJTU-ZhangLei
[fix](fe) Key column varchar length change should't light schema chan…
0b32152 
…ge (#28235) (#28359)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@SWJTU-ZhangLei @HappenLee
[fix](fe) Key column varchar length change should't light schema chan…
4381bdc 
…ge (apache#28235)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@w41ter w41ter w41ter approved these changes

@yiguolei yiguolei yiguolei approved these changes

@dataroaring dataroaring dataroaring left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged p0_b reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] (dynamic partition) Cannot assign hash distribution with different distribution col
7 participants
@SWJTU-ZhangLei @doris-robot @w41ter @yiguolei @dataroaring @xiaokang @wm1581066