Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](Json type) forbit schema change adding JSON columns with none n… #33686

Merged
merged 1 commit into from
Apr 17, 2024
Merged

[Fix](Json type) forbit schema change adding JSON columns with none n… #33686

merged 1 commit into from
Apr 17, 2024

Conversation

eldenmoon
Copy link
Member

…ull default value

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38718 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6954571420fefe09b4e92d5db5c65cd2f6dae7db, data reload: false

------ Round 1 ----------------------------------
q1	17677	4343	4285	4285
q2	2015	192	188	188
q3	10517	1190	1170	1170
q4	10195	766	824	766
q5	7529	2750	2684	2684
q6	221	131	132	131
q7	1038	615	574	574
q8	9226	2057	2049	2049
q9	7380	6584	6612	6584
q10	8589	3540	3492	3492
q11	462	230	243	230
q12	445	219	211	211
q13	17750	2989	2947	2947
q14	262	225	238	225
q15	517	491	480	480
q16	531	396	381	381
q17	972	699	758	699
q18	7406	6801	6736	6736
q19	6846	1534	1505	1505
q20	636	317	310	310
q21	3488	2801	2765	2765
q22	368	308	306	306
Total cold run time: 114070 ms
Total hot run time: 38718 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4807	4223	4203	4203
q2	378	261	259	259
q3	2926	2787	2748	2748
q4	1870	1557	1593	1557
q5	5335	5352	5346	5346
q6	209	123	120	120
q7	2233	1857	1866	1857
q8	3226	3368	3356	3356
q9	8568	8576	8650	8576
q10	4077	3904	4025	3904
q11	596	513	501	501
q12	774	646	625	625
q13	16030	3226	3170	3170
q14	304	291	278	278
q15	532	500	478	478
q16	497	432	449	432
q17	1833	1581	1515	1515
q18	8168	7957	7877	7877
q19	1670	1558	1540	1540
q20	2039	1877	1825	1825
q21	5266	5069	5020	5020
q22	557	465	475	465
Total cold run time: 71895 ms
Total hot run time: 55652 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184504 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6954571420fefe09b4e92d5db5c65cd2f6dae7db, data reload: false

query1	882	378	365	365
query2	6194	2860	2329	2329
query3	6655	211	212	211
query4	22800	21454	21225	21225
query5	4120	401	397	397
query6	283	187	178	178
query7	4582	291	283	283
query8	233	176	182	176
query9	8468	2329	2319	2319
query10	417	255	259	255
query11	14817	14206	14153	14153
query12	138	89	85	85
query13	1621	364	352	352
query14	9785	7402	7465	7402
query15	263	177	179	177
query16	8184	262	251	251
query17	1942	590	542	542
query18	2101	282	259	259
query19	197	147	147	147
query20	90	85	86	85
query21	208	128	123	123
query22	4969	4801	4848	4801
query23	33798	33125	33305	33125
query24	11178	3062	3073	3062
query25	624	379	379	379
query26	843	155	151	151
query27	2385	367	366	366
query28	6231	2079	2085	2079
query29	885	606	600	600
query30	280	184	184	184
query31	1024	781	748	748
query32	94	50	55	50
query33	762	254	249	249
query34	1192	491	497	491
query35	854	734	720	720
query36	1087	932	970	932
query37	135	81	70	70
query38	3466	3395	3305	3305
query39	1653	1584	1576	1576
query40	175	128	125	125
query41	44	43	40	40
query42	107	95	94	94
query43	590	531	548	531
query44	1267	731	726	726
query45	292	268	282	268
query46	1122	768	799	768
query47	2018	1927	2014	1927
query48	380	297	293	293
query49	888	397	384	384
query50	773	384	390	384
query51	6944	6837	6792	6792
query52	107	88	94	88
query53	347	280	282	280
query54	303	230	230	230
query55	77	72	72	72
query56	242	234	233	233
query57	1201	1132	1134	1132
query58	227	205	207	205
query59	3458	3355	3217	3217
query60	260	239	238	238
query61	108	104	107	104
query62	600	436	440	436
query63	309	282	282	282
query64	5131	3825	4057	3825
query65	3049	3061	3017	3017
query66	764	335	337	335
query67	15398	15006	15066	15006
query68	5258	533	530	530
query69	546	307	299	299
query70	1282	1177	1147	1147
query71	462	286	263	263
query72	6910	2758	2566	2566
query73	720	320	316	316
query74	6861	6484	6470	6470
query75	3697	2764	2691	2691
query76	4086	977	973	973
query77	572	267	258	258
query78	11116	10236	10266	10236
query79	11877	518	501	501
query80	2115	430	421	421
query81	528	238	242	238
query82	974	97	93	93
query83	199	162	166	162
query84	261	81	81	81
query85	1368	268	331	268
query86	456	301	270	270
query87	3501	3294	3300	3294
query88	5643	2322	2324	2322
query89	563	368	366	366
query90	1992	178	179	178
query91	127	111	95	95
query92	58	45	45	45
query93	7791	503	494	494
query94	1178	176	174	174
query95	383	296	303	296
query96	603	256	258	256
query97	3151	2911	2958	2911
query98	234	226	219	219
query99	1231	837	868	837
Total cold run time: 297364 ms
Total hot run time: 184504 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6954571420fefe09b4e92d5db5c65cd2f6dae7db, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.09	0.09
query5	0.49	0.50	0.48
query6	1.47	0.74	0.74
query7	0.01	0.01	0.01
query8	0.04	0.04	0.04
query9	0.55	0.48	0.49
query10	0.55	0.56	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.58	0.59
query14	0.76	0.77	0.76
query15	0.83	0.80	0.82
query16	0.35	0.37	0.35
query17	1.00	0.93	0.99
query18	0.21	0.23	0.25
query19	1.79	1.69	1.69
query20	0.02	0.01	0.01
query21	15.40	0.66	0.65
query22	4.06	7.63	1.88
query23	18.29	1.39	1.24
query24	2.04	0.20	0.19
query25	0.15	0.08	0.08
query26	0.26	0.16	0.16
query27	0.08	0.07	0.07
query28	13.33	0.99	0.97
query29	12.61	3.27	3.28
query30	0.26	0.07	0.06
query31	2.86	0.38	0.37
query32	3.28	0.46	0.46
query33	2.77	2.88	2.84
query34	17.09	4.39	4.42
query35	4.45	4.45	4.51
query36	0.65	0.46	0.46
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.17	0.17	0.15
query41	0.08	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.3 s
Total hot run time: 30.23 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6954571420fefe09b4e92d5db5c65cd2f6dae7db with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

dataroaring
dataroaring previously approved these changes Apr 16, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@eldenmoon
Copy link
Member Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 16, 2024
@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38873 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c46e58ab318d6b359bfd89eb93953aa9daa00c7d, data reload: false

------ Round 1 ----------------------------------
q1	17604	4380	4266	4266
q2	2021	193	190	190
q3	10469	1274	1242	1242
q4	10629	764	855	764
q5	7833	2755	2688	2688
q6	221	134	132	132
q7	1041	627	602	602
q8	9444	2132	2108	2108
q9	7545	6770	6574	6574
q10	8546	3564	3542	3542
q11	437	235	228	228
q12	469	221	222	221
q13	17755	2932	2919	2919
q14	268	225	234	225
q15	521	478	459	459
q16	516	376	374	374
q17	962	658	715	658
q18	7300	6763	6717	6717
q19	2463	1527	1514	1514
q20	657	320	309	309
q21	3554	2847	2848	2847
q22	360	294	309	294
Total cold run time: 110615 ms
Total hot run time: 38873 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4247	4229	4214	4214
q2	385	272	262	262
q3	3027	2794	2769	2769
q4	1836	1615	1640	1615
q5	5344	5326	5342	5326
q6	212	122	123	122
q7	2237	1886	1900	1886
q8	3183	3361	3349	3349
q9	8554	8582	8516	8516
q10	3928	3665	3711	3665
q11	598	491	487	487
q12	763	583	564	564
q13	16324	2932	2964	2932
q14	307	271	268	268
q15	504	483	467	467
q16	474	433	422	422
q17	1763	1494	1479	1479
q18	7554	7503	7391	7391
q19	1674	1577	1556	1556
q20	1969	1771	1745	1745
q21	4839	4759	4816	4759
q22	534	451	461	451
Total cold run time: 70256 ms
Total hot run time: 54245 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183395 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c46e58ab318d6b359bfd89eb93953aa9daa00c7d, data reload: false

query1	917	375	364	364
query2	6491	2440	2574	2440
query3	6657	205	210	205
query4	25621	21426	21265	21265
query5	4190	409	408	408
query6	281	173	175	173
query7	4587	289	286	286
query8	231	174	172	172
query9	8508	2326	2305	2305
query10	590	250	247	247
query11	14660	14130	14262	14130
query12	139	91	86	86
query13	1640	352	348	348
query14	9892	7810	7439	7439
query15	270	185	174	174
query16	8162	252	254	252
query17	1947	581	539	539
query18	2108	279	264	264
query19	320	144	151	144
query20	90	87	81	81
query21	199	126	125	125
query22	5051	4887	4969	4887
query23	33741	33237	33049	33049
query24	11623	2936	2927	2927
query25	609	366	377	366
query26	1648	152	150	150
query27	2909	303	304	303
query28	7571	2010	1981	1981
query29	981	591	590	590
query30	298	178	171	171
query31	987	757	750	750
query32	92	55	52	52
query33	739	245	240	240
query34	1026	476	481	476
query35	818	701	698	698
query36	1038	894	902	894
query37	134	68	68	68
query38	3354	3248	3150	3150
query39	1572	1506	1520	1506
query40	279	123	127	123
query41	48	43	44	43
query42	103	96	97	96
query43	617	552	537	537
query44	1199	711	720	711
query45	288	269	276	269
query46	1081	712	751	712
query47	1931	1888	1904	1888
query48	360	291	292	291
query49	1133	378	372	372
query50	759	370	377	370
query51	6705	6592	6630	6592
query52	101	91	91	91
query53	344	291	282	282
query54	315	253	231	231
query55	78	72	70	70
query56	249	235	226	226
query57	1202	1124	1101	1101
query58	223	199	200	199
query59	3453	3140	3183	3140
query60	266	256	241	241
query61	110	107	108	107
query62	647	468	428	428
query63	306	274	279	274
query64	6404	3822	3980	3822
query65	3150	3013	3036	3013
query66	1396	367	340	340
query67	15619	15018	14947	14947
query68	7827	536	536	536
query69	544	319	319	319
query70	1270	1112	1107	1107
query71	481	288	271	271
query72	6862	2748	2549	2549
query73	735	318	316	316
query74	6791	6459	6497	6459
query75	4151	2636	2597	2597
query76	4525	955	983	955
query77	663	293	273	273
query78	11079	10273	10209	10209
query79	10086	516	517	516
query80	1443	451	449	449
query81	527	383	244	244
query82	857	90	93	90
query83	203	163	163	163
query84	272	82	80	80
query85	1362	262	263	262
query86	424	277	300	277
query87	3416	3286	3315	3286
query88	5152	2315	2323	2315
query89	520	363	360	360
query90	1987	178	175	175
query91	125	98	94	94
query92	62	45	47	45
query93	7240	510	492	492
query94	1144	180	177	177
query95	383	299	291	291
query96	604	260	263	260
query97	3112	2946	2928	2928
query98	239	215	216	215
query99	1293	883	864	864
Total cold run time: 305436 ms
Total hot run time: 183395 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c46e58ab318d6b359bfd89eb93953aa9daa00c7d, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.68	0.06	0.08
query5	0.50	0.49	0.49
query6	1.49	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.55	0.50	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.12
query12	0.14	0.11	0.12
query13	0.60	0.58	0.59
query14	0.77	0.79	0.78
query15	0.82	0.81	0.82
query16	0.37	0.36	0.34
query17	0.97	0.96	1.01
query18	0.19	0.28	0.26
query19	1.77	1.68	1.78
query20	0.01	0.02	0.01
query21	15.41	0.66	0.65
query22	4.78	7.57	1.66
query23	18.36	1.24	1.27
query24	1.51	0.28	0.27
query25	0.15	0.08	0.09
query26	0.25	0.16	0.15
query27	0.08	0.08	0.07
query28	13.36	1.01	0.98
query29	12.58	3.30	3.27
query30	0.26	0.06	0.05
query31	2.88	0.38	0.37
query32	3.31	0.46	0.46
query33	2.84	2.80	2.81
query34	17.14	4.40	4.42
query35	4.47	4.50	4.47
query36	0.63	0.47	0.46
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.71 s
Total hot run time: 30.1 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c46e58ab318d6b359bfd89eb93953aa9daa00c7d with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

dataroaring
dataroaring approved these changes Apr 16, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@yiguolei yiguolei merged commit 6aa7665 into apache:master Apr 17, 2024
26 of 29 checks passed
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@eldenmoon @Doris-Extras
[Fix](Json type) forbit schema change adding JSON columns with none n…
223af45 
…ull default value (#33686)
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@eldenmoon @Doris-Extras
[Fix](Json type) forbit schema change adding JSON columns with none n…
c6d1d75 
…ull default value (#33686)
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Apr 17, 2024
@eldenmoon
[Fix](Json type) forbit schema change adding JSON columns with none n…
e745695 
…ull default value (apache#33686)
@eldenmoon eldenmoon mentioned this pull request Apr 17, 2024
Merged
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Apr 17, 2024
@eldenmoon
[Fix](Json type) forbit schema change adding JSON columns with none n…
28be543 
…ull default value (apache#33686)
dataroaring pushed a commit that referenced this pull request Apr 18, 2024
@eldenmoon
[Fix](Json type) forbit schema change adding JSON columns with none n…
d9970be 
…ull default value (#33686) (#33805)
dataroaring pushed a commit that referenced this pull request Apr 24, 2024
@eldenmoon @dataroaring
[Fix](Json type) forbit schema change adding JSON columns with none n…
c322a63 
…ull default value (#33686)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.9-merged p0_c reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@eldenmoon @doris-robot @yiguolei @dataroaring @xiaokang @wm1581066