Skip to content

[feature](hive/iceberg)add doris's version in table properties#32774

Merged
morningman merged 1 commit intoapache:masterfrom
wuwenchi:add-doris-for-hive
Mar 26, 2024
Merged

[feature](hive/iceberg)add doris's version in table properties#32774
morningman merged 1 commit intoapache:masterfrom
wuwenchi:add-doris-for-hive

Conversation

@wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Mar 25, 2024

Proposed changes

issue #31442
when create a external table, we can add doris's version in table's properties.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38127 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43ad127af51d6d8e36ca0f2d80a7d22b7492b040, data reload: false

------ Round 1 ----------------------------------
q1	17501	4385	4115	4115
q2	2118	164	152	152
q3	10177	1194	1227	1194
q4	10135	839	779	779
q5	7461	3067	3016	3016
q6	211	125	120	120
q7	1009	594	577	577
q8	9342	2067	2000	2000
q9	7241	6647	6608	6608
q10	8412	3471	3501	3471
q11	447	240	218	218
q12	365	201	196	196
q13	17815	2837	2861	2837
q14	250	203	204	203
q15	501	463	449	449
q16	494	374	372	372
q17	974	597	502	502
q18	7429	6777	6607	6607
q19	2840	1350	1421	1350
q20	535	240	253	240
q21	3587	2838	3066	2838
q22	346	302	283	283
Total cold run time: 109190 ms
Total hot run time: 38127 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4166	4137	4138	4137
q2	328	231	227	227
q3	3021	2871	2849	2849
q4	1866	1603	1583	1583
q5	5456	5486	5483	5483
q6	207	116	117	116
q7	2315	1909	1898	1898
q8	3222	3311	3311	3311
q9	8733	8732	8696	8696
q10	3788	3800	3765	3765
q11	536	450	445	445
q12	703	538	554	538
q13	17119	2855	2827	2827
q14	273	248	251	248
q15	496	450	449	449
q16	464	407	425	407
q17	1728	1508	1476	1476
q18	7769	7376	7528	7376
q19	1610	1506	1553	1506
q20	1907	1747	1701	1701
q21	5016	4683	4608	4608
q22	511	434	437	434
Total cold run time: 71234 ms
Total hot run time: 54080 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183261 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43ad127af51d6d8e36ca0f2d80a7d22b7492b040, data reload: false

query1	916	364	368	364
query2	7396	2008	1940	1940
query3	5159	203	207	203
query4	31701	21235	21203	21203
query5	3785	384	386	384
query6	270	172	161	161
query7	4227	282	289	282
query8	213	173	164	164
query9	8275	2243	2206	2206
query10	568	244	266	244
query11	15547	14361	14311	14311
query12	142	89	88	88
query13	1940	427	422	422
query14	12163	10784	10665	10665
query15	272	205	198	198
query16	7953	260	274	260
query17	1862	574	553	553
query18	2093	286	283	283
query19	283	158	153	153
query20	99	89	90	89
query21	212	126	130	126
query22	5111	4796	4739	4739
query23	32632	31789	31732	31732
query24	10608	2889	2811	2811
query25	716	396	354	354
query26	1451	151	154	151
query27	2870	334	346	334
query28	6972	1827	1808	1808
query29	893	609	585	585
query30	302	148	144	144
query31	939	738	722	722
query32	95	56	54	54
query33	746	245	238	238
query34	1078	469	479	469
query35	794	595	601	595
query36	986	848	859	848
query37	110	73	74	73
query38	3586	3455	3408	3408
query39	1542	1435	1410	1410
query40	251	111	106	106
query41	50	46	45	45
query42	100	88	89	88
query43	488	444	448	444
query44	1213	694	710	694
query45	274	252	260	252
query46	1112	718	708	708
query47	1812	1740	1743	1740
query48	438	350	339	339
query49	1153	326	333	326
query50	737	348	356	348
query51	6686	6670	6556	6556
query52	98	85	80	80
query53	334	265	262	262
query54	292	235	238	235
query55	77	75	76	75
query56	236	228	217	217
query57	1200	1139	1130	1130
query58	234	194	194	194
query59	2667	2462	2604	2462
query60	262	251	268	251
query61	103	101	110	101
query62	651	452	434	434
query63	293	274	264	264
query64	6025	4113	3847	3847
query65	3124	3001	3003	3001
query66	1422	387	375	375
query67	14590	14366	14110	14110
query68	4973	496	525	496
query69	554	382	381	381
query70	1199	1194	1128	1128
query71	437	282	278	278
query72	6310	2898	2716	2716
query73	714	311	301	301
query74	7067	6680	6744	6680
query75	3601	2822	2768	2768
query76	3536	928	885	885
query77	446	257	258	257
query78	10979	10185	10174	10174
query79	8247	509	509	509
query80	1858	378	379	378
query81	530	219	219	219
query82	660	199	193	193
query83	281	147	146	146
query84	289	77	80	77
query85	1719	334	307	307
query86	455	291	290	290
query87	3704	3496	3524	3496
query88	4559	2275	2240	2240
query89	500	352	358	352
query90	1726	172	173	172
query91	176	136	138	136
query92	64	46	46	46
query93	7577	479	463	463
query94	888	168	171	168
query95	409	321	326	321
query96	599	266	266	266
query97	3077	2827	2842	2827
query98	228	210	210	210
query99	1248	923	935	923
Total cold run time: 301039 ms
Total hot run time: 183261 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 43ad127af51d6d8e36ca0f2d80a7d22b7492b040 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38166 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd0a9e4fb6331fc729603a89e748cdbf89f02afc, data reload: false

------ Round 1 ----------------------------------
q1	17692	4203	4096	4096
q2	2108	160	149	149
q3	10769	1183	1214	1183
q4	10237	696	741	696
q5	7488	3022	3000	3000
q6	203	123	123	123
q7	1019	601	567	567
q8	10291	2016	1987	1987
q9	7249	6673	6670	6670
q10	8478	3479	3656	3479
q11	431	230	218	218
q12	452	201	191	191
q13	18579	2887	2865	2865
q14	239	203	207	203
q15	522	462	460	460
q16	503	373	381	373
q17	941	528	583	528
q18	7183	6448	6956	6448
q19	4104	1477	1491	1477
q20	565	265	254	254
q21	3659	2892	2892	2892
q22	364	307	307	307
Total cold run time: 113076 ms
Total hot run time: 38166 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4287	4271	4230	4230
q2	356	240	239	239
q3	3123	2927	2874	2874
q4	1845	1555	1663	1555
q5	5350	5383	5313	5313
q6	194	115	116	115
q7	2231	1869	1823	1823
q8	3115	3279	3247	3247
q9	8715	8675	8659	8659
q10	3767	3734	3774	3734
q11	535	441	437	437
q12	712	562	558	558
q13	16945	2840	2845	2840
q14	276	234	249	234
q15	488	446	455	446
q16	473	407	427	407
q17	1736	1491	1483	1483
q18	7421	7266	6976	6976
q19	1624	1523	1508	1508
q20	1913	1722	1716	1716
q21	4877	4738	4788	4738
q22	520	450	462	450
Total cold run time: 70503 ms
Total hot run time: 53582 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180795 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd0a9e4fb6331fc729603a89e748cdbf89f02afc, data reload: false

query1	947	355	351	351
query2	6552	1943	1807	1807
query3	6709	209	204	204
query4	31601	21176	21277	21176
query5	4304	395	387	387
query6	270	178	168	168
query7	4626	281	294	281
query8	250	177	182	177
query9	9274	2281	2266	2266
query10	548	228	239	228
query11	17155	14126	14277	14126
query12	140	87	83	83
query13	1617	410	405	405
query14	9629	7339	7891	7339
query15	240	196	193	193
query16	8143	260	257	257
query17	1956	601	565	565
query18	2052	295	280	280
query19	265	157	161	157
query20	99	86	86	86
query21	203	137	129	129
query22	5155	4982	4916	4916
query23	33583	32580	32720	32580
query24	10984	2854	2874	2854
query25	624	389	391	389
query26	1200	155	157	155
query27	2974	347	356	347
query28	7689	1849	1835	1835
query29	904	619	645	619
query30	306	155	147	147
query31	968	732	736	732
query32	92	57	57	57
query33	768	250	254	250
query34	1038	472	490	472
query35	818	608	602	602
query36	1021	859	880	859
query37	125	67	66	66
query38	3539	3433	3416	3416
query39	1465	1436	1577	1436
query40	217	127	110	110
query41	57	47	47	47
query42	113	95	92	92
query43	459	445	430	430
query44	1183	716	722	716
query45	279	264	256	256
query46	1118	738	718	718
query47	1945	1882	1908	1882
query48	445	353	353	353
query49	1137	340	333	333
query50	774	369	363	363
query51	6772	6754	6650	6650
query52	112	91	88	88
query53	340	285	279	279
query54	298	225	228	225
query55	81	74	78	74
query56	238	228	221	221
query57	1212	1149	1171	1149
query58	233	202	209	202
query59	2684	2490	2596	2490
query60	256	230	234	230
query61	96	94	90	90
query62	664	442	438	438
query63	300	269	273	269
query64	5263	3980	4023	3980
query65	3055	3031	3036	3031
query66	860	371	352	352
query67	15516	14837	14849	14837
query68	5207	500	513	500
query69	573	363	374	363
query70	1208	1152	1111	1111
query71	417	269	260	260
query72	6486	2743	2613	2613
query73	695	309	309	309
query74	6808	6455	6415	6415
query75	2965	2209	2224	2209
query76	3633	891	862	862
query77	622	261	263	261
query78	10850	10289	10157	10157
query79	7813	529	522	522
query80	1977	394	382	382
query81	556	212	214	212
query82	1122	85	85	85
query83	333	150	146	146
query84	292	83	83	83
query85	2053	389	369	369
query86	493	308	280	280
query87	3671	3501	3513	3501
query88	5162	2274	2276	2274
query89	471	375	367	367
query90	1973	173	174	173
query91	164	135	141	135
query92	57	47	49	47
query93	5928	489	485	485
query94	1254	172	172	172
query95	428	321	325	321
query96	628	264	263	263
query97	2708	2484	2470	2470
query98	224	217	206	206
query99	1199	953	940	940
Total cold run time: 303510 ms
Total hot run time: 180795 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cd0a9e4fb6331fc729603a89e748cdbf89f02afc with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.2 seconds inserted 10000000 Rows, about 495K ops/s

@wuwenchi
Copy link
Contributor Author

run buildall

1 similar comment
@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37776 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9c718c27b662ff4aab004f8a6a2d60d92a56a556, data reload: false

------ Round 1 ----------------------------------
q1	17715	4146	4111	4111
q2	2113	174	156	156
q3	10563	1171	1180	1171
q4	10225	777	746	746
q5	7463	3007	2949	2949
q6	201	125	122	122
q7	1012	575	555	555
q8	9338	1974	1956	1956
q9	7189	6634	6605	6605
q10	8471	3498	3581	3498
q11	428	218	215	215
q12	392	196	192	192
q13	17802	2852	2865	2852
q14	231	201	202	201
q15	505	457	456	456
q16	500	369	369	369
q17	935	605	570	570
q18	6935	6413	6304	6304
q19	2941	1389	1403	1389
q20	543	247	244	244
q21	3487	2833	2920	2833
q22	338	296	282	282
Total cold run time: 109327 ms
Total hot run time: 37776 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4178	4075	4050	4050
q2	331	237	227	227
q3	2954	2830	2836	2830
q4	1830	1534	1538	1534
q5	5282	5314	5288	5288
q6	212	116	117	116
q7	2246	1840	1845	1840
q8	3175	3417	3268	3268
q9	8671	8661	8638	8638
q10	3795	3764	3800	3764
q11	533	455	443	443
q12	754	558	569	558
q13	16932	2828	2834	2828
q14	283	253	258	253
q15	486	455	457	455
q16	474	428	428	428
q17	1712	1474	1486	1474
q18	7361	7259	7128	7128
q19	1590	1526	1505	1505
q20	1889	1748	1697	1697
q21	4809	4631	4615	4615
q22	547	471	450	450
Total cold run time: 70044 ms
Total hot run time: 53389 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181253 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9c718c27b662ff4aab004f8a6a2d60d92a56a556, data reload: false

query1	936	363	357	357
query2	6523	1978	1841	1841
query3	6707	215	214	214
query4	31877	21276	21409	21276
query5	4350	389	388	388
query6	269	190	170	170
query7	4639	297	291	291
query8	232	171	171	171
query9	9618	2402	2340	2340
query10	550	250	248	248
query11	15305	14239	14166	14166
query12	129	92	83	83
query13	1653	417	406	406
query14	9836	7925	7064	7064
query15	247	191	191	191
query16	8219	275	265	265
query17	1944	563	553	553
query18	2100	307	282	282
query19	351	157	156	156
query20	101	87	88	87
query21	203	129	124	124
query22	5069	4950	4878	4878
query23	33483	32750	32772	32750
query24	10809	2887	2837	2837
query25	638	386	388	386
query26	1184	155	157	155
query27	2730	349	351	349
query28	7596	1914	1890	1890
query29	878	645	637	637
query30	303	152	152	152
query31	982	751	716	716
query32	91	58	56	56
query33	773	252	259	252
query34	1042	479	493	479
query35	836	617	615	615
query36	1001	860	895	860
query37	119	65	67	65
query38	3565	3454	3392	3392
query39	1483	1445	1428	1428
query40	209	114	113	113
query41	50	48	47	47
query42	105	98	93	93
query43	484	442	442	442
query44	1150	716	706	706
query45	287	261	264	261
query46	1120	688	732	688
query47	1932	1859	1882	1859
query48	483	352	364	352
query49	1098	317	327	317
query50	753	373	369	369
query51	6822	6790	6789	6789
query52	101	91	90	90
query53	343	273	272	272
query54	293	222	223	222
query55	85	80	79	79
query56	241	220	217	217
query57	1197	1125	1141	1125
query58	224	199	202	199
query59	2623	2545	2554	2545
query60	258	230	237	230
query61	95	97	96	96
query62	703	444	468	444
query63	296	269	274	269
query64	5256	4019	4009	4009
query65	3049	3009	3035	3009
query66	847	345	357	345
query67	15155	14745	14770	14745
query68	5971	524	523	523
query69	606	372	390	372
query70	1307	1202	1229	1202
query71	476	255	261	255
query72	6619	2741	2650	2650
query73	719	328	327	327
query74	8130	6393	6417	6393
query75	3282	2220	2227	2220
query76	4285	881	847	847
query77	626	265	266	265
query78	10958	10241	10136	10136
query79	7286	507	509	507
query80	1676	394	397	394
query81	535	220	226	220
query82	1651	87	90	87
query83	241	153	150	150
query84	288	83	84	83
query85	1606	371	340	340
query86	493	292	267	267
query87	3729	3547	3520	3520
query88	5007	2401	2393	2393
query89	487	366	371	366
query90	1997	175	174	174
query91	176	139	134	134
query92	67	48	47	47
query93	5860	491	485	485
query94	1166	177	175	175
query95	441	343	333	333
query96	603	276	271	271
query97	2657	2488	2501	2488
query98	224	215	204	204
query99	1194	936	923	923
Total cold run time: 303993 ms
Total hot run time: 181253 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9c718c27b662ff4aab004f8a6a2d60d92a56a556 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       22.1 seconds inserted 10000000 Rows, about 452K ops/s

Copy link
Contributor

@wsjz wsjz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 26, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit a2b0abe into apache:master Mar 26, 2024
morningman pushed a commit that referenced this pull request Apr 12, 2024
issue #31442
when create a external table, we can add doris's version in table's properties.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants