[feature](cloud) introduce copy into #32759

sollhui · 2024-03-25T07:10:30Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-25T07:10:35Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-03-26T07:50:35Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-26T07:57:23Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-26T08:00:37Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-26T13:37:37Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-26T15:17:27Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-27T03:12:24Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-27T04:05:22Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-27T04:38:59Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-27T08:01:13Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-27T08:02:15Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-27T08:06:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-27T08:07:31Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-27T08:20:56Z

clang-tidy review says "All clean, LGTM! 👍"

sollhui · 2024-03-27T08:27:58Z

run buildall

github-actions · 2024-03-27T08:32:26Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-27T08:45:56Z

TPC-H: Total hot run time: 38262 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8993b9510f33e61b5538ace651ee71f5bf0fbabc, data reload: false

------ Round 1 ----------------------------------
q1	17886	4330	4221	4221
q2	2723	164	160	160
q3	11395	1125	1258	1125
q4	10289	768	763	763
q5	7558	3097	3041	3041
q6	213	132	125	125
q7	1048	595	602	595
q8	9399	2030	1974	1974
q9	7173	6607	6622	6607
q10	8492	3487	3567	3487
q11	430	230	212	212
q12	447	195	191	191
q13	17832	2858	2853	2853
q14	239	200	213	200
q15	505	455	464	455
q16	501	371	372	371
q17	952	530	581	530
q18	7168	6526	6456	6456
q19	1558	1439	1488	1439
q20	549	259	260	259
q21	3628	2960	2902	2902
q22	342	296	328	296
Total cold run time: 110327 ms
Total hot run time: 38262 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4046	4075	4089	4075
q2	327	232	232	232
q3	2955	2846	2848	2846
q4	1823	1562	1568	1562
q5	5279	5299	5321	5299
q6	196	117	116	116
q7	2244	1859	1868	1859
q8	3157	3285	3276	3276
q9	8662	8673	8682	8673
q10	3820	3761	3763	3761
q11	549	460	442	442
q12	736	539	550	539
q13	16911	2842	2838	2838
q14	284	263	254	254
q15	521	466	464	464
q16	472	412	420	412
q17	1708	1491	1458	1458
q18	7388	7136	7043	7043
q19	1613	1475	1545	1475
q20	1899	1739	1719	1719
q21	4831	4671	4638	4638
q22	534	445	465	445
Total cold run time: 69955 ms
Total hot run time: 53426 ms

doris-robot · 2024-03-27T08:54:25Z

TeamCity be ut coverage result:
Function Coverage: 35.25% (8741/24796)
Line Coverage: 27.03% (71545/264660)
Region Coverage: 26.28% (37131/141287)
Branch Coverage: 23.17% (18984/81920)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8993b9510f33e61b5538ace651ee71f5bf0fbabc_8993b9510f33e61b5538ace651ee71f5bf0fbabc/report/index.html

doris-robot · 2024-03-27T08:57:44Z

TPC-DS: Total hot run time: 181677 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8993b9510f33e61b5538ace651ee71f5bf0fbabc, data reload: false

query1	920	385	347	347
query2	6530	1977	1917	1917
query3	6712	212	204	204
query4	31753	21306	21395	21306
query5	4309	401	391	391
query6	272	198	166	166
query7	4633	289	283	283
query8	229	171	178	171
query9	9180	2309	2325	2309
query10	555	240	259	240
query11	17163	14246	14289	14246
query12	141	87	89	87
query13	1626	403	413	403
query14	10222	7910	7326	7326
query15	251	198	189	189
query16	8205	268	279	268
query17	1926	592	558	558
query18	2108	292	284	284
query19	345	154	156	154
query20	96	88	92	88
query21	210	129	128	128
query22	5003	4797	4822	4797
query23	33583	32626	32618	32618
query24	11665	2883	2886	2883
query25	628	367	365	365
query26	1788	155	153	153
query27	3014	356	352	352
query28	7624	1846	1850	1846
query29	1028	606	597	597
query30	307	152	146	146
query31	988	712	741	712
query32	92	54	52	52
query33	761	260	242	242
query34	1068	483	493	483
query35	840	609	594	594
query36	1011	902	879	879
query37	266	63	65	63
query38	3566	3442	3441	3441
query39	1481	1446	1423	1423
query40	283	108	110	108
query41	47	48	45	45
query42	107	95	98	95
query43	504	447	472	447
query44	1268	729	721	721
query45	285	267	268	267
query46	1115	690	720	690
query47	1914	1839	1834	1834
query48	441	362	346	346
query49	1230	335	331	331
query50	764	360	364	360
query51	6763	6653	6664	6653
query52	103	87	86	86
query53	336	273	270	270
query54	310	240	230	230
query55	83	79	77	77
query56	240	223	214	214
query57	1229	1151	1145	1145
query58	233	210	215	210
query59	2772	2670	2641	2641
query60	281	240	248	240
query61	115	113	120	113
query62	662	442	451	442
query63	302	272	273	272
query64	6519	4116	4095	4095
query65	3137	3050	3057	3050
query66	1406	383	373	373
query67	15345	14942	14877	14877
query68	5281	545	514	514
query69	551	378	389	378
query70	1252	1196	1204	1196
query71	393	277	266	266
query72	6284	2932	2617	2617
query73	715	315	314	314
query74	7594	6521	6405	6405
query75	2974	2249	2192	2192
query76	3533	908	933	908
query77	384	258	246	246
query78	10869	10293	10251	10251
query79	8756	519	518	518
query80	1624	382	369	369
query81	521	218	218	218
query82	682	85	92	85
query83	217	152	147	147
query84	292	82	78	78
query85	1331	321	316	316
query86	399	284	307	284
query87	3689	3530	3553	3530
query88	4769	2318	2317	2317
query89	486	375	372	372
query90	2048	177	177	177
query91	169	138	148	138
query92	61	48	47	47
query93	6557	496	498	496
query94	1202	186	178	178
query95	436	342	334	334
query96	607	268	275	268
query97	2666	2489	2470	2470
query98	228	212	211	211
query99	1191	894	898	894
Total cold run time: 307654 ms
Total hot run time: 181677 ms

doris-robot · 2024-03-27T09:06:05Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8993b9510f33e61b5538ace651ee71f5bf0fbabc with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

github-actions · 2024-04-17T02:32:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-17T02:53:43Z

TPC-H: Total hot run time: 38743 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ddf8272a135a339edb2190c27a69757113b0531, data reload: false

------ Round 1 ----------------------------------
q1	17791	4587	4368	4368
q2	2531	199	188	188
q3	11473	1228	1169	1169
q4	11172	859	746	746
q5	7805	2794	2635	2635
q6	217	134	131	131
q7	956	545	538	538
q8	9240	2059	2040	2040
q9	7420	6572	6511	6511
q10	8479	3512	3541	3512
q11	446	228	231	228
q12	424	219	212	212
q13	17771	2918	2949	2918
q14	267	235	242	235
q15	526	488	490	488
q16	507	400	382	382
q17	967	722	688	688
q18	7407	6896	6809	6809
q19	6394	1527	1490	1490
q20	664	333	323	323
q21	3553	2830	2827	2827
q22	366	309	305	305
Total cold run time: 116376 ms
Total hot run time: 38743 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4335	4280	4235	4235
q2	368	278	267	267
q3	3033	2772	2727	2727
q4	1877	1595	1552	1552
q5	5306	5295	5349	5295
q6	212	128	123	123
q7	1829	1412	1420	1412
q8	3206	3367	3363	3363
q9	8613	8560	8554	8554
q10	3905	3713	3679	3679
q11	571	489	477	477
q12	771	590	606	590
q13	16915	2919	2915	2915
q14	309	260	277	260
q15	509	467	469	467
q16	488	420	424	420
q17	1777	1494	1474	1474
q18	7573	7560	7316	7316
q19	1639	1600	1533	1533
q20	1945	1752	1771	1752
q21	4886	4916	4815	4815
q22	544	459	454	454
Total cold run time: 70611 ms
Total hot run time: 53680 ms

doris-robot · 2024-04-17T03:00:17Z

TeamCity be ut coverage result:
Function Coverage: 35.50% (8908/25093)
Line Coverage: 27.22% (73158/268740)
Region Coverage: 26.35% (37834/143559)
Branch Coverage: 23.13% (19276/83344)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6ddf8272a135a339edb2190c27a69757113b0531_6ddf8272a135a339edb2190c27a69757113b0531/report/index.html

doris-robot · 2024-04-17T03:05:23Z

TPC-DS: Total hot run time: 185072 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ddf8272a135a339edb2190c27a69757113b0531, data reload: false

query1	900	372	362	362
query2	6493	2545	2362	2362
query3	6661	209	210	209
query4	23067	21612	21466	21466
query5	4205	412	408	408
query6	272	173	172	172
query7	4583	292	289	289
query8	227	176	176	176
query9	8604	2356	2352	2352
query10	580	267	269	267
query11	14780	14144	14218	14144
query12	133	91	86	86
query13	1636	365	355	355
query14	9512	7571	7018	7018
query15	235	192	182	182
query16	7848	273	259	259
query17	1896	577	544	544
query18	1973	274	271	271
query19	186	146	153	146
query20	92	82	85	82
query21	194	128	128	128
query22	5110	4912	5025	4912
query23	33628	33314	33377	33314
query24	12200	2910	2969	2910
query25	675	363	374	363
query26	1768	157	153	153
query27	3122	308	318	308
query28	7813	2040	2010	2010
query29	1095	614	592	592
query30	304	170	170	170
query31	995	741	733	733
query32	87	50	52	50
query33	746	248	250	248
query34	1104	478	488	478
query35	842	698	690	690
query36	1080	868	922	868
query37	271	72	69	69
query38	3376	3235	3164	3164
query39	1568	1523	1539	1523
query40	280	127	126	126
query41	50	45	45	45
query42	105	96	96	96
query43	613	553	551	551
query44	1246	726	721	721
query45	270	266	262	262
query46	1092	733	723	723
query47	1951	1904	1863	1863
query48	361	294	299	294
query49	1167	370	367	367
query50	749	389	393	389
query51	6736	6619	6645	6619
query52	98	90	93	90
query53	351	284	284	284
query54	313	251	227	227
query55	77	70	71	70
query56	246	247	225	225
query57	1239	1110	1174	1110
query58	229	195	193	193
query59	3415	3105	3168	3105
query60	278	239	239	239
query61	91	117	98	98
query62	661	439	431	431
query63	305	286	279	279
query64	6252	3677	3593	3593
query65	3157	3051	3036	3036
query66	1390	332	328	328
query67	15498	14873	15321	14873
query68	5176	538	531	531
query69	476	307	301	301
query70	1258	1194	1133	1133
query71	1396	1273	1271	1271
query72	6420	2649	2414	2414
query73	720	321	321	321
query74	6905	6509	6405	6405
query75	3343	2642	2680	2642
query76	3347	1028	943	943
query77	418	264	267	264
query78	11014	10345	10244	10244
query79	3229	543	538	538
query80	1793	459	490	459
query81	529	248	239	239
query82	805	95	99	95
query83	265	171	166	166
query84	258	93	84	84
query85	2168	276	270	270
query86	502	293	299	293
query87	3517	3267	3267	3267
query88	4688	2413	2415	2413
query89	483	374	365	365
query90	2025	182	185	182
query91	129	99	99	99
query92	59	48	46	46
query93	5043	517	504	504
query94	1254	180	180	180
query95	1092	1085	1093	1085
query96	599	263	267	263
query97	3160	2930	2941	2930
query98	248	216	215	215
query99	1243	849	840	840
Total cold run time: 291643 ms
Total hot run time: 185072 ms

doris-robot · 2024-04-17T03:10:54Z

ClickBench: Total hot run time: 30.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ddf8272a135a339edb2190c27a69757113b0531, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.04	0.04
query4	1.68	0.07	0.07
query5	0.50	0.50	0.50
query6	1.47	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.55	0.54
query11	0.16	0.11	0.12
query12	0.13	0.11	0.12
query13	0.61	0.59	0.59
query14	0.75	0.79	0.76
query15	0.83	0.81	0.82
query16	0.36	0.38	0.36
query17	1.02	1.03	1.02
query18	0.19	0.24	0.22
query19	1.76	1.78	1.72
query20	0.01	0.01	0.01
query21	15.41	0.64	0.63
query22	4.10	7.30	1.80
query23	18.26	1.31	1.27
query24	1.24	0.44	0.20
query25	0.12	0.08	0.08
query26	0.28	0.17	0.16
query27	0.08	0.08	0.07
query28	13.54	1.00	0.96
query29	12.63	3.30	3.27
query30	0.26	0.07	0.06
query31	2.84	0.38	0.37
query32	3.30	0.46	0.45
query33	2.82	2.92	2.82
query34	17.24	4.43	4.38
query35	4.46	4.46	4.53
query36	0.64	0.47	0.49
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 108.94 s
Total hot run time: 30.22 s

doris-robot · 2024-04-17T03:14:02Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6ddf8272a135a339edb2190c27a69757113b0531 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       12.9 seconds inserted 10000000 Rows, about 775K ops/s

sollhui · 2024-04-24T03:07:07Z

run buildall

github-actions · 2024-04-24T03:12:04Z

clang-tidy review says "All clean, LGTM! 👍"

sollhui · 2024-04-24T04:01:38Z

run buildall

github-actions · 2024-04-24T04:07:00Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-24T04:38:18Z

TeamCity be ut coverage result:
Function Coverage: 35.21% (8916/25322)
Line Coverage: 26.98% (73345/271846)
Region Coverage: 26.15% (37884/144878)
Branch Coverage: 22.97% (19292/83992)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e02470ee34fdd383e9e19e1662ff0137cd96aa4f_e02470ee34fdd383e9e19e1662ff0137cd96aa4f/report/index.html

dataroaring

LGTM

github-actions · 2024-04-24T11:33:42Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-24T11:33:44Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

sollhui · 2024-04-24T11:51:57Z

run buildall

github-actions · 2024-04-24T11:56:23Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-24T12:33:18Z

TeamCity be ut coverage result:
Function Coverage: 35.19% (8915/25332)
Line Coverage: 26.97% (73294/271805)
Region Coverage: 26.14% (37864/144860)
Branch Coverage: 22.97% (19283/83962)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4cc26e67208886dc3464d0601823a482af31bf15_4cc26e67208886dc3464d0601823a482af31bf15/report/index.html

dataroaring

LGTM

## Proposed changes introduce by #32759 when upgrade 2.1 to master, error happened: ``` java.io.EOFException: null at java.io.DataInputStream.readInt(DataInputStream.java:398) ~[?:?] at org.apache.doris.common.io.Text.readString(Text.java:413) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT] at org.apache.doris.load.loadv2.LoadJobFinalOperation.readFields(LoadJobFinalOperation.java:160) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.transaction.TxnCommitAttachment.read(TxnCommitAttachment.java:77) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.journal.JournalEntity.readFields(JournalEntity.java:569) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.journal.bdbje.BDBJournalCursor.next(BDBJournalCursor.java:115) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.replayJournal(Env.java:2863) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.transferToMaster(Env.java:1503) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.access$1300(Env.java:332) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2764) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT] ``` Using a comparison of the FE meta version to solve this problem. 2.1 fe meta version is 129, use the fe version 131 used by [pr](#32759) at that time as the comparison value.

sollhui changed the title ~~[draft](merge-cloud) support stage operation~~ [draft](merge-cloud) support copy into stage operation Mar 25, 2024

sollhui force-pushed the cloud_copy_into branch 2 times, most recently from 85757cf to ba5ba04 Compare March 25, 2024 15:15

sollhui marked this pull request as ready for review March 25, 2024 15:20

sollhui changed the title ~~[draft](merge-cloud) support copy into stage operation~~ [draft](merge-cloud) introduce copy into Mar 26, 2024

sollhui force-pushed the cloud_copy_into branch from 217cc53 to 67ef0ba Compare March 26, 2024 07:52

sollhui force-pushed the cloud_copy_into branch from b35fdbb to ffb6b32 Compare March 27, 2024 04:33

sollhui changed the title ~~[draft](merge-cloud) introduce copy into~~ [feature](merge-cloud) introduce copy into Mar 27, 2024

sollhui force-pushed the cloud_copy_into branch 2 times, most recently from bc17818 to 74d53dd Compare March 27, 2024 08:01

gavinchou approved these changes Apr 23, 2024

View reviewed changes

Merge branch 'master' into cloud_copy_into

199169f

sollhui changed the title ~~[feature](merge-cloud) introduce copy into~~ [feature](cloud) introduce copy into Apr 24, 2024

fix merge error

e02470e

dataroaring approved these changes Apr 24, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 24, 2024

github-actions bot added the reviewed label Apr 24, 2024

dataroaring approved these changes Apr 24, 2024

View reviewed changes

Merge branch 'master' into cloud_copy_into

4cc26e6

dataroaring approved these changes Apr 24, 2024

View reviewed changes

dataroaring merged commit 79890c1 into apache:master Apr 24, 2024
23 of 26 checks passed

sollhui mentioned this pull request Jun 12, 2024

[fix](cloud) fix replay journal fail when FE node upgrade #36156

Merged

[feature](cloud) introduce copy into #32759

[feature](cloud) introduce copy into #32759

Conversation

sollhui commented Mar 25, 2024

Proposed changes

Further comments

doris-robot commented Mar 25, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

sollhui commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

doris-robot commented Mar 27, 2024

doris-robot commented Mar 27, 2024

doris-robot commented Mar 27, 2024

doris-robot commented Mar 27, 2024

github-actions bot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

doris-robot commented Apr 17, 2024

sollhui commented Apr 24, 2024

github-actions bot commented Apr 24, 2024

sollhui commented Apr 24, 2024

github-actions bot commented Apr 24, 2024

doris-robot commented Apr 24, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 24, 2024

github-actions bot commented Apr 24, 2024

dataroaring left a comment

Choose a reason for hiding this comment

sollhui commented Apr 24, 2024

github-actions bot commented Apr 24, 2024

doris-robot commented Apr 24, 2024

dataroaring left a comment

Choose a reason for hiding this comment