Skip to content

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Jan 2, 2025

What problem does this PR solve?

Issue Number: DORIS-17763
Related PR: #46318
Problem Summary:
It has been reported that too many error logs are generated when load stream stubs encounter errors.
This PR is the second step in addressing the issue by reducing duplicate error reports from downstream to upstream.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Member Author

kaijchen commented Jan 2, 2025

run buildall

@kaijchen
Copy link
Member Author

kaijchen commented Jan 2, 2025

这个PR暂时不打 3.0 和 2.1 的标签,等下个版本发布后再进

@doris-robot
Copy link

TPC-H: Total hot run time: 33618 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf43ffce38193379da560504402a3e63a98e09f8, data reload: false

------ Round 1 ----------------------------------
q1	17690	6231	6100	6100
q2	2050	316	191	191
q3	10924	1217	748	748
q4	10199	876	435	435
q5	7492	2197	1948	1948
q6	207	186	148	148
q7	917	771	603	603
q8	9269	1393	1178	1178
q9	6386	5179	5153	5153
q10	6789	2309	1880	1880
q11	485	286	262	262
q12	343	358	222	222
q13	17978	3760	3053	3053
q14	236	238	220	220
q15	570	533	505	505
q16	649	620	603	603
q17	573	851	344	344
q18	8444	6970	6935	6935
q19	4554	977	562	562
q20	300	306	192	192
q21	2888	2192	2019	2019
q22	368	336	317	317
Total cold run time: 109311 ms
Total hot run time: 33618 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6441	6574	6358	6358
q2	238	325	230	230
q3	2234	2655	2350	2350
q4	1407	1838	1359	1359
q5	4405	5013	4936	4936
q6	196	184	144	144
q7	2084	1967	1834	1834
q8	2612	2830	2661	2661
q9	7307	7232	7280	7232
q10	2970	3307	2856	2856
q11	594	519	505	505
q12	714	793	653	653
q13	3575	3927	3349	3349
q14	299	304	280	280
q15	568	528	515	515
q16	640	687	638	638
q17	1237	1744	1245	1245
q18	7871	7452	7493	7452
q19	874	1200	1126	1126
q20	2004	2023	1894	1894
q21	5648	5309	5098	5098
q22	605	648	593	593
Total cold run time: 54523 ms
Total hot run time: 53308 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196866 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf43ffce38193379da560504402a3e63a98e09f8, data reload: false

query1	1320	989	929	929
query2	6388	2299	2281	2281
query3	11097	4884	5107	4884
query4	33068	23717	23773	23717
query5	4430	607	469	469
query6	265	190	171	171
query7	3988	488	301	301
query8	296	242	232	232
query9	9441	2650	2635	2635
query10	467	302	261	261
query11	18350	15587	15257	15257
query12	163	107	102	102
query13	1562	520	386	386
query14	10275	6757	7639	6757
query15	245	194	203	194
query16	7862	676	467	467
query17	1601	800	628	628
query18	2116	432	331	331
query19	221	185	166	166
query20	143	117	117	117
query21	210	131	110	110
query22	4718	4850	4538	4538
query23	34945	33462	33417	33417
query24	6540	2294	2293	2293
query25	461	451	402	402
query26	999	270	151	151
query27	2821	450	336	336
query28	5655	2465	2438	2438
query29	613	550	436	436
query30	212	179	149	149
query31	963	959	865	865
query32	74	60	55	55
query33	482	353	299	299
query34	769	833	516	516
query35	799	825	758	758
query36	1032	1038	973	973
query37	113	109	73	73
query38	4224	4285	4408	4285
query39	1541	1435	1443	1435
query40	208	117	101	101
query41	46	45	46	45
query42	123	106	102	102
query43	517	515	497	497
query44	1313	835	812	812
query45	191	181	178	178
query46	894	1087	662	662
query47	2037	2042	1945	1945
query48	388	400	323	323
query49	718	483	424	424
query50	631	677	397	397
query51	7274	7280	7228	7228
query52	107	100	93	93
query53	240	258	184	184
query54	470	493	408	408
query55	88	83	95	83
query56	266	257	248	248
query57	1247	1253	1165	1165
query58	234	228	217	217
query59	3213	3351	3189	3189
query60	294	266	243	243
query61	122	108	127	108
query62	886	827	776	776
query63	232	194	196	194
query64	3209	1002	682	682
query65	3387	3196	3267	3196
query66	826	412	305	305
query67	16247	16086	15610	15610
query68	8311	701	538	538
query69	489	289	257	257
query70	1196	1098	1083	1083
query71	430	288	250	250
query72	6547	3799	3942	3799
query73	647	743	351	351
query74	10496	8988	8827	8827
query75	4365	3141	2670	2670
query76	3860	1189	772	772
query77	758	449	283	283
query78	10056	10014	9630	9630
query79	3688	792	580	580
query80	696	507	435	435
query81	486	265	229	229
query82	507	148	120	120
query83	189	170	142	142
query84	284	89	68	68
query85	819	358	300	300
query86	364	308	304	304
query87	4431	4506	4328	4328
query88	4013	2163	2166	2163
query89	409	333	303	303
query90	1927	184	178	178
query91	131	135	103	103
query92	69	57	48	48
query93	1791	856	532	532
query94	661	384	297	297
query95	322	256	259	256
query96	477	608	268	268
query97	2878	3011	2854	2854
query98	215	207	196	196
query99	1725	1518	1473	1473
Total cold run time: 299114 ms
Total hot run time: 196866 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf43ffce38193379da560504402a3e63a98e09f8, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.10	0.10
query5	0.40	0.41	0.41
query6	1.18	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.52
query10	0.56	0.57	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.61	0.59
query14	2.73	2.73	2.86
query15	0.91	0.83	0.85
query16	0.39	0.39	0.38
query17	1.08	1.07	1.04
query18	0.24	0.21	0.21
query19	1.97	1.81	1.98
query20	0.01	0.01	0.01
query21	15.38	0.95	0.58
query22	0.77	0.71	0.65
query23	15.41	1.44	0.62
query24	2.93	1.81	1.84
query25	0.14	0.27	0.15
query26	0.18	0.15	0.14
query27	0.08	0.06	0.04
query28	14.46	1.49	1.06
query29	12.59	3.99	3.26
query30	0.25	0.09	0.07
query31	2.83	0.61	0.38
query32	3.23	0.54	0.46
query33	3.26	3.16	3.18
query34	16.60	5.14	4.57
query35	4.50	4.48	4.57
query36	0.69	0.50	0.51
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.77 s
Total hot run time: 32.42 s

@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32384 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e5c3d035c89f16c4e7942b17e776a56ac05e3cba, data reload: false

------ Round 1 ----------------------------------
q1	17581	5674	5358	5358
q2	2040	312	170	170
q3	10429	1201	725	725
q4	10246	955	503	503
q5	8312	2353	2135	2135
q6	192	161	130	130
q7	919	753	609	609
q8	9250	1351	1198	1198
q9	5231	4901	4887	4887
q10	6862	2358	1881	1881
q11	485	282	258	258
q12	340	362	219	219
q13	17764	3675	3074	3074
q14	234	228	201	201
q15	513	474	479	474
q16	642	635	612	612
q17	577	861	338	338
q18	6914	6664	6489	6489
q19	2795	980	565	565
q20	309	320	193	193
q21	2810	2259	2047	2047
q22	387	348	318	318
Total cold run time: 104832 ms
Total hot run time: 32384 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5605	5489	5397	5397
q2	251	329	238	238
q3	2356	2649	2288	2288
q4	1432	1793	1345	1345
q5	4358	4744	4710	4710
q6	168	166	129	129
q7	2060	1976	1817	1817
q8	2645	2809	2697	2697
q9	7305	7191	7249	7191
q10	2989	3297	2789	2789
q11	560	502	507	502
q12	665	759	583	583
q13	3530	3957	3291	3291
q14	287	306	281	281
q15	512	497	464	464
q16	666	683	632	632
q17	1242	1731	1255	1255
q18	7774	7437	7289	7289
q19	837	1173	1059	1059
q20	1996	2058	1928	1928
q21	5866	5185	4920	4920
q22	579	619	600	600
Total cold run time: 53683 ms
Total hot run time: 51405 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194397 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e5c3d035c89f16c4e7942b17e776a56ac05e3cba, data reload: false

query1	1300	977	924	924
query2	6267	2030	2062	2030
query3	10948	4459	4515	4459
query4	60727	35473	23270	23270
query5	5356	622	459	459
query6	333	189	199	189
query7	4950	504	297	297
query8	290	242	224	224
query9	6452	2684	2674	2674
query10	414	297	249	249
query11	15362	15230	15482	15230
query12	163	112	112	112
query13	1043	552	423	423
query14	11334	7494	7007	7007
query15	199	203	192	192
query16	7127	617	484	484
query17	1173	744	625	625
query18	1580	409	296	296
query19	200	187	156	156
query20	117	117	111	111
query21	210	120	100	100
query22	4354	4501	4448	4448
query23	34175	33411	33198	33198
query24	5600	2331	2260	2260
query25	486	456	388	388
query26	637	301	158	158
query27	1857	489	324	324
query28	3927	2498	2488	2488
query29	560	582	435	435
query30	216	201	159	159
query31	947	865	811	811
query32	65	60	55	55
query33	452	365	319	319
query34	770	868	528	528
query35	816	833	757	757
query36	994	1028	948	948
query37	123	99	76	76
query38	4230	4242	4278	4242
query39	1484	1441	1457	1441
query40	207	113	102	102
query41	53	51	49	49
query42	123	101	103	101
query43	518	522	470	470
query44	1306	845	805	805
query45	184	170	169	169
query46	892	1064	661	661
query47	1904	1884	1851	1851
query48	390	413	328	328
query49	698	510	405	405
query50	670	684	405	405
query51	7149	6986	6926	6926
query52	103	100	106	100
query53	233	255	179	179
query54	495	521	436	436
query55	87	84	86	84
query56	283	274	260	260
query57	1184	1239	1163	1163
query58	261	249	262	249
query59	3081	3010	2923	2923
query60	283	286	277	277
query61	142	141	138	138
query62	772	715	641	641
query63	222	178	183	178
query64	1153	1013	670	670
query65	3250	3141	3171	3141
query66	710	400	305	305
query67	16062	15575	15350	15350
query68	5073	830	526	526
query69	513	296	250	250
query70	1169	1140	1135	1135
query71	424	303	269	269
query72	6190	3819	3886	3819
query73	908	763	370	370
query74	10249	8943	8880	8880
query75	3239	3185	2704	2704
query76	3720	1174	778	778
query77	498	364	274	274
query78	10163	9929	9392	9392
query79	3239	806	590	590
query80	1511	529	456	456
query81	563	271	250	250
query82	460	148	118	118
query83	275	166	159	159
query84	287	94	74	74
query85	812	363	307	307
query86	452	325	305	305
query87	4584	4519	4330	4330
query88	4559	2216	2206	2206
query89	404	330	295	295
query90	1537	192	192	192
query91	138	149	115	115
query92	68	55	55	55
query93	2302	865	544	544
query94	774	411	302	302
query95	338	265	274	265
query96	483	616	278	278
query97	2768	2914	2783	2783
query98	245	198	194	194
query99	1279	1380	1270	1270
Total cold run time: 308877 ms
Total hot run time: 194397 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e5c3d035c89f16c4e7942b17e776a56ac05e3cba, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.42	0.43	0.41
query6	1.14	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.51	0.52
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.86	2.75	2.75
query15	0.91	0.82	0.83
query16	0.37	0.38	0.39
query17	1.05	1.03	1.00
query18	0.23	0.20	0.21
query19	1.96	1.89	2.03
query20	0.01	0.01	0.02
query21	15.36	0.98	0.60
query22	0.79	0.84	0.68
query23	15.21	1.42	0.56
query24	3.00	1.32	1.26
query25	0.20	0.22	0.06
query26	0.21	0.15	0.14
query27	0.06	0.06	0.05
query28	14.16	1.04	0.43
query29	12.60	3.95	3.23
query30	0.24	0.08	0.07
query31	2.83	0.60	0.38
query32	3.24	0.55	0.47
query33	2.97	3.02	3.05
query34	16.54	5.18	4.52
query35	4.54	4.48	4.49
query36	0.67	0.50	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.14 s
Total hot run time: 30.97 s

@kaijchen kaijchen closed this Mar 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants