Skip to content

[fix](group commit) add debug log show why group commit not work; delete wal when replay success #38611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Aug 1, 2024
Merged

[fix](group commit) add debug log show why group commit not work; delete wal when replay success #38611

dataroaring merged 1 commit into from
Aug 1, 2024

Conversation

@mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Jul 31, 2024

Proposed changes

  1. add debug log to show which condition mismatch, cause group commit insert does not work
  2. delete wal immediately after replay, the original logic is replay all wals and then delete
  3. add a replay wal case

@doris-robot
Copy link

doris-robot commented Jul 31, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi mymeiyi changed the title [fix](group commit) add debug log show why not use group commit for insert; delete wal when replay wal success [fix](group commit) add debug log show why group commit nor work; delete wal when replay success Jul 31, 2024
@mymeiyi mymeiyi force-pushed the debug-group-commit branch from e76f8fa to 06559b5 Compare July 31, 2024 10:59
@github-actions
Copy link
Contributor

github-actions bot commented Jul 31, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jul 31, 2024

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring approved these changes Jul 31, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mymeiyi mymeiyi changed the title [fix](group commit) add debug log show why group commit nor work; delete wal when replay success [fix](group commit) add debug log show why group commit not work; delete wal when replay success Jul 31, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 31, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 31, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 31, 2024

PR approved by anyone and no changes requested.

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Jul 31, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 31, 2024

TPC-H: Total hot run time: 42031 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06559b54007c7f5c211c4fd7ac4ea1ebb1b4bbb7, data reload: false

------ Round 1 ----------------------------------
q1	18421	4157	4163	4157
q2	2530	222	214	214
q3	11745	1345	1411	1345
q4	10680	811	920	811
q5	7758	3131	3012	3012
q6	231	145	139	139
q7	1035	615	608	608
q8	9457	1868	1966	1868
q9	8589	6663	6633	6633
q10	8707	3816	3841	3816
q11	427	264	262	262
q12	417	228	232	228
q13	17756	2929	2953	2929
q14	264	246	246	246
q15	527	487	508	487
q16	496	399	386	386
q17	975	914	935	914
q18	8086	7429	7344	7344
q19	1387	1217	1224	1217
q20	570	318	362	318
q21	5396	4808	4831	4808
q22	362	293	289	289
Total cold run time: 115816 ms
Total hot run time: 42031 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4067	4011	4032	4011
q2	339	229	218	218
q3	3012	2994	2979	2979
q4	1882	1872	1894	1872
q5	5255	5274	5219	5219
q6	219	135	132	132
q7	2062	1675	1705	1675
q8	3219	3308	3307	3307
q9	8336	8325	8292	8292
q10	3758	3890	3823	3823
q11	544	447	456	447
q12	710	543	536	536
q13	11767	2935	2949	2935
q14	285	252	259	252
q15	526	475	481	475
q16	465	401	386	386
q17	1747	1690	1691	1690
q18	7776	7335	7233	7233
q19	1725	1662	1666	1662
q20	1983	1752	1742	1742
q21	5354	5214	5200	5200
q22	510	463	459	459
Total cold run time: 65541 ms
Total hot run time: 54545 ms

@doris-robot
Copy link

doris-robot commented Jul 31, 2024

TPC-DS: Total hot run time: 170204 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06559b54007c7f5c211c4fd7ac4ea1ebb1b4bbb7, data reload: false

query1	912	374	366	366
query2	6460	1701	1670	1670
query3	6660	227	222	222
query4	20497	17203	17266	17203
query5	4303	522	516	516
query6	299	184	163	163
query7	4597	317	299	299
query8	244	197	195	195
query9	8492	2399	2404	2399
query10	446	287	294	287
query11	10586	10082	9969	9969
query12	137	89	88	88
query13	1633	387	377	377
query14	9270	8489	8452	8452
query15	213	164	162	162
query16	7088	469	426	426
query17	949	557	554	554
query18	1868	280	272	272
query19	201	141	143	141
query20	92	83	87	83
query21	207	100	100	100
query22	4164	3957	3938	3938
query23	33819	33003	33042	33003
query24	10304	3126	3040	3040
query25	675	378	383	378
query26	1751	159	146	146
query27	2956	276	271	271
query28	6948	1992	1992	1992
query29	1283	417	434	417
query30	291	147	151	147
query31	946	739	748	739
query32	100	55	56	55
query33	699	311	311	311
query34	908	472	493	472
query35	878	756	742	742
query36	1002	864	865	864
query37	301	80	85	80
query38	2850	2772	2773	2772
query39	857	801	830	801
query40	275	112	116	112
query41	47	48	45	45
query42	127	101	103	101
query43	464	427	408	408
query44	1206	732	744	732
query45	213	176	180	176
query46	1094	840	772	772
query47	1787	1711	1688	1688
query48	366	294	289	289
query49	1187	432	462	432
query50	894	425	440	425
query51	6776	6665	6656	6656
query52	103	92	91	91
query53	269	190	190	190
query54	657	450	457	450
query55	79	79	79	79
query56	279	246	258	246
query57	1127	1045	1041	1041
query58	277	260	254	254
query59	2675	2489	2521	2489
query60	325	279	280	279
query61	99	105	96	96
query62	939	651	656	651
query63	223	190	186	186
query64	5921	1903	1875	1875
query65	3166	3077	3067	3067
query66	1437	340	344	340
query67	15282	14720	14812	14720
query68	4292	563	579	563
query69	444	291	299	291
query70	1100	1079	1073	1073
query71	357	269	273	269
query72	7182	2686	2472	2472
query73	759	334	335	334
query74	6032	5659	5619	5619
query75	3400	2755	2723	2723
query76	2295	1368	1376	1368
query77	448	322	315	315
query78	9500	8901	8803	8803
query79	1325	556	551	551
query80	1055	522	507	507
query81	558	226	221	221
query82	1030	136	128	128
query83	236	174	173	173
query84	271	79	79	79
query85	1266	323	308	308
query86	391	307	292	292
query87	3288	3144	3104	3104
query88	2981	2400	2424	2400
query89	390	292	289	289
query90	1774	196	203	196
query91	139	111	177	111
query92	64	53	51	51
query93	1406	621	617	617
query94	874	291	301	291
query95	372	265	262	262
query96	607	279	285	279
query97	3235	3021	3039	3021
query98	216	202	203	202
query99	1629	1334	1290	1290
Total cold run time: 262096 ms
Total hot run time: 170204 ms

@doris-robot
Copy link

doris-robot commented Jul 31, 2024

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06559b54007c7f5c211c4fd7ac4ea1ebb1b4bbb7, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.69	0.08	0.08
query5	0.49	0.48	0.48
query6	1.13	0.72	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.50	0.50
query10	0.55	0.56	0.56
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.61	0.60
query14	0.77	0.79	0.78
query15	0.90	0.86	0.86
query16	0.36	0.36	0.36
query17	0.98	1.00	0.97
query18	0.22	0.20	0.20
query19	1.81	1.75	1.74
query20	0.01	0.02	0.01
query21	15.43	0.75	0.65
query22	3.94	8.01	1.34
query23	17.99	1.41	1.30
query24	2.24	0.22	0.22
query25	0.19	0.09	0.08
query26	0.31	0.22	0.21
query27	0.47	0.23	0.24
query28	13.18	0.99	0.94
query29	12.52	3.30	3.38
query30	0.26	0.06	0.05
query31	2.85	0.40	0.40
query32	3.24	0.48	0.47
query33	2.89	2.96	2.91
query34	15.46	4.28	4.28
query35	4.30	4.28	4.28
query36	0.68	0.46	0.48
query37	0.21	0.16	0.16
query38	0.17	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.55 s
Total hot run time: 29.99 s

@dataroaring dataroaring merged commit e1b8ee5 into apache:master Aug 1, 2024
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Aug 1, 2024
@mymeiyi
pick [fix](group commit) add debug log show why group commit not work…
b4e39b4 
…; delete wal when replay success (apache#38611)
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Aug 1, 2024
@mymeiyi
pick [fix](group commit) add debug log show why group commit not work…
2d58452 
…; delete wal when replay success (apache#38611)
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Aug 1, 2024
@mymeiyi
[fix](group commit) Pick add debug log show why group commit not work…
0782218 
…; delete wal when replay success (apache#38611)
@mymeiyi mymeiyi mentioned this pull request Aug 1, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 1, 2024
@mymeiyi
[fix](group commit) Pick add debug log show why group commit not work…
e8690b6 
…; delete wal when replay success (#38611) (#38659)

Pick #38611
feiniaofeiafei pushed a commit to feiniaofeiafei/doris that referenced this pull request Aug 9, 2024
@mymeiyi @feiniaofeiafei
[fix](group commit) add debug log show why group commit not work; del…
08a45eb 
…ete wal when replay success (apache#38611)

## Proposed changes

1. add debug log to show which condition mismatch, cause group commit
insert does not work
2. delete wal immediately after replay, the original logic is replay all
wals and then delete
3. add a replay wal case
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@mymeiyi @dataroaring
[fix](group commit) add debug log show why group commit not work; del…
76492b4 
…ete wal when replay success (#38611)

## Proposed changes

1. add debug log to show which condition mismatch, cause group commit
insert does not work
2. delete wal immediately after replay, the original logic is replay all
wals and then delete
3. add a replay wal case
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@mymeiyi @dataroaring
[fix](group commit) add debug log show why group commit not work; del…
1eabe31 
…ete wal when replay success (#38611)

## Proposed changes

1. add debug log to show which condition mismatch, cause group commit
insert does not work
2. delete wal immediately after replay, the original logic is replay all
wals and then delete
3. add a replay wal case
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mymeiyi @doris-robot @Yukang-Lian @dataroaring @wm1581066