Skip to content

[optimize](Transaction) Reduce the memory usage of transactions #57084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
0AyanamiRei wants to merge 12 commits into
base: master
Choose a base branch
from
Open

[optimize](Transaction) Reduce the memory usage of transactions #57084

0AyanamiRei wants to merge 12 commits into from
+47 −10

Conversation

@0AyanamiRei
Copy link
Contributor

@0AyanamiRei 0AyanamiRei commented Oct 17, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #27128

Problem Summary:

For a transaction with a status of visible, we only need to retain its label and successful status for EOS deduplication. Therefore, we have further cleared some unnecessary fields in the original code pruneAfterVisible to reduce the memory usage of completed transactions.

According to the dump data analysis from JProfiler, the memory usage of transactions has been reduced to approximately one-tenth of the original.

(Be Careful) after this pr, some SHOW sql will not show any msg for a visible transaction , but error. e.g.

  • SHOW TRANSACTION statements
  • SHOW PROC '/transactions/dbId/txnId/tables'

(TODO) In the future, we will impose certain limits on the number of transaction records for each table.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Oct 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei
Copy link
Contributor Author

0AyanamiRei commented Oct 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 17, 2025

TPC-DS: Total hot run time: 190963 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1cdfdd27d798529415c4f04ad6f0ca28146f11a7, data reload: false

query1	1067	442	403	403
query2	6559	1696	1704	1696
query3	6767	226	225	225
query4	26217	23912	23574	23574
query5	5636	667	494	494
query6	338	252	236	236
query7	4663	507	296	296
query8	317	286	276	276
query9	8723	2592	2595	2592
query10	516	345	277	277
query11	15527	15145	14879	14879
query12	185	128	121	121
query13	1696	578	443	443
query14	12006	9493	9436	9436
query15	271	198	181	181
query16	7849	695	527	527
query17	1620	805	709	709
query18	2203	446	344	344
query19	297	224	190	190
query20	140	133	130	130
query21	216	144	125	125
query22	4594	4770	4637	4637
query23	34826	33568	33821	33568
query24	8815	2492	2487	2487
query25	626	568	471	471
query26	1155	290	163	163
query27	2858	519	385	385
query28	4589	2242	2210	2210
query29	821	624	513	513
query30	311	243	202	202
query31	925	838	787	787
query32	86	85	85	85
query33	587	432	358	358
query34	845	888	539	539
query35	858	861	764	764
query36	953	964	908	908
query37	121	121	85	85
query38	3547	3524	3607	3524
query39	1485	1428	1432	1428
query40	231	126	117	117
query41	60	58	58	58
query42	121	118	109	109
query43	473	466	452	452
query44	1360	838	827	827
query45	184	180	170	170
query46	847	994	632	632
query47	1740	1758	1690	1690
query48	395	430	332	332
query49	764	534	421	421
query50	655	692	426	426
query51	3860	3945	3891	3891
query52	117	111	106	106
query53	241	276	203	203
query54	613	606	530	530
query55	89	93	87	87
query56	325	315	318	315
query57	1185	1171	1140	1140
query58	285	285	312	285
query59	2593	2629	2567	2567
query60	346	344	327	327
query61	158	153	152	152
query62	798	704	693	693
query63	232	200	198	198
query64	4124	1179	855	855
query65	4068	3980	3958	3958
query66	954	425	346	346
query67	15329	15269	15241	15241
query68	9363	896	591	591
query69	514	331	293	293
query70	1343	1288	1226	1226
query71	514	335	308	308
query72	5641	4865	4876	4865
query73	702	580	361	361
query74	8937	9348	8917	8917
query75	4558	3375	2873	2873
query76	5002	1167	755	755
query77	1012	402	316	316
query78	9550	9671	8970	8970
query79	5434	822	582	582
query80	751	597	495	495
query81	488	270	228	228
query82	408	170	134	134
query83	299	269	255	255
query84	314	119	99	99
query85	852	532	427	427
query86	331	322	289	289
query87	3850	3714	3640	3640
query88	2930	2271	2240	2240
query89	451	333	300	300
query90	2047	224	217	217
query91	176	178	134	134
query92	85	74	61	61
query93	3171	972	624	624
query94	693	443	333	333
query95	399	322	323	322
query96	502	574	288	288
query97	2922	2963	2896	2896
query98	237	306	213	213
query99	1453	1430	1315	1315
Total cold run time: 287891 ms
Total hot run time: 190963 ms

@doris-robot
Copy link

doris-robot commented Oct 17, 2025

ClickBench: Total hot run time: 32.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1cdfdd27d798529415c4f04ad6f0ca28146f11a7, data reload: false

query1	0.07	0.05	0.04
query2	0.10	0.05	0.05
query3	0.27	0.12	0.10
query4	1.61	0.13	0.14
query5	0.40	0.39	0.37
query6	1.18	0.80	0.84
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.74	0.68	0.64
query10	0.68	0.73	0.72
query11	0.20	0.13	0.16
query12	0.18	0.18	0.16
query13	0.68	0.66	0.70
query14	0.86	0.91	0.88
query15	0.97	0.94	0.91
query16	0.43	0.43	0.43
query17	1.09	1.13	1.13
query18	0.25	0.26	0.24
query19	1.99	1.97	2.17
query20	0.02	0.02	0.02
query21	15.37	0.96	0.64
query22	0.75	1.24	1.07
query23	14.70	1.44	0.67
query24	6.80	2.10	0.81
query25	0.47	0.29	0.14
query26	0.58	0.18	0.18
query27	0.07	0.06	0.07
query28	10.01	1.47	1.09
query29	12.55	4.01	3.31
query30	0.28	0.15	0.12
query31	2.83	0.64	0.44
query32	3.26	0.61	0.55
query33	3.09	3.20	3.14
query34	16.09	5.53	4.95
query35	4.99	4.98	5.01
query36	0.70	0.56	0.54
query37	0.11	0.10	0.09
query38	0.07	0.06	0.05
query39	0.04	0.04	0.04
query40	0.21	0.18	0.19
query41	0.09	0.04	0.04
query42	0.04	0.04	0.03
query43	0.04	0.04	0.05
Total cold run time: 104.95 s
Total hot run time: 32.61 s

@0AyanamiRei
Copy link
Contributor Author

0AyanamiRei commented Oct 17, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 17, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 17, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (8/8) 🎉
Increment coverage report
Complete coverage report

@0AyanamiRei
Copy link
Contributor Author

0AyanamiRei commented Oct 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

ClickBench: Total hot run time: 32.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3db68692d1edbe81b99c1a60395bde2dff37e49, data reload: false

query1	0.07	0.06	0.06
query2	0.10	0.05	0.06
query3	0.26	0.09	0.09
query4	1.63	0.13	0.13
query5	0.29	0.28	0.29
query6	1.18	0.69	0.68
query7	0.03	0.03	0.03
query8	0.06	0.05	0.06
query9	0.66	0.57	0.58
query10	0.62	0.64	0.61
query11	0.20	0.13	0.14
query12	0.19	0.14	0.14
query13	0.65	0.64	0.63
query14	1.08	1.08	1.07
query15	0.92	0.91	0.92
query16	0.42	0.44	0.44
query17	1.16	1.28	1.16
query18	0.25	0.22	0.21
query19	2.02	2.00	1.91
query20	0.02	0.01	0.02
query21	15.36	1.07	0.68
query22	0.79	1.43	1.03
query23	14.69	1.58	0.75
query24	7.74	0.76	1.05
query25	0.42	0.12	0.15
query26	0.68	0.18	0.15
query27	0.09	0.07	0.07
query28	9.33	1.47	0.97
query29	12.56	4.55	3.75
query30	0.30	0.16	0.12
query31	2.86	0.69	0.44
query32	3.26	0.58	0.49
query33	3.15	3.23	3.29
query34	16.25	5.64	4.99
query35	5.05	4.95	5.10
query36	0.73	0.54	0.53
query37	0.11	0.08	0.08
query38	0.07	0.05	0.06
query39	0.04	0.04	0.04
query40	0.19	0.16	0.15
query41	0.10	0.04	0.04
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 105.68 s
Total hot run time: 32.63 s

@0AyanamiRei
Copy link
Contributor Author

0AyanamiRei commented Oct 22, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 22, 2025

ClickBench: Total hot run time: 28.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f8d80e3c7e25c2e98a51941cad1d87941a861c8, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.06	0.05
query3	0.27	0.09	0.08
query4	1.62	0.12	0.12
query5	0.30	0.27	0.25
query6	1.19	0.66	0.66
query7	0.03	0.03	0.02
query8	0.05	0.04	0.05
query9	0.62	0.54	0.53
query10	0.59	0.59	0.59
query11	0.17	0.11	0.12
query12	0.15	0.12	0.13
query13	0.65	0.61	0.61
query14	1.01	1.02	1.01
query15	0.87	0.86	0.87
query16	0.40	0.40	0.40
query17	1.01	1.04	1.04
query18	0.22	0.21	0.21
query19	1.90	1.79	1.80
query20	0.01	0.01	0.01
query21	15.43	0.18	0.13
query22	5.06	0.08	0.05
query23	15.67	0.26	0.10
query24	2.64	1.21	1.35
query25	0.09	0.07	0.06
query26	0.14	0.13	0.13
query27	0.07	0.06	0.05
query28	5.56	1.16	0.94
query29	12.56	3.99	3.31
query30	0.28	0.13	0.12
query31	2.82	0.60	0.39
query32	3.25	0.55	0.48
query33	3.12	3.11	3.06
query34	15.76	5.19	4.60
query35	4.59	4.56	4.57
query36	0.69	0.51	0.50
query37	0.11	0.08	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.13
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 99.53 s
Total hot run time: 28.6 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 22, 2025

FE UT Coverage Report

Increment line coverage 26.92% (7/26) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@0AyanamiRei @Thearas @doris-robot @hello-stephen