Skip to content

[fix](compaction) Fix sparse vertical compaction destination offset - #66306

Open
Yukang-Lian wants to merge 1 commit into
apache:masterfrom
Yukang-Lian:agent/fix-sparse-vertical-compaction-master
Open

[fix](compaction) Fix sparse vertical compaction destination offset#66306
Yukang-Lian wants to merge 1 commit into
apache:masterfrom
Yukang-Lian:agent/fix-sparse-vertical-compaction-master

Conversation

@Yukang-Lian

@Yukang-Lian Yukang-Lian commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • Save the original target-column row count before sparse-column preparation.
  • Use the saved row count as the destination offset.
  • Add regression coverage for a reserve-only leading BOOLEAN column followed by a nullable fixed-width BIGINT column in the same value group.

Why

The leading value column may reserve capacity without increasing its size. Deriving the destination offset from its post-prepare size can therefore select the wrong position.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian
Yukang-Lian marked this pull request as ready for review July 31, 2026 02:47
@Yukang-Lian

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29438 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7f914854e931ed2294d4da7717b488a5b63e55c5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17590	4063	4042	4042
q2	2062	319	196	196
q3	10239	1362	811	811
q4	4706	464	339	339
q5	7754	842	565	565
q6	205	173	150	150
q7	757	822	608	608
q8	10028	1501	1565	1501
q9	6095	4287	4293	4287
q10	6921	1751	1474	1474
q11	501	348	331	331
q12	782	573	468	468
q13	18123	3358	2706	2706
q14	280	266	249	249
q15	q16	779	783	709	709
q17	2931	1094	887	887
q18	6989	5838	5480	5480
q19	2056	1344	1085	1085
q20	960	741	605	605
q21	6261	2751	2629	2629
q22	444	376	316	316
Total cold run time: 106463 ms
Total hot run time: 29438 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4936	4748	5040	4748
q2	301	318	226	226
q3	4877	5262	4726	4726
q4	2054	2181	1385	1385
q5	4882	4649	4709	4649
q6	238	182	133	133
q7	1843	1690	1459	1459
q8	2203	1924	1916	1916
q9	7137	7115	7113	7113
q10	4602	4588	4128	4128
q11	527	383	347	347
q12	725	751	523	523
q13	2932	3375	2742	2742
q14	285	277	250	250
q15	q16	675	692	619	619
q17	1270	1238	1241	1238
q18	7272	6944	6783	6783
q19	1100	1061	1071	1061
q20	2213	2196	1924	1924
q21	5232	4553	4399	4399
q22	542	456	424	424
Total cold run time: 55846 ms
Total hot run time: 50793 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 178867 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7f914854e931ed2294d4da7717b488a5b63e55c5, data reload: false

query5	4311	652	504	504
query6	461	223	222	222
query7	4954	580	352	352
query8	335	193	172	172
query9	8761	4163	4164	4163
query10	519	383	303	303
query11	5910	2364	2174	2174
query12	157	108	104	104
query13	1264	604	438	438
query14	6287	5227	4944	4944
query14_1	4626	4380	4308	4308
query15	213	213	181	181
query16	1002	482	417	417
query17	1104	713	581	581
query18	2451	471	339	339
query19	201	181	148	148
query20	110	103	106	103
query21	232	160	142	142
query22	13546	13567	13313	13313
query23	17326	16523	16168	16168
query23_1	16228	16119	16310	16119
query24	7527	1766	1300	1300
query24_1	1297	1299	1308	1299
query25	552	432	364	364
query26	1340	355	214	214
query27	2624	567	386	386
query28	4470	2068	2030	2030
query29	1068	599	460	460
query30	352	263	224	224
query31	1108	1088	1006	1006
query32	110	68	59	59
query33	514	318	246	246
query34	1152	1095	651	651
query35	765	785	669	669
query36	1017	1019	879	879
query37	156	107	94	94
query38	1900	1698	1661	1661
query39	881	875	857	857
query39_1	835	838	838	838
query40	283	164	151	151
query41	65	64	63	63
query42	94	91	90	90
query43	326	327	286	286
query44	1475	794	769	769
query45	198	181	172	172
query46	1059	1160	758	758
query47	2145	2100	1994	1994
query48	382	403	278	278
query49	572	431	312	312
query50	1079	442	346	346
query51	10975	11121	10992	10992
query52	86	87	76	76
query53	259	284	206	206
query54	291	229	220	220
query55	75	71	64	64
query56	308	299	293	293
query57	1313	1249	1195	1195
query58	281	267	260	260
query59	1569	1666	1433	1433
query60	323	285	275	275
query61	157	149	155	149
query62	538	513	439	439
query63	248	206	202	202
query64	2966	1131	872	872
query65	4733	4645	4663	4645
query66	1833	494	384	384
query67	29459	29246	29124	29124
query68	3251	1574	1037	1037
query69	431	322	269	269
query70	927	846	850	846
query71	372	346	315	315
query72	3025	2685	2662	2662
query73	850	812	441	441
query74	5088	4938	4721	4721
query75	2541	2531	2159	2159
query76	2323	1183	825	825
query77	359	407	293	293
query78	11758	11642	11324	11324
query79	1247	1177	722	722
query80	647	576	522	522
query81	455	345	299	299
query82	244	166	132	132
query83	335	334	317	317
query84	285	172	143	143
query85	967	701	595	595
query86	294	247	243	243
query87	1828	1813	1773	1773
query88	3867	2884	2891	2884
query89	422	367	337	337
query90	2155	205	196	196
query91	207	197	161	161
query92	61	58	54	54
query93	1616	1511	951	951
query94	540	348	327	327
query95	794	497	589	497
query96	1080	771	338	338
query97	2597	2594	2480	2480
query98	214	210	197	197
query99	1094	1125	988	988
Total cold run time: 262958 ms
Total hot run time: 178867 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7f914854e931ed2294d4da7717b488a5b63e55c5, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.05	0.05
query3	0.25	0.14	0.14
query4	1.60	0.14	0.14
query5	0.25	0.22	0.23
query6	1.15	0.84	0.79
query7	0.04	0.00	0.01
query8	0.06	0.04	0.04
query9	0.38	0.32	0.31
query10	0.55	0.58	0.57
query11	0.20	0.13	0.14
query12	0.18	0.14	0.13
query13	0.48	0.48	0.50
query14	1.03	1.02	1.02
query15	0.62	0.62	0.61
query16	0.32	0.31	0.34
query17	1.08	1.11	1.10
query18	0.23	0.22	0.23
query19	2.06	1.95	1.97
query20	0.02	0.01	0.01
query21	15.43	0.21	0.15
query22	4.86	0.05	0.06
query23	16.11	0.30	0.13
query24	2.93	0.43	0.30
query25	0.10	0.04	0.04
query26	0.73	0.20	0.16
query27	0.04	0.04	0.03
query28	3.53	0.88	0.58
query29	12.48	4.11	3.29
query30	0.27	0.16	0.16
query31	2.77	0.59	0.31
query32	3.23	0.59	0.49
query33	3.14	3.16	3.33
query34	15.65	4.24	3.55
query35	3.59	3.53	3.52
query36	0.55	0.42	0.44
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.02	0.03
query40	0.18	0.17	0.15
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.04
Total cold run time: 96.54 s
Total hot run time: 24.93 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.70% (31968/42230)
Line Coverage 60.36% (356334/590349)
Region Coverage 56.95% (298884/524854)
Branch Coverage 58.39% (134677/230643)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants