Skip to content

[test](variant) Add version-neutral compaction benchmark - #67551

Draft
eldenmoon wants to merge 1 commit into
apache:masterfrom
eldenmoon:codex/variant-compaction-benchmark
Draft

[test](variant) Add version-neutral compaction benchmark#67551
eldenmoon wants to merge 1 commit into
apache:masterfrom
eldenmoon:codex/variant-compaction-benchmark

Conversation

@eldenmoon

@eldenmoon eldenmoon commented Sep 6, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: #66858

Problem Summary: Replace the implementation-specific Variant microbenchmark with a source-version-neutral, high-level cumulative compaction workload. The workload validates output semantics and storage layouts across sparse, whole-Variant, mixed physical placement, and Doc Mode scenarios and records reproducibility metadata for commit-to-commit comparisons.

Release note

None

Check List (For Author)

  • Test: Manual test
    • Release compilation and link of benchmark_test
    • 1,000-row smoke run of all seven cumulative compaction scenarios with five fresh-input samples each
    • build-support/check-format.sh
    • build-support/check-build-hygiene.sh
    • bash -n be/benchmark/run_variant_segment_benchmark.sh
  • Behavior changed: No
  • Does this need documentation: No

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66858

Problem Summary: Replace the implementation-specific Variant microbenchmark with a source-version-neutral, high-level cumulative compaction workload. The workload validates output semantics and storage layouts across sparse, whole-Variant, mixed physical placement, and Doc Mode scenarios and records reproducibility metadata for commit-to-commit comparisons.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - Release compilation and link of `benchmark_test`
    - 1,000-row smoke run of all seven cumulative compaction scenarios with five fresh-input samples each
    - `build-support/check-format.sh`
    - `build-support/check-build-hygiene.sh`
    - `bash -n be/benchmark/run_variant_segment_benchmark.sh`
- Behavior changed: No
- Does this need documentation: No
@eldenmoon
eldenmoon force-pushed the codex/variant-compaction-benchmark branch from 65ed6e1 to f7745cf Compare September 6, 2026 14:06
@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

1 similar comment
@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16841 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f7745cfd74c6d4dc4a0fcf09a1133dad5e59d8cd, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17559	3097	3060	3060
q2	2104	279	223	223
q3	10211	914	502	502
q4	4670	246	203	203
q5	7677	581	393	393
q6	136	115	91	91
q7	521	499	381	381
q8	9227	918	950	918
q9	3442	2328	2348	2328
q10	6502	854	717	717
q11	397	202	190	190
q12	613	259	203	203
q13	18145	1523	1148	1148
q14	159	160	142	142
q15	q16	442	400	366	366
q17	1392	837	856	837
q18	3028	2224	2235	2224
q19	1272	915	723	723
q20	372	277	207	207
q21	5562	1758	1843	1758
q22	338	263	227	227
Total cold run time: 93769 ms
Total hot run time: 16841 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3432	3403	3372	3372
q2	511	382	368	368
q3	2248	2286	2109	2109
q4	1184	1157	887	887
q5	2144	2096	2097	2096
q6	170	116	89	89
q7	1012	949	850	850
q8	1603	1403	1436	1403
q9	3087	3055	3058	3055
q10	1844	1769	1608	1608
q11	360	267	249	249
q12	453	429	334	334
q13	1472	1515	1137	1137
q14	162	175	166	166
q15	q16	385	401	356	356
q17	3588	3340	3191	3191
q18	4788	4388	4717	4388
q19	876	789	900	789
q20	1018	974	826	826
q21	3875	3064	3278	3064
q22	393	351	324	324
Total cold run time: 34605 ms
Total hot run time: 30661 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81648 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f7745cfd74c6d4dc4a0fcf09a1133dad5e59d8cd, data reload: false

query5	4240	405	328	328
query6	378	134	123	123
query7	4984	418	235	235
query8	284	124	114	114
query9	8678	2859	2877	2859
query10	392	232	192	192
query11	5378	1035	909	909
query12	120	71	78	71
query13	1195	451	321	321
query14	6173	2191	2082	2082
query14_1	1950	1956	1955	1955
query15	174	122	110	110
query16	878	389	362	362
query17	797	459	375	375
query18	2320	342	241	241
query19	173	139	114	114
query20	77	70	73	70
query21	201	100	86	86
query22	5393	5404	5376	5376
query23	6840	6226	5934	5934
query23_1	6025	6207	6184	6184
query24	7280	1095	787	787
query24_1	762	747	770	747
query25	432	312	270	270
query26	1229	230	129	129
query27	2799	427	262	262
query28	4658	1482	1493	1482
query29	928	466	354	354
query30	266	155	132	132
query31	813	400	331	331
query32	123	77	73	73
query33	462	216	190	190
query34	1012	822	484	484
query35	407	401	351	351
query36	572	555	540	540
query37	121	83	73	73
query38	998	841	840	840
query39	498	502	459	459
query39_1	467	476	467	467
query40	204	93	78	78
query41	58	55	55	55
query42	77	75	70	70
query43	241	245	211	211
query44	1019	545	588	545
query45	117	102	95	95
query46	774	846	508	508
query47	755	763	703	703
query48	314	310	215	215
query49	534	241	188	188
query50	719	252	195	195
query51	8130	7939	7974	7939
query52	66	74	61	61
query53	193	194	144	144
query54	212	168	150	150
query55	77	58	56	56
query56	184	179	155	155
query57	736	669	661	661
query58	185	169	174	169
query59	1200	1210	1111	1111
query60	226	189	168	168
query61	126	109	137	109
query62	342	201	173	173
query63	170	142	138	138
query64	2806	672	549	549
query65	1587	1569	1546	1546
query66	1850	278	222	222
query67	9739	9588	9569	9569
query68	2827	1208	740	740
query69	334	213	191	191
query70	665	622	648	622
query71	250	171	163	163
query72	2313	1720	1496	1496
query73	672	540	341	341
query74	1837	1207	1133	1133
query75	1163	1076	968	968
query76	2298	736	532	532
query77	251	256	199	199
query78	3908	3562	3127	3127
query79	2870	831	590	590
query80	1630	332	284	284
query81	510	155	130	130
query82	637	125	96	96
query83	275	208	190	190
query84	296	110	91	91
query85	805	367	296	296
query86	466	181	172	172
query87	1014	953	884	884
query88	2786	2104	2120	2104
query89	287	197	173	173
query90	1994	128	127	127
query91	130	120	105	105
query92	92	68	69	68
query93	1432	1041	674	674
query94	630	255	213	213
query95	514	247	286	247
query96	812	544	276	276
query97	1074	1040	1034	1034
query98	166	140	132	132
query99	417	343	309	309
Total cold run time: 177889 ms
Total hot run time: 81648 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f7745cfd74c6d4dc4a0fcf09a1133dad5e59d8cd, data reload: false

query1	0.00	0.00	0.01
query2	0.08	0.04	0.03
query3	0.25	0.11	0.11
query4	1.60	0.10	0.10
query5	0.18	0.16	0.16
query6	1.25	0.71	0.67
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.28	0.21	0.21
query10	0.35	0.36	0.33
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.30	0.30	0.31
query14	0.45	0.44	0.46
query15	0.36	0.35	0.35
query16	0.22	0.22	0.21
query17	0.65	0.70	0.72
query18	0.18	0.15	0.17
query19	1.19	1.14	1.10
query20	0.01	0.00	0.00
query21	15.44	0.15	0.11
query22	5.07	0.04	0.04
query23	16.18	0.26	0.10
query24	2.97	0.30	0.26
query25	0.10	0.05	0.03
query26	0.87	0.16	0.12
query27	0.03	0.03	0.03
query28	3.67	0.56	0.26
query29	12.49	3.20	2.59
query30	0.25	0.12	0.12
query31	2.76	0.37	0.17
query32	3.53	0.32	0.24
query33	1.35	1.36	1.52
query34	15.46	2.15	1.76
query35	1.73	1.71	1.69
query36	0.46	0.30	0.30
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.03	0.03	0.02
query40	0.12	0.08	0.07
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 90.49 s
Total hot run time: 14.52 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 63.03% (29572/46917)
Line Coverage 48.07% (310102/645042)
Region Coverage 43.65% (250479/573851)
Branch Coverage 45.23% (116581/257724)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.29% (34670/45443)
Line Coverage 61.37% (391347/637679)
Region Coverage 57.70% (329827/571667)
Branch Coverage 58.47% (150264/256985)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants