Skip to content

[opt](exprs) Remove unreachable mixed-width decimal registrations for add/subtract/mod - #66617

Merged
morningman merged 5 commits into
apache:masterfrom
morningman:be-build-opt-c-arith-dead-reg
Aug 11, 2026
Merged

[opt](exprs) Remove unreachable mixed-width decimal registrations for add/subtract/mod#66617
morningman merged 5 commits into
apache:masterfrom
morningman:be-build-opt-c-arith-dead-reg

Conversation

@morningman

@morningman morningman commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Split out of #66510. That PR carries the
whole BE build-time batch and its end-to-end measurements — please refer to
#66510 for the complete benefit numbers
. This is the only piece of that batch
that touches the function registration surface, so it is broken out here to
get a proper review from the expression owners. It has no dependency on the
rest of the batch.

What problem does this PR solve?

Related PR: #66510

Problem Summary:

The mixed-width decimal registrations for add / subtract / mod cannot be
reached at runtime, and each one costs a full set of template instantiations in
some of the heaviest TUs in be/src/exprs/function/.

FE casts both children of a decimal Add / Subtract / Mod to exactly the
return type (TypeCoercionUtils#processDecimalV3BinaryArithmetic, since 2.0 via
#17393), so only same-width pairs can ever reach BE.

For add and subtract the registrations were not merely unreachable, they
were already broken: since #52837 the impl carried PTypeB = TypeA, which made
every mixed-width variant register under the same-width factory key, the last
one overwriting the diagonal. Mixed-width add/subtract has therefore been
unresolvable on 4.0 / 4.1 / master for over a year with zero field reports.

That key collapse also had a runtime consequence worth calling out: the entries
that survived on the diagonal were the <Type, DECIMAL256> variants, whose
plain (non-overflow-checked) path promoted Decimal32/64/128 arithmetic to
Int256. Collapsing the impl to a single type parameter puts same-width inputs
back on their natural width.

For mod there was no such typo — its 16 decimal combinations were all live in
the factory — so this is a genuine narrowing of the registered signature set.
It is still safe across the supported upgrade window (old FE + new BE):

  • Nereids has cast both children of Mod to exactly the return type since 2.0
    ([enhancement](Nereids) support decimalv3 type #17393; briefly removed and restored within two days in May 2023, a
    master-only window);
  • the legacy planner's ArithmeticExpr#analyzeDecimalV3Op cast both children
    unconditionally for MOD (only ADD/SUBTRACT had the
    scale-only-comparison hole), and its builtin table only ever registered
    same-width decimal MOD;
  • supported upgrade sources for master (4.0 / 4.1) are Nereids-only — the legacy
    expression analyzer no longer exists there.

A mixed-width lookup now fails loudly with function name, argument types and
return type instead of silently resolving to the wrong instantiation.
Multiply is exempt from the FE cast and keeps its full width cross product.

Why this is a build-time win

Dropping mixed-width removes roughly two thirds of the decimal instantiations of
these TUs — a whole template family per width pair, not a handful of functions.

Measured with the real build's own compile commands (Release, PCH on, the exact
flags build.sh --be produces), recompiling each TU serialized and uncontended,
alternating base/PR across three rounds. multiply.cpp is the control: this PR
deliberately does not touch it, so it must not move.

TU wall before wall after delta peak RSS before after delta
plus.cpp 10.19s 6.06s -40.5% 1518M 1159M -23.7%
minus.cpp 9.94s 5.89s -40.7% 1523M 1162M -23.7%
modulo.cpp 8.46s 5.89s -30.4% 1394M 1151M -17.4%
multiply.cpp (control) 15.95s 15.88s -0.4% 1959M 1961M +0.1%

Run-to-run spread is tight (plus.cpp base 9.93 / 10.19 / 10.24s, after 6.01 /
6.06 / 6.11s) and the control is flat, so these deltas are the change rather than
scheduling noise. The ~350MB drop in peak RSS per TU is worth as much as the wall
time if you build at high -j.

What this does not claim. These are 3 TUs out of 8382 in a cold BE build, and
35.5s out of 7237s of total TU CPU. The ~11s of CPU saved is about 0.15% of a
full build
— real, but well below what a single end-to-end run can resolve, so
I am deliberately not quoting an end-to-end percentage for it. The value of this
PR is the per-TU cost of these three files, the peak-memory drop, the removal of
code that cannot be reached, and the contract test that keeps it gone.

Release note

None

Check List (For Author)

  • Test
    • Regression test — the 56 decimal regression suites match the
      pre-change baseline suite for suite (the only failure is a known,
      unrelated S3-credential outfile case that also fails on the baseline).
    • Unit Test — new BinaryArithmeticRegistrationTest
      (be/test/exprs/function/binary_arithmetic_registration_test.cpp, 5/5
      passing) pins the registration surface: same-width add/subtract/mod
      lookups must resolve, mixed-width ones must return nullptr (there is no
      bare-name fallback), multiply's 4x4 cross product must stay, and DecimalV2
      stays resolvable. If someone re-adds mixed-width registrations — paying
      ~2/3 of those TUs' instantiations for unreachable code — or drops a
      reachable signature, this fails fast.
    • Manual test — microbenchmark A/B, below.

Runtime A/B: no regression on any case

The first commit adds benchmark_binary_arithmetic (13 cases through the real
SimpleFunctionFactory dispatch: multiply int64/same-width/mixed-width/
vector_constant/constant_constant, plus add and subtract in int64 vec_vec,
same-width DECIMAL32 and DECIMAL64 vec_vec, vector_constant and one-row
constant_constant). Two benchmark_test binaries were built from one tree
differing only in be/src/exprs/function/, then run alternately so both share
thermal and scheduling conditions. Figures are the better of two rounds'
5-repetition medians, CPU ns/iteration, 4096-row blocks, macOS arm64 / clang 20
/ Release.

case before after delta
add_d64_d64_const_const 98.1 98.9 +0.8%
subtract_d64_d64_const_const 97.8 96.7 -1.2%
multiply_d64_d64_const_const 107.8 107.5 -0.3%
add_d64_d64_vec_vec 1578.5 1551.0 -1.7%
add_d32_d32_vec_vec 1487.8 1474.4 -0.9%
subtract_d64_d64_vec_vec 1558.9 1537.8 -1.4%
add_d64_d64_vec_const 5877.4 5770.7 -1.8%
add_int64_vec_vec 565.1 545.2 -3.5%
subtract_int64_vec_vec 568.2 550.3 -3.1%
multiply_d64_d64_vec_vec 14816.0 14806.1 -0.1%
multiply_d32_d64_vec_vec 14746.9 14705.4 -0.3%
multiply_d64_d64_vec_const 14673.3 14641.3 -0.2%
multiply_int64_vec_vec 1211.6 1189.2 -1.8%

Every case is inside the +-3.5% round-to-round noise band; nothing regresses.

  • Behavior changed:

    • Yes.
      1. A mixed-width decimal add/subtract/mod lookup now fails
        loudly (Could not find function ... with argument and return types)
        instead of resolving. For add/subtract nothing changes in practice —
        those keys were already unreachable. For mod this is a real
        narrowing, argued safe above; FE never emits such a call.
      2. On the plain, non-overflow-checked branch, same-width decimal
        add/subtract now compute at their natural width instead of being
        promoted to Int256. Results are unchanged; the intermediate width
        is not.
  • Does this need documentation?

    • No.

Proactive disclosure

  • The claimed narrow-width restoration is not visible in the A/B, by
    construction.
    Same-width decimal add/subtract come out flat. The
    <Type, DECIMAL256> promotion sits on the plain, non-overflow-checked
    branch of the kernel, and the benchmark runner pins
    check_overflow_for_decimal to the production default (true), so it never
    executes that branch. The restoration is a code-level fact you can read in the
    diff; it is not something this A/B measured, and I would rather say so than
    let the table imply otherwise.
  • mod is the only place where a signature genuinely disappears. add and
    subtract were already unresolvable. If a reviewer disagrees with the mod
    argument, that is the single hunk to contest, and the contract UT is what
    would need updating.
  • Runtime numbers were measured on macOS arm64 / clang 20 only, on a binary that
    links system malloc rather than tcmalloc (an arm64 branch-range workaround).
    Nothing here is platform-specific — it is registration and template code — but
    the numbers are single-platform.
  • A companion change that removed the hand-written constant_constant paths was
    originally part of this PR and has been taken back out: isolating it showed
    it carried a reproducible +60-70% per-call regression on the constant-folding
    path, while this registration change measured flat. It will be proposed
    separately, on its own evidence.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

morningman and others added 5 commits August 10, 2026 20:31
…ails

Thirteen cases through the real SimpleFunctionFactory dispatch, covering
multiply, add and subtract:

- multiply: int64 vec_vec, Decimal64*Decimal64->Decimal128 (same-width),
  Decimal32*Decimal64->Decimal128 (mixed-width), vector_constant, and
  one-row constant_constant;
- add/subtract: int64 vec_vec (the integral control case), same-width
  Decimal32 and Decimal64 vec_vec, vector_constant, and one-row
  constant_constant.

FE casts both children of a decimal add/subtract to exactly the return
type, so - unlike multiply - only the same-width shapes are reachable
there, and those are exactly the ones the upcoming registration change
rewrites: today they are served by the <Type, DECIMAL256> variants that
won the collapsed factory keys, so they compute in Int256 regardless of
operand width.

These pin rows/s baselines for the template refactors of the arithmetic
kernels: dropping dead mixed-width registrations, removing custom
constant_constant paths, un-templating the bool variants, and the FE
width-unification A/B all compare against these numbers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND7L1ZVTJf91TBpLwYSqct
FE casts both children of a decimal Add to exactly the same type as the
return type (TypeCoercionUtils#processDecimalV3BinaryArithmetic, since 2.0
via apache#17393), so only same-width pairs can reach BE. The 12 mixed-width
registrations were pure dead weight: since apache#52837 the impl carried
"PTypeB = TypeA", which made every mixed-width variant register under the
same-width factory key anyway (last one overwriting the diagonal), so
mixed-width add has been unresolvable on 4.0/4.1/master for over a year
with zero field reports.

Deleting them removes ~2/3 of the template instantiations of this TU
(probe: -31% compile wall, -54% .text on the sibling plus experiment) and
fixes a runtime quirk: the surviving same-width entries were the
<Type, DECIMAL256> variants, whose plain (non-overflow-check) path
promoted Decimal32/64/128 arithmetic to Int256.

Collapse the impl to a single type parameter so mixed-width
instantiations cannot be registered again. A mixed-width lookup now
fails loudly ("Could not find function add, arg ... return ...").

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0121aRtZYjjYdNr2a6z8BLzR
…tions

Same rationale as the add change: FE casts both children of a decimal
Subtract to exactly the return type (TypeCoercionUtils#
processDecimalV3BinaryArithmetic, since 2.0 via apache#17393), so only
same-width pairs reach BE, and since apache#52837 the "PTypeB = TypeA" typo
made all 12 mixed-width registrations collapse onto same-width factory
keys anyway - mixed-width subtract has been unresolvable in production
for over a year with zero field reports.

Also restores narrow-width arithmetic for same-width inputs (the
surviving key winners were the <Type, DECIMAL256> variants) and
collapses the impl to a single type parameter so mixed-width
instantiations cannot come back.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0121aRtZYjjYdNr2a6z8BLzR
Unlike add/subtract, mod's 16 decimal combos were all live in the
factory (no key-collapsing typo here), so this is a real narrowing of
the registered signature set. It is still safe across the supported
upgrade window (old FE + new BE):

- Nereids has cast both children of Mod to exactly the return type
  since 2.0 (TypeCoercionUtils#processDecimalV3BinaryArithmetic,
  apache#17393; briefly removed and restored within two days in May 2023,
  master-only window);
- the legacy planner's ArithmeticExpr#analyzeDecimalV3Op cast both
  children unconditionally for MOD (only ADD/SUBTRACT had the
  scale-only-comparison hole), and its builtin table only ever
  registered same-width decimal MOD;
- supported upgrade sources for master (4.0/4.1) are Nereids-only:
  the legacy expression analyzer no longer exists there.

A mixed-width lookup now fails loudly with function name, argument
types and return type instead of resolving. Impl collapsed to a single
type parameter like add/subtract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0121aRtZYjjYdNr2a6z8BLzR
FE casts both children of Add/Subtract/Mod to exactly the return type
(TypeCoercionUtils#processDecimalV3BinaryArithmetic), so BE registers only
same-width decimal pairs for them; Multiply is exempt and keeps the full
width cross product. These tests pin that surface: same-width add/subtract/
mod lookups must resolve, mixed-width ones must return nullptr (no bare-name
fallback exists), multiply's 4x4 cross product must stay, and DecimalV2
stays resolvable. If someone re-adds mixed-width registrations (paying ~2/3
of those TUs' template instantiations for unreachable code) or drops a
reachable signature, this fails fast.

Co-Authored-By: Claude Fable 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND7L1ZVTJf91TBpLwYSqct
@morningman
morningman force-pushed the be-build-opt-c-arith-dead-reg branch from 9e24fdb to bbd3de2 Compare August 10, 2026 12:33
@morningman morningman changed the title [opt](exprs) Remove unreachable binary-arithmetic registrations and hand-written constant-constant paths [opt](exprs) Remove unreachable mixed-width decimal registrations for add/subtract/mod Aug 10, 2026
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28376 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bbd3de2fa586718db182e2c59d690c123274ddfc, data reload: false

------ Round 1 ----------------------------------
============================================
q1	18430	3883	3864	3864
q2	2426	319	199	199
q3	10997	1365	760	760
q4	4858	458	336	336
q5	8369	815	558	558
q6	179	167	130	130
q7	747	810	608	608
q8	10031	1408	1492	1408
q9	6946	3986	4065	3986
q10	7275	1598	1363	1363
q11	866	341	317	317
q12	1092	566	449	449
q13	19280	3267	2665	2665
q14	252	257	237	237
q15	q16	738	720	657	657
q17	981	995	865	865
q18	6636	5585	5562	5562
q19	1605	1281	1132	1132
q20	821	691	560	560
q21	5540	2513	2425	2425
q22	427	349	295	295
Total cold run time: 108496 ms
Total hot run time: 28376 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4270	4181	4172	4172
q2	276	314	203	203
q3	4539	4868	4376	4376
q4	2164	2238	1429	1429
q5	4192	4294	4209	4209
q6	224	175	124	124
q7	1772	1592	1548	1548
q8	2194	1904	1977	1904
q9	7007	7128	7143	7128
q10	4292	4242	3837	3837
q11	533	394	366	366
q12	698	716	511	511
q13	3135	3419	2874	2874
q14	323	306	274	274
q15	q16	702	702	633	633
q17	1388	1208	1195	1195
q18	12081	11047	11823	11047
q19	1070	1086	1082	1082
q20	2221	2219	1924	1924
q21	5291	4563	4558	4558
q22	520	483	443	443
Total cold run time: 58892 ms
Total hot run time: 53837 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 159597 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bbd3de2fa586718db182e2c59d690c123274ddfc, data reload: false

query5	4347	591	432	432
query6	492	214	245	214
query7	4930	585	335	335
query8	446	163	164	163
query9	8797	4009	4004	4004
query10	500	342	301	301
query11	5865	2155	2058	2058
query12	152	96	93	93
query13	1282	601	414	414
query14	6167	4224	3967	3967
query14_1	3762	3779	3756	3756
query15	204	191	174	174
query16	3263	487	446	446
query17	1040	717	541	541
query18	2192	464	330	330
query19	293	181	143	143
query20	111	101	103	101
query21	983	162	136	136
query22	13917	13658	13549	13549
query23	16133	15315	14856	14856
query23_1	15019	14856	14864	14856
query24	7811	1685	1217	1217
query24_1	1229	1270	1231	1231
query25	563	453	419	419
query26	1362	359	209	209
query27	2523	608	379	379
query28	4611	2006	2002	2002
query29	1073	589	465	465
query30	505	259	217	217
query31	1254	1108	1041	1041
query32	113	59	54	54
query33	549	302	252	252
query34	1311	1160	625	625
query35	715	731	627	627
query36	818	768	705	705
query37	159	106	89	89
query38	1837	1758	1700	1700
query39	844	806	794	794
query39_1	795	781	764	764
query40	316	161	141	141
query41	65	60	60	60
query42	91	89	87	87
query43	324	333	274	274
query44	1416	764	764	764
query45	181	170	163	163
query46	1032	1161	710	710
query47	1567	1604	1476	1476
query48	400	407	313	313
query49	673	392	302	302
query50	1047	429	334	334
query51	10731	10369	10499	10369
query52	92	87	76	76
query53	269	264	200	200
query54	366	234	222	222
query55	73	72	67	67
query56	301	281	271	271
query57	1233	1015	967	967
query58	303	266	254	254
query59	1562	1571	1404	1404
query60	358	277	260	260
query61	151	147	147	147
query62	433	319	264	264
query63	221	184	196	184
query64	2531	1017	849	849
query65	3877	3811	3816	3811
query66	1767	448	348	348
query67	20169	20134	20125	20125
query68	3298	1563	1013	1013
query69	649	302	268	268
query70	870	736	778	736
query71	397	356	330	330
query72	3512	2766	2531	2531
query73	867	806	465	465
query74	4639	4527	4325	4325
query75	2401	2344	2011	2011
query76	1942	1122	760	760
query77	388	376	305	305
query78	11143	11229	10599	10599
query79	1656	1192	746	746
query80	1223	550	455	455
query81	588	336	288	288
query82	743	174	141	141
query83	381	320	299	299
query84	452	156	126	126
query85	1160	611	507	507
query86	462	221	224	221
query87	1986	1965	1824	1824
query88	3759	2766	2733	2733
query89	411	318	285	285
query90	1934	196	191	191
query91	201	186	164	164
query92	61	59	51	51
query93	1766	1504	1001	1001
query94	739	361	316	316
query95	821	588	468	468
query96	1046	797	335	335
query97	2489	2495	2324	2324
query98	200	187	181	181
query99	788	724	621	621
Total cold run time: 254394 ms
Total hot run time: 159597 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bbd3de2fa586718db182e2c59d690c123274ddfc, data reload: false

query1	0.01	0.01	0.01
query2	0.43	0.05	0.05
query3	0.74	0.14	0.14
query4	2.31	0.14	0.14
query5	0.24	0.22	0.22
query6	2.02	0.81	0.81
query7	0.04	0.01	0.00
query8	0.05	0.04	0.04
query9	0.78	0.30	0.31
query10	0.55	0.57	0.54
query11	0.37	0.14	0.14
query12	0.25	0.14	0.14
query13	0.46	0.48	0.46
query14	1.00	0.98	0.99
query15	0.62	0.58	0.58
query16	0.31	0.33	0.30
query17	1.07	1.08	1.03
query18	0.22	0.20	0.20
query19	2.37	1.93	2.11
query20	0.03	0.01	0.01
query21	16.53	0.17	0.14
query22	4.81	0.05	0.05
query23	16.92	0.31	0.13
query24	16.00	0.43	0.39
query25	0.14	0.05	0.04
query26	0.77	0.20	0.14
query27	0.04	0.05	0.04
query28	3.68	0.81	0.37
query29	13.79	4.01	3.28
query30	0.29	0.15	0.15
query31	2.97	0.56	0.31
query32	3.71	0.58	0.50
query33	3.10	3.31	3.14
query34	15.62	3.96	3.31
query35	3.29	3.22	3.25
query36	0.59	0.43	0.42
query37	0.27	0.07	0.06
query38	0.06	0.04	0.04
query39	0.19	0.03	0.03
query40	0.24	0.15	0.15
query41	0.47	0.03	0.03
query42	0.23	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 117.62 s
Total hot run time: 24.02 s

@morningman

Copy link
Copy Markdown
Contributor Author

run check_coverage

@HappenLee HappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
morningman merged commit c29075a into apache:master Aug 11, 2026
32 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants