[opt](build) Cut three more waves of hot include edges in the BE header graph by morningman · Pull Request #66672 · apache/doris

morningman · 2026-08-11T16:21:49Z

Part of the BE build-time optimization series tracked in #66715.

Split out of #66510, which carries the whole
BE build-time batch. This PR continues the header-closure surgery merged as
#66400 with three more waves, and is independent of the rest of that batch —
it can be reviewed and merged on its own.

What problem does this PR solve?

Related PR: #66400, #66510

Problem Summary:

Three waves of include-edge surgery in the BE header graph, each in the same
"seed first, then cut" shape as #66400: one purely additive commit that outlines
the inline bodies forcing the instantiation and seeds the direct includes the cut
will expose, then one commit that deletes the edges and pins them with new
build-support/check-header-deps.py rules.

Wave 1 — three hot edges (`Prepare cutting three hot edges` + `Cut three hot include edges`)

edge cut	why it cost so much	how it is made removable
`core/pod_array.h` → `runtime/thread_context.h`	dead include left over from the PODArray memory-tracking experiment (#50549); the tracking logic has since moved into `Allocator` and `pod_array.h` names no `thread_context` symbol	just delete it — 203 TUs stop seeing `thread_context.h` and 195 of them stop seeing `exec_env.h` (1.37 MB/TU differential payload)
`core/column/column.h` → `exec/sort/hybrid_sorter.h`	a `core` → `exec` layering violation reaching 808 TUs	`HybridSorter` only appears in virtual signatures, so a forward declaration covers it; the `BE_TEST`-only `get_permutation_default` body (it constructs a `HybridSorter` by value) moves to `column.cpp`
`core/wide_integer_impl.h` → `boost/multiprecision` (unconditional)	4.27 MB of preprocessed closure in 1169 TUs on every platform without an 80-bit long double	the from-double members stay inline (and `constexpr`) only where `LDBL_MANT_DIG == 64`; elsewhere they compile once in a new `core/wide_integer_from_double.cpp` with explicit instantiations for `integer<128\|256, signed\|unsigned>`. They were never `constexpr` on those platforms, so no constant evaluation is lost

Preprocessed closure of column.h: 16.72 MB → 10.15 MB (-39%).

Wave 2 — two instantiation amplifiers: RLE and FMT_COMPILE

Both parquet decoder.h trees included util/rle_encoding.h only because inline
BaseDictDecoder bodies made ~530 TUs instantiate the
RleBatchDecoder<uint32_t> → GetLiteralValues → UnpackBatch → UnpackValues
chain — measured at 0.43 CPU s per TU, 231 CPU s total — on top of reparsing
the 2038-line rle/bit-stream/bit-packing family each time. The eight per-page /
per-batch methods plus the ctor and dtor move out of line, so the
unique_ptr<RleBatchDecoder<uint32_t>> member only needs the complete type in
decoder.cpp. Everything moved is dispatched through the vtable at every call
site already, so no generated call changes; the real decode TUs keep including
rle_encoding.h directly and inline the chain exactly as before, and the
per-value-hot LevelDecoder::get_next path is deliberately untouched.

core/uint24.h and core/value/large_int_value.h pushed their FMT_COMPILE
formatter instantiations (the "{:04d}-{:02d}-{:02d}" date formatter is the
single biggest fmt instantiation in the tree, plus the int128 formatters) into
~1150 TUs, 53.5 CPU s. Those bodies move to .cpp files and both headers
drop <fmt/compile.h> / <fmt/format.h>. They return std::string and are
allocation-dominated, so the now-outlined call is noise.

Wave 3 — the DataVariants amplifier behind `dependency.h`

exec/pipeline/dependency.h and exec/pipeline/rec_cte_shared_state.h hold
non-template in-class inline bodies — SharedState constructors, destructors,
close paths and three std::visit dispatches — that name the full
AggregatedDataVariants / JoinDataVariants / SetDataVariants /
DistinctDataVariants surface. Such bodies are semantically analyzed when the
header is parsed, not when they are called, so every one of the ~128 TUs that
transitively include dependency.h instantiated that whole surface at a flat
~0.85 CPU s per TU (~130 CPU s) — and 105 of those TUs never touch a variant.

All of it is per-query setup/teardown, so it moves to dependency.cpp and a new
rec_cte_shared_state.cpp. With the bodies gone the headers can drop:

exec/common/agg_utils.h, set_utils.h, distinct_agg_utils.h — forward declarations suffice;
exec/common/join_utils.h → the new light exec/common/join_op_utils.h (JoinOpVariants and the AsofIndexGroup family split out of join_utils.h, which re-exports them; dependency.h holds these by value, and the new header depends only on thrift enums, pdqsort and std containers — not on the hash tables);
exec/operator/join/process_hash_table_probe.h — dead include;
util/brpc_closure.h — dead include, and the sole route carrying query_context.h, thread_context.h and service/brpc.h into ~100 pipeline TUs (1.36 MB/TU);
<concurrentqueue.h> — dead include (152 KB third-party single header);
util/brpc_client_cache.h from rec_cte_shared_state.h — the rpc send bodies live in the .cpp now.

BucketedAggSharedState::init_instances also becomes a non-template taking
std::function: non-dependent constructs in a member-template body are checked
at definition time, so the old inline template forced the destructor of
unique_ptr<BucketedAggDataVariants> on every includer despite never being
called there.

One CI-critical commit rides along

Opt wide_integer_from_double.cpp out of the PCH must ship in this PR, not as
a follow-up. Upstream's clang toolchain defaults to ENABLE_PCH=ON
(be/CMakeLists.txt), and pch.h transitively includes wide_integer_impl.h,
whose include guard is then already consumed when the impl TU is compiled — the
explicit instantiations would find no definition. It is the only file in the tree
with this interaction, but splitting the two commits apart would leave a state
that fails the clang CI lane.

Benefit

Compile-time only; no runtime behavior change.

Head-to-head on exactly this PR's content

Cold, cache-free BE builds of this PR's merge base (c29075a7e10) and its head
(03129023ee5), run back-to-back on the same machine with the repo's own
--compile-bench harness — dedicated always-cold build dirs, ccache disabled,
-j10, ENABLE_PCH=ON, and both runs started from the same cooled state
(load1 2.1 vs 2.3) so neither side pays for the other's heat.

metric	before	after	delta
build phase wall	14m50s	14m14s	-36.3 s (-4.1%)
Σ TU wall time (parallelism-independent)	2h19m	2h13m	-6 min (-4.3%)
Σ TU cpu (user+sys)	2h18m	2h12m	-6 min
effective parallelism	9.4×	9.4×	identical ⇒ clean attribution
per-file wall	—	—	197 improved / 12 regressed / 3 new

The improvements land exactly on the predicted targets — pipeline_fragment_context.cpp
-6.0 s, rec_cte_anchor_sink_operator.cpp -5.0 s, operator.cpp -3.7 s,
data_queue.cpp -3.0 s (all W3 dependency.h consumers), and
byte_array_dict_decoder.cpp -2.5 s (the W2 decoder edge).

Of the 12 regressions, dependency.cpp +2.6 s is by design: that is the TU the
SharedState bodies were moved into, so it pays once for what ~128 TUs stop paying.
The rest sit in the noise floor of a -j10 run, and there is direct evidence for
that floor: gensrc/build/gen_cpp/cloud.pb.cc, a generated protobuf TU this PR
cannot touch, moved +2.3 s between the two runs.

Why no end-to-end total-wall number is quoted: the two trees differed outside
the build phase — the baseline tree skipped the contrib submodule step while the PR
tree re-fetched it (+58.1 s), and its gensrc was already generated (-5.6 s). Those
phases are not compilation, so only the build-phase and Σ-TU figures above are
attributable to the change.

One caveat worth stating: these numbers are with ENABLE_PCH=ON. A PCH already
amortizes exactly the kind of shared headers this PR is cutting, so it masks part of
the win — and the upstream compile lane builds without a PCH (its command line
carries no -include-pch). The effect there should be larger, not smaller.

Per-wave numbers from the development branch

wave	effect	measurement
W1	-24.3 s wall, `be/src` CPU -3.6%	same-codebase A/B on the batch branch
W2	-125 s / -6.9% (30m24s → 28m19s)	same-codebase A/B, back-to-back
W3	~130 CPU s of parse-time instantiation removed (~0.85 CPU s × ~128 TUs), plus 1.36 MB/TU × ~100 TUs of dead brpc payload	P3.5 research traces (per-TU CPU, not end-to-end wall)

These were taken on the batch branch at -j5/-j6 against an older base, so they
do not add up to the head-to-head figure above; they are included because they
attribute the win to each wave separately.

Risk and verification

Header sweeps. Every wave was validated with a full -fsyntax-only sweep over the natural (no-PCH) include closure: W1 1358 TUs with only the 4 known pre-existing failures, W2 1364/1364 clean, W3 0 failing of 1365.
BE unit tests. The first sweeps covered be/src only, so a full ninja -k 0 over 2422 targets was run to reach the 1024 be/test TUs; it surfaced 8 failing TUs in 4 families, all repaired in Repair the BE UT build after the include-edge cuts. Nothing there restores a cut edge and no production header gains an include. Two of those repairs fix a latent defect that predates this PR: BaseDictDecoder's defaulted-in-class constructor odr-uses ~unique_ptr<RleBatchDecoder<uint32_t>> in every TU constructing a derived decoder, so the header's own claim that the complete type is only needed in decoder.cpp did not hold — in both trees, though only the format/ one had a test reaching it.
Rebased onto current master (c29075a7e10) and rebuilt from scratch in a clean worktree, macOS/arm64 + clang 20, ENABLE_PCH=ON (upstream's clang default, so the PCH opt-out above is exercised): 8554/8554 ninja edges, zero failures, doris_be links. compile_commands.json confirms wide_integer_from_double.cpp is the one first-party TU compiled without the PCH.
BE unit tests rebuilt on the rebased tree: all 9602 objects compile, zero failures, and the final link resolves every symbol (0 undefined). Two pre-existing macOS-only obstacles were hit on the way and are called out under Proactive disclosure.
Natural-closure sweep (syntax_sweep.py --no-pch, the gate added in [opt](build) Add build-timing and header-closure sweep tooling #66616): 0 failing TUs of 1380. This is the gate that matters for a PR that cuts include edges — a normal build with ENABLE_PCH=ON cannot see a missing include that the PCH happens to supply, which is precisely how the <ranges> breakage in the last commit reached CI before it reached me.
Runtime cost of outlining. Everything moved out of line is either per-query setup/teardown (SharedState ctor/dtor/close), or a per-page/per-batch method already reached through a vtable. No per-value or per-row hot path was outlined.
Guard rules. check-header-deps.py gains rules pinning pod_array.h !-> thread_context.h, column.h !-> exec/sort/, and both decoder.h headers !-> util/rle_encoding.h (19 rules total, all passing).

Proactive disclosure

One commit here is unrelated to include edges: Make hierarchical_data_iterator_test compile on macOS arm64. std::min(*rows, ROWS - current_ordinal) cannot deduce _Tp where size_t is unsigned long and ordinal_t (uint64_t) is unsigned long long, which is the case on macOS/arm64 but not on Linux — so CI is green while be/test does not build on macOS at all. It arrived with [feature](variant) Support ColumnVariantV2 segment reads and writes #66204. It is fixed here because this branch is verified on macOS and the break blocks that verification; happy to split it out if a reviewer prefers.
The macOS Debug UT link has outgrown the Mach-O format, independently of this PR: section __debug_names's file offset exceeds 4GB. Master's own test growth is what crossed the line — the same worktree layout linked fine on 2026-08-10 at 7.6 GB of test debug info, and master is now at 8.0 GB across 10 more test TUs, while this PR adds one #include to each of 5 test files. Omitting the debug map (-Wl,-S) links the binary cleanly with zero undefined symbols, which is how the link was verified here. Worth someone's attention as a separate issue.
Cross-platform is the blind spot. All measurements and all sweeps ran on macOS/arm64 + clang 20. Nothing here is platform-specific by construction, but the Linux gcc/clang lanes are covered only by upstream CI, not by any local gate — please give those two lanes a look.
Two cuts have no scanner rule. The boost/multiprecision edge is preprocessor-gated and check-header-deps.py is preprocessor-blind (it would flag the impl TU's gated include); the macro structure is self-guarding instead — breaking it fails the impl TU's build. The fmt cuts are likewise only noted in a comment, since the scanner follows quoted project includes.
Three test files carry clang-format off/on around their includes. asof_join_test and the two fix_length_dict_decoder tests are include-order-sensitive: the supplying include has to come before the header under test (ADL cannot reach the global pdqsort from std::vector's iterators; a unique_ptr<RleBatchDecoder<uint32_t>> dereference depends on no template parameter, so it binds where the template is parsed). Without the marker the formatter sorts the include back and breaks the build. The cost was deliberately kept in the tests rather than paid by a production header.
DISALLOW_COPY_AND_ASSIGN in storage/olap_define.h loses its trailing semicolon. butil/macros.h defines the same macro without one and wins under #ifndef in TUs that see butil first, so the two expansions have to stay call-site compatible. All 45 call sites in the tree already write the ;.

Release note

None

Check List (For Author)

Test
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change. (full BE UT build + link, and the -fsyntax-only sweeps above)
Behavior changed:
- No.
Does this need documentation?
- No.

Pure-additive / behavior-preserving preparation so that three include edges can be cut next: core/pod_array.h -> runtime/thread_context.h (dead include), core/column/column.h -> exec/sort/hybrid_sorter.h (core->exec layering violation reaching 808 TUs) and wide_integer_impl.h's unconditional boost/multiprecision include (4.27MB of preprocessed closure in 1169 TUs on platforms without an 80-bit long double). No include edge is removed here: - core/wide_integer_from_double.cpp (new): compiles the from-double conversion once and explicitly instantiates it for integer<128|256, signed|unsigned>; harmless while the header still defines the members inline - column.h: forward-declare HybridSorter; move the BE_TEST-only get_permutation_default body to column.cpp (it constructs HybridSorter by value); the production get_permutation default keeps its inline throw body - column_decimal.h: move the permutation<U> helper body to column_decimal.cpp (its only caller, get_permutation, is already defined there); member templates are instantiated by that call, not by the class-level explicit instantiations - column_const.h: include <span> directly (rode in via hybrid_sorter.h) - storage/olap_define.h: drop the trailing semicolon inside DISALLOW_COPY_AND_ASSIGN so the call sites' `;` completes the expansion: butil/macros.h defines the same macro without one and wins under #ifndef in TUs that happen to see butil first, so the two definitions must stay call-site compatible - format_v2/native/native_reader.h: include runtime/runtime_profile.h (RuntimeProfile::Counter members need the complete type; it rode in via pod_array -> thread_context) - bm25_similarity.cpp: drop the dead `using namespace inverted_index` (the only declaration of that namespace lives in exec_env.h and rode the same edge) - inverted_index_searcher.h, index_file_writer.h, ann_index_writer.h, query/query.h: wrap their bare <CLucene.h> includes in the -Wconversion suppression (inverted_index_common_impl.h pattern); whether CLucene's first expansion lands inside a suppressed region depends on include order Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit c2bf234)

…nteger - core/pod_array.h no longer includes runtime/thread_context.h: dead include left over from the PODArray memory-tracking experiment (apache#50549); the tracking logic has since moved into Allocator and pod_array.h references no thread_context symbol. 203 TUs stop seeing thread_context.h and 195 of them stop seeing exec_env.h (differential payload 1.37MB per TU). - core/column/column.h no longer includes exec/sort/hybrid_sorter.h (core -> exec layering violation): HybridSorter only appears in virtual signatures, which the forward declaration covers; pdqsort/timsort leave 808 TUs. - wide_integer_impl.h defines the from-double members (set_multiplier / wide_integer_from_builtin(double)) inline only where the 80-bit long double exists (LDBL_MANT_DIG == 64 — unchanged and still constexpr there) or inside the dedicated impl TU core/wide_integer_from_double.cpp; every other TU sees declarations only and drops boost/multiprecision + boost/math from its closure (4.27MB preprocessed x 1169 TUs; the members were never constexpr on these platforms, so no constant evaluation is lost). - check-header-deps.py: two new rules (pod_array.h !-> thread_context.h, column.h !-> exec/sort/). No formal rule for the boost edge: the scanner is preprocessor-blind and would flag the impl TU's gated include; the macro structure is self-guarding (breaking it fails the impl TU build). Preprocessed closure of column.h: 16.72MB -> 10.15MB (-39%). Syntax sweep: 1358 TUs; failures identical to the 4 known pre-existing merge artifacts (stale gensrc x3, apache#66242 template-only call site), zero new. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit 13fca3d)

…iers Pure-additive / behavior-preserving preparation so that two kinds of template-instantiation edges can be cut next: both parquet decoder.h headers' util/rle_encoding.h include (their inline BaseDictDecoder bodies force ~530 TUs to instantiate the RleBatchDecoder<uint32_t> -> GetLiteralValues -> UnpackBatch -> UnpackValues chain at ~0.43 CPU s per TU, 231 CPU s total) and the FMT_COMPILE formatter instantiations that uint24.h / large_int_value.h push into ~1150 TUs (53.5 CPU s). No include edge is removed here: - format/parquet/decoder.h: move set_data, skip_values and the defaulted ~BaseDictDecoder out of line into decoder.cpp; everything moved is a per-page/per-batch virtual call that is dispatched through the vtable at every call site already, so outlining changes no generated call. With the dtor out of line the unique_ptr<RleBatchDecoder<uint32_t>> member no longer needs the complete type in the header; forward-declare it - format_v2/parquet/reader/native/decoder.h: same treatment for set_data, decode_dictionary_indices, decode_selected_dictionary_indices, _decode_fragmented_selection, _decode_and_validate_skipped, _decode_dictionary_values and the dtor; the pure orchestrator bodies (skip_values, decode_dictionary_values, decode_selected_dictionary_values) stay inline as they only call the now-declared members - decoder.cpp (both trees): include util/rle_encoding.h and common/cast_set.h directly (cast_set previously rode in through rle_encoding.h) - core/uint24.h: move to_string (the FMT_COMPILE("{:04d}-{:02d}-{:02d}") date formatter, the single biggest fmt instantiation in the codebase) into a new core/uint24.cpp; a std::string-returning formatter is allocation-dominated, so the outlined call is noise - core/value/large_int_value.h: move to_buffer/to_string x2 (the FMT_COMPILE int128 formatters) into large_int_value.cpp, which now includes fmt/compile.h + fmt/format.h directly instead of riding its own header The RleBatchDecoder methods stay fully inline in rle_encoding.h for the real decode TUs; the per-value-hot LevelDecoder::get_next path is deliberately untouched. (cherry picked from commit df9ec8f)

The one-line deletions the previous commit prepared, plus the guard rules that lock them in: - format/parquet/decoder.h and format_v2/parquet/reader/native/decoder.h drop util/rle_encoding.h (RleBatchDecoder<uint32_t> is forward-declared; the complete type is only needed by decoder.cpp). New trace run 20260805_131321 measured the old edges at 531 TUs x 435 ms = 231 CPU s of RleBatchDecoder/BitPacking/BatchedBitReader instantiation, plus the 2038-line rle_encoding/bit_stream_utils/bit_packing family reparsed in each of those TUs. The real decode TUs keep including rle_encoding.h directly and inline the chain exactly as before - core/uint24.h drops <fmt/compile.h> and core/value/large_int_value.h drops <fmt/compile.h> + <fmt/format.h>: their formatter bodies moved to the matching .cpp files, which ends the FMT_COMPILE date/int128 formatter instantiation (53.5 CPU s over ~1150 TUs) in every includer - check-header-deps.py: two new rules pin both decoder.h headers away from util/rle_encoding.h (19 rules total, all passing); the fmt cuts are noted in a comment because the scanner only follows quoted project includes Full -fsyntax-only sweep: 1364/1364 TUs clean with the cuts applied (the single failure during arbitration was large_int_value.cpp riding its own header for fmt, fixed in the preparation commit). (cherry picked from commit 624d039)

The four cut-edge commits on this branch were validated with a -fsyntax-only sweep over be/src only, so the 1024 be/test TUs were never compiled against the slimmed graph. A full `ninja -k 0` run (2422 targets) surfaced 8 failing TUs in 4 families. Nothing here restores a cut edge, and no production header gains an include; the RLE fix in fact tightens the invariant decoder.h already claimed. - core/wide_integer_from_double.cpp: -Werror,-Wunused-macros. Where the 80-bit long double exists, wide_integer_impl.h takes its `#if (LDBL_MANT_DIG == 64)` branch and never evaluates the `#elif defined(...)` that reads DORIS_WIDE_INTEGER_FROM_DOUBLE_IMPL_TU, so the definition looked unused. The existing instantiation guard now reads the macro itself. `defined(...)` must stay the left operand -- the preprocessor short-circuits `&&`, so a false left term skips it and the warning returns. - format/parquet/decoder.h and format_v2/parquet/reader/native/decoder.h: outline BaseDictDecoder's default ctor next to the dtor that was already outlined. A defaulted-in-class ctor is defined in every TU that constructs a derived decoder and it odr-uses ~unique_ptr<RleBatchDecoder<uint32_t>>, so the header's own comment -- that the complete type is only needed in decoder.cpp -- did not hold. Both trees had the defect; only the format/ one had a test reaching it. - block_test / column_variant_v2_test: declare the RuntimeProfile and HybridSorter they use directly. They had been riding pod_array.h -> thread_context.h and column.h -> exec/sort/hybrid_sorter.h. - asof_join_test and the two fix_length_dict_decoder tests: include-order- sensitive, so the cost stays in the tests rather than in a production header. AsofIndexGroup::sort_and_finalize calls the global pdqsort, which ADL cannot reach from std::vector's iterators; FixLengthDictDecoder::_decode_values dereferences a unique_ptr<RleBatchDecoder<uint32_t>> that depends on no template parameter. Both bind where the template is parsed, not where it is instantiated, so the supplying include has to come first. The two decoder tests carry clang-format off/on because the formatter would otherwise sort the include back after the header under test and break the build again. Verified: `sh run-be-ut.sh` links test/doris_be_test with 0 failures; all 10 files pass clang-format --dry-run --Werror. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3BVeKrPS3jQ2bagcuGHBQ (cherry picked from commit f8b01ae)

Pure-additive / behavior-preserving preparation so that the hash-table variant machinery (exec/common/{agg,join,set,distinct_agg}_utils.h) can be cut out of exec/pipeline/dependency.h and rec_cte_shared_state.h next. Those two headers hold non-template in-class inline bodies (SharedState constructors/destructors/close paths, three std::visit dispatches) that reference the full variant types, and such bodies are semantically analyzed when the header is parsed: every one of the ~128 TUs that transitively include dependency.h instantiates the whole AggregatedDataVariants/JoinDataVariants/SetDataVariants surface at a flat ~0.85 CPU s per TU (~130 CPU s total), whether it uses them or not. No include edge is removed here: - dependency.h: move out of line into dependency.cpp everything whose body touches a variant -- AggSharedState ctor/dtor/_close_with_serialized_key/ _close_without_key, BucketedAggSharedState PerInstanceData ctor, dtor, _close and _close_one_agg_data, both HashJoinSharedState ctors, and a new user-declared SetSharedState ctor/dtor pair that takes over the make_unique<SetDataVariants>() default member initializer. All of it is per-query setup/teardown, so outlining costs nothing at runtime - dependency.h: forward-declare the variant structs, spell the members as plain std::unique_ptr/std::vector<size_t> instead of the utils-provided aliases, redeclare AggregateDataPtr, and directly include the light dependencies that used to ride in through the utils chain (core/arena.h, exprs/vexpr_fwd.h, util/stopwatch.hpp, cast_set/exception/ factory_creator, <list>/<queue>/<set>) - exec/common/join_op_utils.h (new): JoinOpVariants, is_asof_join*, and the AsofIndexGroup/AsofIndexVariant family move here out of join_utils.h, which re-exports it; dependency.h holds these by value, and this light header depends only on thrift enums, pdqsort and std containers -- not on the hash tables - rec_cte_shared_state.h: same treatment for the fourth amplifier; the emplace_block std::visit over DistinctDataVariants plus the brpc-heavy build_basic_param/send_data_to_targets bodies move to a new rec_cte_shared_state.cpp (CMake GLOB picks it up), with user-declared ctor/dtor so unique_ptr<DistinctDataVariants> works forward-declared - seed direct includes ahead of the cut: agg_utils.h into the three agg operator headers whose inline code walks method_variant, join_utils.h into join_build_sink_operator.h, distinct_agg_utils.h into rec_cte_source_operator.cpp, and the utils/template_helpers includes into dependency.cpp Verified with a full -fsyntax-only sweep: 0 failing TU(s) of 1365. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit e39c032)

…ncy.h With the SharedState bodies out of line (previous commit), drop the edges that made every pipeline TU pay for the hash-table variant machinery and two dead payload includes. Measured on the P3.5 research traces: a flat ~0.85 CPU s of parse-time template instantiation per TU across the ~128 TUs reached by dependency.h (105 of which never touch a variant), plus 1.36 MB/TU of preprocessed brpc/query_context payload riding the dead brpc_closure.h edge into ~100 TUs. Edges cut from exec/pipeline/dependency.h: - exec/common/agg_utils.h, set_utils.h: forward declarations suffice now - exec/common/join_utils.h -> exec/common/join_op_utils.h: the by-value JoinOpVariants/AsofIndexVariant members need complete types, so keep only the light join-op header split out in the previous commit - exec/operator/join/process_hash_table_probe.h: dead include (no symbol of it is referenced here); the probe machinery belongs to the join TUs - util/brpc_closure.h: dead include; it was the only route that carried runtime/query_context.h, runtime/thread_context.h and service/brpc.h into ~100 pipeline TUs - <concurrentqueue.h>: dead include (152 KB third-party single header); the moodycamel users include it themselves Also make BucketedAggSharedState::init_instances a non-template taking std::function (defined in dependency.cpp): non-dependent constructs in a member-template body are checked at definition time, so the old inline template still forced the destructor of unique_ptr<BucketedAggDataVariants> on every includer despite never being called there. Once-per-query cold path. exec/pipeline/rec_cte_shared_state.h (fourth amplifier, same recipe): - cut exec/common/distinct_agg_utils.h (DistinctDataVariants is only touched by rec_cte_shared_state.cpp now) and util/brpc_client_cache.h (the rpc send bodies live in the .cpp) Seeds arbitrated by two full -fsyntax-only sweeps (16 true positives): - process_hash_table_probe_impl.h includes join_utils.h (the INSTANTIATION_FOR macro names SerializedHashTableContext and friends, fixing all 12 *_join_impl.cpp TUs); process_hash_table_probe.h becomes self-contained (runtime_profile.h for its Counter members) - hashjoin_probe_operator.h includes process_hash_table_probe.h (the by-value std::variant<ProcessHashTableProbe<...>> needs the definition) - direct utils/hash_map_util includes for the TUs that dereference the variants: hashjoin_probe_operator.cpp, hashjoin_build_sink.cpp, the three set operator cpps, the three aggregation source cpps - materialization_opertor.h includes service/brpc.h for its brpc::Controller member; local_exchanger.h includes concurrentqueue.h for its ConcurrentQueue members Guards: check-header-deps.py gains five rules (24 total, all passing): dependency.h and rec_cte_shared_state.h must not reach exec/common/hash_table/ (phmap_fwd_decl.h excepted), dependency.h must not regain process_hash_table_probe.h / brpc_closure.h, and rec_cte_shared_state.h must not regain brpc_client_cache.h; the concurrentqueue.h ban is recorded as a comment (third-party angle include, outside the scanner's reach). Verified: full -fsyntax-only sweep 0 failing of 1365; header layering 24/24. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit 82d8ce7)

The PCH already pulls in wide_integer_impl.h (via storage/olap_common.h) in declaration-only mode, so its pre-spent include guard would leave the from-double explicit instantiations in this TU without definitions. Skip the PCH for this single TU so DORIS_WIDE_INTEGER_FROM_DOUBLE_IMPL_TU takes effect. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit ae1f7b1)

`std::min(*rows, ROWS - _state->current_ordinal)` fails template argument deduction wherever size_t and ordinal_t are distinct types: on macOS/arm64 size_t is `unsigned long` while uint64_t -- and hence ordinal_t -- is `unsigned long long`, so _Tp is deduced conflictingly and none of the four std::min overloads match. On Linux the two spell the same type, which is why CI never saw it. Name the template argument explicitly at the three call sites. Both types are 64-bit unsigned, so nothing is narrowed. The break arrived with apache#66204 and has nothing to do with the include-edge cuts on this branch, but it stops be/test from building on macOS at all -- which is where this branch is verified -- so it is fixed here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gdfkk7RqgD5e3Uv7bTM3NV

hello-stephen · 2026-08-11T16:21:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-08-11T16:22:07Z

run buildall

morningman · 2026-08-11T16:24:26Z

run buildall

morningman · 2026-08-12T00:16:36Z

run buildall

…ply it `column.h -> exec/sort/hybrid_sorter.h` was carrying `gfx/timsort.hpp`, whose line 37 is `#include <ranges>`. Four files consumed `<ranges>` through that chain without ever asking for it, and the previous commit's cut takes it away. Why the local build did not see this: it runs with `ENABLE_PCH=ON`, and `cmake_pch.hxx` supplies `<ranges>` to every TU regardless of that TU's own includes, so the debt stays invisible. The upstream compile lane builds without a PCH, where nothing covers for it. Compiling the same file on macOS with the PCH genuinely stripped reproduces the identical error, so this is not a libc++/libstdc++ difference -- it is the PCH masking a missing include, exactly the failure mode `syntax_sweep.py --no-pch` exists to expose. - storage/tablet/base_tablet.cpp: `std::views::transform` / `std::views::filter` -- this is the one the CI compile lane caught. - cloud/cloud_meta_mgr.{h,cpp}: three declarations and three definitions constrained on `std::ranges::range auto&&`. The Cloud target is compiled after Storage, so the build stopped before reaching it; it would have failed next. - storage/compaction/compaction.cpp: `std::ranges::reverse_view`. This one still compiles, because simdjson.h happens to remain in its closure and also pulls `<ranges>` -- included here so it stops depending on that accident. Found by diffing each consumer's include closure before and after the cut and matching it against the symbols the removed providers supply, rather than by waiting for another CI round trip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gdfkk7RqgD5e3Uv7bTM3NV

morningman · 2026-08-12T00:22:54Z

run buildall

…::agg_data GCC rejects the forward-declared `unique_ptr<DistinctDataVariants>` member that the previous commits introduced, in every TU that merely includes rec_cte_shared_state.h: unique_ptr.h:91:23: error: invalid application of 'sizeof' to incomplete type 'doris::DistinctDataVariants' required from 'std::unique_ptr<_Tp, _Dp>::~unique_ptr()' rec_cte_shared_state.h:42:54: required from here Column 54 is the `nullptr` of `agg_data = nullptr`. A default member initializer is part of the class definition, so GCC instantiates the member's destructor while completing the class -- which needs `DistinctDataVariants` to be complete -- even though the constructor and destructor of RecCTESharedState are both defined out of line in rec_cte_shared_state.cpp, where it is complete. Clang defers that instantiation to the point the constructor is actually defined, which is why the clang lanes and the macOS build were green while the gcc lane failed on rec_cte_anchor_sink_operator.cpp and rec_cte_sink_operator.cpp. `= nullptr` was redundant to begin with: `unique_ptr`'s default constructor already leaves it null, and `RecCTESharedState::RecCTESharedState() = default;` in the .cpp performs that initialization where the type is complete. So dropping the initializer changes no behavior and removes the only instantiation point that required completeness in the header. Swept the rest of the tree for the same shape: the only other `unique_ptr<DistinctDataVariants>` with an initializer is in distinct_streaming_aggregation_operator.h, which includes distinct_agg_utils.h directly and therefore has the complete type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gdfkk7RqgD5e3Uv7bTM3NV

morningman · 2026-08-12T01:53:55Z

run buildall

morningman · 2026-08-12T05:50:51Z

run buildall

hello-stephen · 2026-08-12T07:30:14Z

TPC-H: Total hot run time: 28728 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5adf59ef9ecf14250f9f9435258a91709eb5c648, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17569	3979	3947	3947
q2	2005	337	203	203
q3	10284	1419	794	794
q4	4688	475	340	340
q5	7546	845	553	553
q6	202	173	138	138
q7	733	792	599	599
q8	10141	1339	1331	1331
q9	5427	4059	4026	4026
q10	6790	1622	1358	1358
q11	497	358	329	329
q12	741	563	462	462
q13	18090	3392	2787	2787
q14	263	255	240	240
q15	q16	744	735	661	661
q17	1040	972	979	972
q18	6595	5661	5535	5535
q19	1210	1276	1122	1122
q20	793	681	576	576
q21	5685	2666	2454	2454
q22	433	355	301	301
Total cold run time: 101476 ms
Total hot run time: 28728 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4328	4244	4212	4212
q2	284	314	210	210
q3	4517	4947	4361	4361
q4	2257	2374	1461	1461
q5	4226	4087	4158	4087
q6	235	180	127	127
q7	1688	1592	1412	1412
q8	2815	2103	2120	2103
q9	7362	7198	7306	7198
q10	4331	4301	3870	3870
q11	585	393	370	370
q12	703	721	513	513
q13	3297	3726	2993	2993
q14	295	304	286	286
q15	q16	680	745	629	629
q17	1329	1273	1316	1273
q18	12200	11086	11806	11086
q19	1184	1152	1170	1152
q20	2255	2227	1927	1927
q21	5916	5180	4983	4983
q22	511	449	396	396
Total cold run time: 60998 ms
Total hot run time: 54649 ms

hello-stephen · 2026-08-12T07:40:29Z

TPC-DS: Total hot run time: 158781 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5adf59ef9ecf14250f9f9435258a91709eb5c648, data reload: false

query5	4321	590	454	454
query6	467	221	203	203
query7	4850	571	343	343
query8	332	165	145	145
query9	8771	4096	4061	4061
query10	503	374	308	308
query11	5846	2193	1994	1994
query12	148	100	97	97
query13	1250	592	422	422
query14	6051	4320	4021	4021
query14_1	3863	3863	3828	3828
query15	204	198	175	175
query16	969	470	483	470
query17	898	667	540	540
query18	2419	444	328	328
query19	201	186	143	143
query20	101	98	102	98
query21	233	155	137	137
query22	13075	13029	12813	12813
query23	15835	14872	14664	14664
query23_1	14749	14655	14701	14655
query24	7510	1700	1276	1276
query24_1	1277	1263	1236	1236
query25	560	423	351	351
query26	1334	364	210	210
query27	2580	578	367	367
query28	4533	2026	2026	2026
query29	1017	597	481	481
query30	327	260	223	223
query31	1196	1108	1046	1046
query32	104	61	61	61
query33	528	303	235	235
query34	1153	1131	671	671
query35	756	754	646	646
query36	768	769	709	709
query37	160	114	89	89
query38	1842	1791	1664	1664
query39	826	821	800	800
query39_1	785	798	774	774
query40	242	169	139	139
query41	68	63	63	63
query42	96	93	93	93
query43	318	324	278	278
query44	1456	764	767	764
query45	185	174	167	167
query46	1037	1167	684	684
query47	1543	1516	1449	1449
query48	407	393	290	290
query49	577	409	293	293
query50	1059	445	330	330
query51	10513	10684	10721	10684
query52	93	89	79	79
query53	252	275	191	191
query54	279	230	212	212
query55	74	73	67	67
query56	350	297	301	297
query57	1014	997	917	917
query58	282	264	251	251
query59	1575	1594	1406	1406
query60	317	269	262	262
query61	150	142	155	142
query62	394	319	265	265
query63	234	191	201	191
query64	2848	1016	896	896
query65	3915	3802	3800	3800
query66	1840	481	387	387
query67	20040	20100	19935	19935
query68	3082	1520	956	956
query69	426	317	297	297
query70	931	794	773	773
query71	378	398	301	301
query72	2989	2590	2277	2277
query73	871	771	435	435
query74	4670	4477	4294	4294
query75	2371	2345	2000	2000
query76	2297	1136	766	766
query77	347	361	277	277
query78	11067	11147	10650	10650
query79	1407	1118	782	782
query80	667	596	463	463
query81	453	325	284	284
query82	622	175	138	138
query83	417	322	298	298
query84	326	170	131	131
query85	916	628	515	515
query86	327	230	230	230
query87	1994	1967	1845	1845
query88	3730	2833	2789	2789
query89	395	315	294	294
query90	1868	204	198	198
query91	201	195	163	163
query92	61	61	53	53
query93	1575	1476	1061	1061
query94	529	346	327	327
query95	793	529	490	490
query96	1060	801	373	373
query97	2501	2466	2365	2365
query98	195	185	180	180
query99	761	754	610	610
Total cold run time: 243976 ms
Total hot run time: 158781 ms

hello-stephen · 2026-08-12T07:45:15Z

ClickBench: Total hot run time: 24.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5adf59ef9ecf14250f9f9435258a91709eb5c648, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.05	0.06
query3	0.26	0.14	0.13
query4	1.60	0.14	0.14
query5	0.25	0.24	0.24
query6	1.16	0.86	0.85
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.34	0.33
query10	0.57	0.59	0.56
query11	0.19	0.14	0.14
query12	0.18	0.15	0.14
query13	0.48	0.47	0.47
query14	0.98	0.99	1.00
query15	0.64	0.60	0.60
query16	0.31	0.35	0.33
query17	1.11	1.06	1.07
query18	0.21	0.21	0.20
query19	2.03	1.95	1.94
query20	0.02	0.02	0.01
query21	15.43	0.22	0.15
query22	4.83	0.06	0.06
query23	16.10	0.31	0.12
query24	3.03	0.43	0.33
query25	0.11	0.05	0.04
query26	0.76	0.22	0.16
query27	0.04	0.04	0.03
query28	3.48	0.78	0.35
query29	12.47	4.07	3.22
query30	0.28	0.16	0.15
query31	2.77	0.57	0.32
query32	3.23	0.60	0.48
query33	3.19	3.20	3.20
query34	15.64	3.94	3.27
query35	3.23	3.21	3.23
query36	0.55	0.44	0.43
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.23 s
Total hot run time: 24.11 s

github-actions · 2026-08-12T10:25:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-08-12T10:25:49Z

PR approved by anyone and no changes requested.

morningman · 2026-08-12T14:00:40Z

run vault_p0

eldenmoon

LGTM

morningman · 2026-08-13T03:29:16Z

run check_coverage

…lue targets (#66712) > Part of the BE build-time optimization series tracked in #66715. > > Split out of **#66510, which carries the whole > BE build-time batch. After the header-closure surgery in **#66400** and **#66672**, > this PR opens the second line of that batch: CMake unity builds. It adds the > infrastructure switch and converts the three lowest-risk glue segments as a pilot; > the heavier targets (Exec, Exprs, and the rest) follow in separate PRs once this > one has proven the mechanism cross-platform. ### What problem does this PR solve? Related PR: #66510, #66672 Problem Summary: Most of the BE's cold-build time is not spent compiling our code — it is spent **re-parsing the same shared header closure once per small `.cpp`**. For the glue directories this ratio is extreme: the 51 InformationSchema scanners cost ~124 CPU s (with PCH) of which almost everything is the closure re-parse; the ~58 http handlers sum to ~5.6 min of slot time for a few thousand lines of handler logic. CMake's built-in `UNITY_BUILD` (CMake ≥ 3.16; we require 3.19.2) concatenates groups of `.cpp` files into jumbo TUs, so a closure is parsed **once per batch** instead of once per file. This PR: 1. **Adds the switch** — `option(ENABLE_UNITY_BUILD ON)` in `be/CMakeLists.txt`, plumbed through `build.sh` and `run-be-ut.sh` exactly like `ENABLE_PCH`, overridable from the environment / `custom_env.sh`. Every per-target `UNITY_BUILD` property is set unconditionally from it -- including an explicit OFF, so the switch also wins over CMake's own `CMAKE_UNITY_BUILD` in a reused cache. Turn it OFF for per-file diagnostics, per-file tooling (clang-tidy, coverage), or the finest-grained incremental rebuilds. 2. **Pilots unity on three glue segments** (and only there — everything else is explicitly opted out per file, not by omission): - **InformationSchema**: all 51 scanner TUs → 1 unity TU (`UNITY_BUILD_BATCH_SIZE 0`). - **Service, scoped to `http/`**: ~58 handler TUs → 1 unity TU. The non-http service sources (service entry points, arrow_flight) are heterogeneous heavy TUs that gain nothing from merging and stay individual. - **Storage, scoped to `index/`**: ~111 TUs sharing one CLucene-heavy closure → 4 unity TUs of ≤ 32 sources (batch size bounds jumbo-TU size and memory). 3. **Makes the merged TUs legal C++** with two hygiene commits that stand on their own even without unity: - 27 http action files each carried a private copy of the same file-scope constants (`HEADER_JSON` ×16, `TABLET_ID` ×7, `SCHEMA_HASH` ×4, …). They move to one shared header as C++17 inline variables, values unchanged. - `prefix_query` declared `get_prefix_terms(IndexReader*)` relying on a header-level `CL_NS_USE(index)` using-directive. Inside `doris::segment_v2` that unqualified name silently flips to `doris::segment_v2::IndexReader` as soon as any sibling source brings that type into scope — a landmine with or without unity. The declaration now spells out `lucene::index::IndexReader`. 4. **Fixes a latent bug unity exposed**: `schema_scanner_helper.h` opened with `#ifndef _SCHEMA_SCANNER_HELPER_H_` but never defined the macro (and one include sat outside the guard), so the guard never worked. Harmless while every TU included it exactly once; breaks immediately under unity. Now `#pragma once`. 5. **Keeps the compile-bench tooling honest under unity** — with the default ON, `build.sh --compile-bench` produces a build directory whose sources are jumbo TUs, which two of the already-merged tools silently mis-read: `cut_impact.py` dropped every unity dependency block (and with it all 211 pilot sources) from its TU set, and `report.py` filed each batch's time under the build directory instead of the module it came from. `cut_impact.py` now refuses a unity build directory with the rebuild instruction — its analysis needs per-source closures, and splitting a batch's union back over its members would over-estimate instead of under-report — `report.py` attributes a unity TU to its target's directory, and `rebuild_radius.py` says when its per-header counts are batch-granular. `build-support/tests/test-compile-bench-unity.sh` pins all three behaviours. Files whose file-scope macros must not leak into unity siblings stay individual via `SKIP_UNITY_BUILD_INCLUSION`: `http_parser.cpp` (CR/LF), `be_thread_stack_action.cpp` (`UNW_LOCAL_ONLY`), and four index files (`CL_MAX_PATH` and friends, `IS_CHINESE_CHAR`, `APPLY_FOR_PRIMITITYPE`). A future file whose file-scope symbols clash inside a unity TU can opt out the same way. ### Measured results All numbers from the development branch this series is split from (which also carried the #66672 include cuts), macOS arm64 + clang 20, `-j14`, `ENABLE_PCH=ON`, cold builds, back-to-back A/B: | metric | before | after | |---|---|---| | build phase wall | 11m38s | **10m16s (-82.4 s / -11.8%)** | | Σ per-TU CPU | 147.9 min | **134.8 min (-8.9%)** | | TU count | 8384 | 8180 | | failures | 0 | 0 | Per-segment slot time (sum of compile time the segment occupies across slots): | segment | before | after | ratio | |---|---|---|---| | information_schema | 221.7 s | 17.0 s | **13×** | | service/http | 273.6 s | 26.3 s | **10.4×** | | storage/index | 264.2 s | 120.1 s | **2.2×** (remainder includes the four macro SKIPs) | Why unity rather than more PCH: a serial cold micro-benchmark of the InformationSchema target alone measured **9.9× with PCH** (124 s → 12.5 s) and **14.6× without PCH** (206 s → 14.1 s) — and unity *without* PCH still beats individual TUs *with* PCH by 8.8×. Unity removes the repeated parse instead of amortizing it, and the two compose. Side effects on artifacts: `libInformationSchema.a` 334 MB → 29 MB, `libStorage.a` 1492 MB → 1318 MB (linkonce_odr instantiations dedup inside each unity TU). The largest unity TU peaks at 2.1 GB RSS — *below* the largest existing individual TU in the tree (3.9 GB), so `-jN` memory envelopes are unchanged. ### Risk and verification - **Unity changes TU grouping only; no code changes ride along** beyond the two hygiene commits described above (constant dedup with identical values, one qualified name, one include guard). - **Archive symbol parity** was checked per target on the development branch: InformationSchema keeps all 2303 external defined symbols (plus 5 weak `unique_ptr<SchemaXxxScanner>` instantiations that dedup), Service keeps all 4054, Storage keeps all external symbols with 8 weak linkonce_odr template instantiations deduping away — which is the point of unity, not a loss. - **This exact branch, rebased onto current master, full BE build from scratch** (macOS arm64, clang 20, `ENABLE_PCH=ON`, `ENABLE_UNITY_BUILD=ON`): **8349/8349 ninja edges, zero failures, `doris_be` links.** The six expected unity TUs (InformationSchema ×1, Service http ×1, storage/index ×4) all compile. This includes `schema_tso_status_scanner.cpp`, added upstream after the pilot was measured — it lands inside the InformationSchema unity TU via the existing `GLOB_RECURSE` with zero CMakeLists edits, which is the intended maintenance story. - **The OFF path is verified on the same tree**: reconfiguring with `ENABLE_UNITY_BUILD=OFF` removes every `unity_*.cxx` entry from `compile_commands.json` and flips **exactly 219 ninja edges** — the three targets' per-file objects plus their archives and the final link, nothing else. All of them compile per-file with zero failures and `doris_be` links again. The blast radius of the switch is precisely the three pilot targets. - **`OFF` also wins over a native-unity cache**: configuring with `-DENABLE_UNITY_BUILD=OFF -DCMAKE_UNITY_BUILD=ON` produced 28 unity TUs across the three pilot targets before the gate fix and **0** after, while the ordinary `-DENABLE_UNITY_BUILD=ON` configure still produces exactly the 6 advertised batches (InformationSchema ×1, Service ×1, Storage ×4). - **These three segments have been building as unity TUs on the development branch since 2026-08-08**, through repeated full-tree builds and the BE UT builds that verified #66672 (the UT binaries link against these same target libraries). ### Proactive disclosure - **Cross-platform is the blind spot, and the default is deliberately ON so this PR's own CI closes it.** Every local build and measurement above is macOS arm64 + clang 20. Nothing here is platform-specific by construction, but with the default ON, the Linux compile lanes and every regression pipeline in this PR's CI run against unity builds — that is the validation. Please give the Linux gcc lane in particular a look. If some environment trips over unity after merge, the escape hatches are, in order: per-user `ENABLE_UNITY_BUILD=OFF` (env or `custom_env.sh`), per-file `SKIP_UNITY_BUILD_INCLUSION`, or a one-line default flip — the infrastructure stays either way. - **The incremental-rebuild trade-off is real**: touching one `.cpp` inside a unity batch recompiles the whole batch. For the glue chosen here a batch compiles in ~16–30 s, comparable to single mid-weight TUs elsewhere in the tree; and `ENABLE_UNITY_BUILD=OFF` restores per-file granularity for workflows that need it. This is also why the pilot targets are glue directories and not the hot-edit paths. - **contrib (openblas/clucene) was evaluated and deliberately left alone**: the f2c-generated LAPACK sources and the snowball stemmers define clashing file-scope statics (`static c__1` and friends) — structurally un-unifiable without rewriting generated code — and contrib compiles once and rarely changes. - **The rest of Storage and Service is opted out per file, on purpose.** Merging heavy heterogeneous TUs earns nothing (the closure parse is not the dominant cost there) and risks monster TUs. Follow-up PRs extend unity to Exec, Exprs and the remaining targets with the same SKIP discipline; on the development branch the full rollout takes the same tree from 10m16s to **6m18s**.

> Part of the BE build-time optimization series tracked in #66715. > > Split out of **#66510, which carries the whole > BE build-time batch. #66712 introduced the `ENABLE_UNITY_BUILD` switch and piloted > unity builds on three low-risk glue targets. This PR extends unity to **Exec and > Exprs** — the two heaviest targets in the BE build and the largest single source of > the unity line's win. The remaining targets follow in one more PR. ### What problem does this PR solve? Related PR: #66510, #66712 Problem Summary: Same mechanism as #66712: most of the cold-build cost of glue-heavy targets is **re-parsing the shared header closure once per small `.cpp`**, and CMake's `UNITY_BUILD` makes a batch pay that parse once. What is new here is the scale — Exec (174 TUs, `libExec.a` 2450 MB) and Exprs (287 TUs, `libExprs.a` 2721 MB) are the two heaviest targets in the tree, and their glue shares the heaviest closures (`operator.h`/`dependency.h` for Exec; the vexpr/factory closure for Exprs). The four commits: 1. **Deduplicate exec file-scope names that clash under unity** (no behavior change): `file_scanner.cpp`/`file_scanner_v2.cpp` both defined the Iceberg delete content codes and `is_iceberg_position_deletes_sys_table()` in anonymous namespaces — the shared trio moves to `iceberg_scan_semantics.h` (`file_scanner_v2_test.cpp` carried a third copy, kept file-local by #66615 because this header move had not landed yet; it now uses the header too). `vtablet_writer.cpp`/`vtablet_writer_v2.cpp` both defined a file-scope `CLOSE_WAIT_EVENT_FALLBACK_MS` — scoped into `IndexChannel` and `VTabletWriterV2`; v2's file-scope `on_partitions_created()` trampoline renamed `on_partitions_created_v2` (the two functions cast to different writer types). `exchange_sink_operator.cpp`'s namespace-scope `timer_name` renamed `wait_for_dependency_timer_name` (shadowed unity siblings' locals under `-Wshadow -Werror`). 2. **Unity for the whole Exec target**: 167 of 174 TUs join 14 unity batches of ≤12 sources, gated on `ENABLE_UNITY_BUILD` like the pilot targets. Opted out: five files whose file-scope macros must not leak into siblings, plus the two heaviest template-instantiation TUs (`operator.cpp`, `hashjoin_build_sink.cpp`) which would dominate any batch they join; `scan_operator.cpp` is both. 3. **Three latent defects the Exprs conversion surfaced** (stand on their own): `dictionary_factory.h` had **no include guard at all** — any TU reaching it through two include paths fails with a class redefinition, and under unity the clang error recovery poisoned unrelated batch members with spurious `-Warray-bounds` diagnostics. Now `#pragma once`. And `function_dict_get_many.cpp` had copy-pasted the `DictGetState` struct from `function_dict_get.cpp` at namespace scope — renamed `DictGetManyState` so the two TUs can share a batch. And `function_variant_element_v2.cpp` kept `OwnedPathSegment` in an anonymous namespace while using it as a field of the externally-visible `ResolvedVariantElementV2Path::Impl` — gcc's `-Wsubobject-linkage` (`-Werror`) rejects exactly that once the file is `#include`d into a unity batch instead of being the main file of its TU (clang has no such warning); the struct moves to namespace scope. 4. **Unity for the Exprs glue**: 246 of 287 TUs join 31 unity batches of ≤8 sources, same switch. Opted out: the flex/bison/gperf generated tables, seven macro-leaking files, the 30 heavy template-instantiation TUs (>15 s wall or >2.2 GB RSS in the compile bench: the min_max/collect/topn/percentile aggregate family, `in.cpp`, `multiply.cpp`, `function_array_aggregation.cpp`, …) whose per-file codegen would only stack into jumbo poles — and three files that tests compile a second time by `#include`ing the `.cpp` (`function_variant_element.cpp`, `uuid.cpp`, `function_jsonb_transform.cpp`), see the verification section. ### Measured results All numbers from the development branch this series is split from, macOS arm64 + clang 20, `-j14`, `ENABLE_PCH=ON`, cold builds, back-to-back A/B. The baseline is the #66712 state of that branch (10m16s), so the two waves compose with the pilot: | wave | build phase wall | target slot time | archive size | |---|---|---|---| | Exec unity | 10m16s → **8m57s (-79.2 s / -12.9%)** | 1533 s → 579 s (2.6×) | `libExec.a` 2450 MB → 641 MB | | Exprs unity | 8m57s → **7m57s (-60.0 s / -11.2%)** | 2147 s → 1333 s | `libExprs.a` 2721 MB → 1392 MB | Jumbo-TU envelope: the largest Exec unity TU compiles in 32 s / 2.6 GB RSS standalone, the largest Exprs one in 25 s / 1.95 GB — both below the largest existing individual TU in the tree (3.9 GB), so `-jN` memory envelopes are unchanged. ### Risk and verification - **Unity changes TU grouping only.** The code commits riding along are hygiene: constants deduplicated with identical values, one constant scoped into its class, two renames, one `#pragma once`. No logic change. - **Archive symbol parity** (checked on the development branch): Exec keeps all external defined symbols — three weak linkonce_odr template instantiations dedup away, which is the point of unity, not a loss. Exprs likewise (one weak instantiation dedups; the `DictGetManyState` rename carries its `shared_ptr` machinery under the new name). - **This exact branch, rebased onto current master, full BE build from scratch** (macOS arm64, clang 20, `ENABLE_PCH=ON`, `ENABLE_UNITY_BUILD=ON`): **7981/7981 ninja edges, zero failures, `doris_be` links (325 MB).** Exec produces exactly 14 unity TUs and Exprs exactly 31, as advertised. This includes `pipeline/rec_cte_shared_state.cpp`, added upstream after the waves were measured — it lands inside an Exec unity batch via the existing `GLOB_RECURSE` with zero CMakeLists edits, which is the intended maintenance story. - **The first CI round of this PR did its job and caught two issues; both are fixed in the current revision.** 1. *BE UT lane, duplicate symbols at link*: three test files compile a src `.cpp` a second time by `#include`ing it (`function_variant_element_test`, `function_uuid_test`, `function_json_object_flatten_test`). Pre-unity this linked only by archive-member selectivity: the test object defines the symbols first and the library member is never pulled. A unity batch, however, is pulled in for its *siblings* and brings a second strong definition. The three `#include`d files (`function_variant_element.cpp`, `uuid.cpp`, `function_jsonb_transform.cpp`) are now `SKIP_UNITY_BUILD_INCLUSION` — individual archive members restore exactly the shadowing semantics master links with today. 2. *Performance lane (the one gcc lane), `-Werror=subobject-linkage`*: `function_variant_element_v2.cpp` held `OwnedPathSegment` in an anonymous namespace as a field type of the externally-visible `...Path::Impl`. gcc only raises `-Wsubobject-linkage` when the definition sits in an `#include`d file — which is what unity turns a `.cpp` into; clang has no such warning, so every local build was green. The struct moves to namespace scope (name unique to the TU); fixed at the source rather than SKIPped. - **The OFF path, checked on the same tree**: reconfiguring with `ENABLE_UNITY_BUILD=OFF` drops **all 51** `unity_*.cxx` entries from `compile_commands.json` (Exec 14, Exprs 31, the #66712 pilots 6) and the TU count goes 8464 → 9042 — the batches return to exactly their 629 member files. Reconfiguring back ON restores exactly the same 51 batches. The switch semantics themselves (including winning over a stale `CMAKE_UNITY_BUILD` cache) were established in #66712. - **These two targets have been building as unity TUs on the development branch since 2026-08-08**, through repeated full-tree builds and the BE UT builds that verified #66672 (the UT binaries link against these same target libraries). - The `file_scanner_v2_test.cpp` hunk was compile-verified standalone against this branch (`-fsyntax-only` with the test TU's full include closure). ### Proactive disclosure - **Cross-platform is the blind spot, closed by this PR's own CI** — every local build and measurement above is macOS arm64 + clang 20. With `ENABLE_UNITY_BUILD` defaulting ON since #66712, the Linux compile lanes and every regression pipeline in this PR's CI run against unity Exec/Exprs — that is the validation, and the first round proved it works: the BE UT and gcc lanes each caught one real unity interaction (detailed above), fixed in this revision. Escape hatches, in order: per-user `ENABLE_UNITY_BUILD=OFF`, per-file `SKIP_UNITY_BUILD_INCLUSION`, or a one-line default flip. - **The incremental-rebuild trade-off is real**: touching one `.cpp` inside a batch recompiles the whole batch (≤12 sources for Exec, ≤8 for Exprs; a batch compiles in ~25–32 s). This is why the heaviest, most-edited TUs (`operator.cpp`, `hashjoin_build_sink.cpp`, the aggregate families, `in.cpp`, `multiply.cpp`, …) are deliberately SKIPped and keep per-file granularity, and `ENABLE_UNITY_BUILD=OFF` restores it everywhere. - **The SKIP lists are coverage policy, not leftovers**: 7 Exec + 38 Exprs files stay individual on purpose — generated parsers (flex/bison/gperf), files whose file-scope macros would leak into siblings, and the heavy codegen TUs where merging saves no closure parse worth the jumbo-TU cost. A future file whose file-scope symbols clash inside a unity TU opts out the same one-line way. - Unity covers the *glue* of these targets, not the codegen-heavy families — the 30 heavy Exprs SKIPs mean the headline per-target ratios (2.6× Exec slot time) are earned on the batched part; the SKIPped monsters keep their cost and their per-file granularity.

morningman and others added 9 commits August 11, 2026 22:57

morningman requested review from Gabriel39, airborne12, csun5285, eldenmoon, gavinchou and yiguolei as code owners August 11, 2026 16:21

morningman requested a review from luwei16 as a code owner August 12, 2026 00:16

morningman force-pushed the be-build-opt-1-include-cuts branch from 06351a4 to 0312902 Compare August 12, 2026 00:22

gavinchou approved these changes Aug 12, 2026

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2026

github-actions Bot added the reviewed label Aug 12, 2026

Gabriel39 approved these changes Aug 13, 2026

View reviewed changes

eldenmoon approved these changes Aug 13, 2026

View reviewed changes

morningman merged commit 1cd984a into apache:master Aug 13, 2026
33 of 35 checks passed

This was referenced Aug 13, 2026

[opt](build) Add ENABLE_UNITY_BUILD and pilot unity builds on three glue targets #66712

Merged

[Tracking] BE build-time optimization: header-closure surgery, unity builds, and instantiation cuts #66715

Open

morningman mentioned this pull request Aug 14, 2026

[opt](build) Enable unity builds for the Exec and Exprs targets #66776

Merged

5 tasks

Uh oh!

Conversation

morningman commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Wave 1 — three hot edges (Prepare cutting three hot edges + Cut three hot include edges)

Wave 2 — two instantiation amplifiers: RLE and FMT_COMPILE

Wave 3 — the DataVariants amplifier behind dependency.h

One CI-critical commit rides along

Benefit

Head-to-head on exactly this PR's content

Per-wave numbers from the development branch

Risk and verification

Proactive disclosure

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Aug 11, 2026

Uh oh!

morningman commented Aug 11, 2026

Uh oh!

morningman commented Aug 11, 2026

Uh oh!

morningman commented Aug 12, 2026

Uh oh!

morningman commented Aug 12, 2026

Uh oh!

morningman commented Aug 12, 2026

Uh oh!

morningman commented Aug 12, 2026

Uh oh!

hello-stephen commented Aug 12, 2026

Uh oh!

hello-stephen commented Aug 12, 2026

Uh oh!

hello-stephen commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

morningman commented Aug 12, 2026

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Aug 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

morningman commented Aug 11, 2026 •

edited

Loading

Wave 1 — three hot edges (`Prepare cutting three hot edges` + `Cut three hot include edges`)

Wave 3 — the DataVariants amplifier behind `dependency.h`