Skip to content

[improvement](be) Optimize projected fixed-width Parquet predicate filtering#65934

Merged
Gabriel39 merged 2 commits into
apache:masterfrom
Gabriel39:agent/parquet-projected-raw-filter-project
Jul 23, 2026
Merged

[improvement](be) Optimize projected fixed-width Parquet predicate filtering#65934
Gabriel39 merged 2 commits into
apache:masterfrom
Gabriel39:agent/parquet-projected-raw-filter-project

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Related PR: #65921

Problem Summary:

When a fixed-width Parquet predicate column was also projected, Doris materialized all selected predicate values and compacted the complete column after filtering. The native decoder had already produced the values needed by the raw predicate, so this added an avoidable materialize-and-compact pass. The overhead is visible in TPC-DS Q88.

What is changed?

Evaluate eligible predicates on decoded fixed-width values and append only matching values to the projected column in the same decoder pass.

  • Generalize the PLAIN-specific consumer and reader APIs into a fixed-width raw filter-and-project path.
  • Support PLAIN and BYTE_STREAM_SPLIT for identity-width INT32, INT64, FLOAT, and DOUBLE values.
  • Support DELTA_BINARY_PACKED for INT32 and INT64 values.
  • Prevalidate advertised chunk encodings before consuming definition levels.
  • Reject an unexpected unsupported late-page encoding instead of attempting a fallback after cursor progress.
  • Keep the existing dictionary-id filter and dictionary materialization path unchanged.

The raw decoder cannot rewind after predicate evaluation. Therefore, when the predicate column is projected, survivors must be appended before the encoded values and definition-level cursor are consumed.

Predicate-only columns retain their placeholder behavior. Nested columns, converted logical types, residual/delete predicates, unsupported expressions, and unsupported encodings continue to use the existing materializing fallback.

Microbenchmark

The reader microbenchmark from #65921 was run with a Release build, warm fixture cache, CPU 8, a one-second minimum time, and 10 repetitions.

The matrix adds BYTE_STREAM_SPLIT and DELTA_BINARY_PACKED coverage with:

  • 10% alternating NULLs;
  • predicate-only and predicate-projected modes;
  • 1%, 10%, 50%, and 90% selectivity.

All 16 combinations completed with the expected raw and selected row counts.

Projected, 10% selectivity:

Encoding CPU ns/raw row, master CPU ns/raw row, PR Improvement CPU CV master / PR
BYTE_STREAM_SPLIT 36.41 33.43 8.2% 1.25% / 1.47%
DELTA_BINARY_PACKED 38.75 36.84 4.9% 5.40% / 2.94%

The Delta baseline ran under sustained host load, so its result should be treated as directional.

The original projected PLAIN results remain:

Selectivity CPU ns/raw row, master CPU ns/raw row, PR Improvement
1% 32.80 30.80 6.1%
10% 33.95 31.82 6.3%
50% 37.76 35.05 7.2%
90% 42.06 37.66 10.5%

Release note

None

Check List (For Author)

  • Test: Unit Test and Manual test
    • 17 targeted ASAN unit tests covering raw expression evaluation, nullable mapping, projected PLAIN/BSS/Delta scans, unsupported-type fallback, mixed-page safety, residual predicates, and benchmark matrix constraints
    • Release benchmark target linked successfully
    • 16-case BSS/Delta reader microbenchmark matrix
    • Before/after projected-filter microbenchmarks
    • clang-format 16, check-format, and git diff --check
    • clang-tidy was attempted but blocked by existing master/toolchain errors: unmatched NOLINTEND in be/src/core/types.h and missing stddef.h from the configured toolchain
  • Behavior changed: No. This extends the existing raw predicate optimization to equivalent fixed-width decoder output.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65921

Problem Summary: Projected PLAIN predicate columns could not use the raw fixed-value predicate path. Doris materialized every selected predicate value and later compacted the complete column even though the raw decoder had already identified survivors. Append matching raw values to the projected column in the same decoder pass while preserving predicate-only placeholders and all existing fallback conditions. The Parquet reader microbenchmark from apache#65921 improves median CPU time per raw row by 6.1%, 6.3%, 7.2%, and 10.5% at 1%, 10%, 50%, and 90% selectivity respectively.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Manual test
    - Nine targeted ASAN unit tests, including projected/predicate-only/fallback/residual/nullable and benchmark scenario coverage
    - Release warm-cache microbenchmark with 10 repetitions on CPU 8
    - clang-format and check-format
- Behavior changed: No. This only changes how eligible projected PLAIN predicate survivors are materialized.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39
Gabriel39 marked this pull request as ready for review July 23, 2026 01:40
@Gabriel39
Gabriel39 requested a review from yiguolei as a code owner July 23, 2026 01:40
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29505 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93b1a6f3fd16dfab817146a2cec0168284d8d382, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17745	4221	4188	4188
q2	2081	326	198	198
q3	10217	1431	851	851
q4	4680	466	338	338
q5	7566	847	576	576
q6	179	174	133	133
q7	754	804	621	621
q8	9332	1582	1568	1568
q9	5630	4338	4318	4318
q10	6787	1725	1478	1478
q11	508	368	320	320
q12	779	585	459	459
q13	18089	3401	2725	2725
q14	267	260	240	240
q15	q16	792	775	734	734
q17	1035	1008	933	933
q18	6901	5725	5596	5596
q19	1357	1369	1028	1028
q20	836	685	590	590
q21	5926	2720	2322	2322
q22	441	354	289	289
Total cold run time: 101902 ms
Total hot run time: 29505 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4422	4329	4309	4309
q2	316	319	212	212
q3	4660	4945	4451	4451
q4	2054	2140	1367	1367
q5	4375	4256	4322	4256
q6	233	177	126	126
q7	1767	2027	1877	1877
q8	2697	2283	2156	2156
q9	8084	8184	7682	7682
q10	4706	4759	4241	4241
q11	578	436	387	387
q12	737	767	550	550
q13	3370	3538	2984	2984
q14	296	308	275	275
q15	q16	714	726	629	629
q17	1379	1322	1323	1322
q18	8046	7597	7425	7425
q19	1169	1138	1055	1055
q20	2223	2206	1949	1949
q21	5213	4583	4460	4460
q22	555	452	392	392
Total cold run time: 57594 ms
Total hot run time: 52105 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: the projected fixed-width PLAIN fast path has two patch-scoped P2 issues. It can degenerate into per-survivor virtual appends for fragmented match patterns, and its fused decode/projection work is omitted from the native decode/materialization stage timers.

Critical checkpoint conclusions:

  • Concurrency and thread safety: no new thread, shared mutable state, lock, or atomic behavior is introduced; the reader scratch and destination column are mutated synchronously within one scan path.
  • Error handling and lifecycle: Status values are propagated. Capability fallback occurs before value consumption, while incompatibility/corruption after progress remains terminal; no successful retry can expose a partially appended column.
  • Memory safety and ownership: the destination is the mutable file-type column owned by the block. Exact physical/logical identity gates and nullable raw insertion preserve representation and null-map alignment. No lifetime or ownership defect was found, although the repeated growth operations are the performance issue called out inline.
  • Data correctness and nullable handling: the compact filter and projected values come from the same ordered physical match stream. NULL remapping, prior sparse selections, later predicate/delete compaction, page/range fragmentation, and widened pending slices remain aligned. Converted or unsupported types/encodings fall back before raw consumption.
  • Observability: incomplete. Eligible projected direct scans underreport the separately published DecodeValueTime and MaterializationTime stages; see the inline finding.
  • Compatibility, configuration, persistence, and data writes: no configuration, serialized protocol, FE/BE variable, transaction, persistence, or data-write contract changes are involved. Predicate-only, dictionary, residual/delete, unsupported-type, and ordinary materialization paths were checked.
  • Tests and performance: the added unit and scan tests cover sparse nullable mapping and the projected happy path, but both use a single contiguous survivor run and miss the alternating-match hot-loop case. I did not run builds or tests because this review environment explicitly prohibits them; validation was static against the authoritative bundled diff.

No user-specific focus was supplied. Review completion status: formally incomplete under the supplied convergence contract. A new valuable candidate appeared in round 3, and the contract caps review at three rounds; both current candidates were independently validated, deduplicated, and included here, but a fourth convergence round was not permitted.

++row;
}
if (row != run_begin) {
_projected_column->insert_many_raw_data(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep native timing complete on projected direct scans

Projected predicate columns now append survivors in this branch and no longer pass through materialize_values(), which is where decode_value_time and materialization_time are incremented. filter_plain_values() calls decode_selected_fixed_values() and this line copies survivors, but neither contributes to the separately published DecodeValueTime/MaterializationTime, so every eligible projected direct scan underreports both stages. Please mirror the existing timing convention: an inclusive batch/span decode timer around the direct decoder/consumer call and a nested materialization timer around projected survivor writes, avoiding per-row timers.

_data_type, _column_id,
_matches->data() + old_size));
}
if (_projected_column != nullptr) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Avoid per-match-run column appends

With alternating predicate results, this loop invokes virtual insert_many_raw_data() once per survivor (N/2 calls), and each call resizes the vector before copying only 4/8 bytes; nullable output also grows its null map and nested column each time. The previous projected path materialized the batch and used vectorized filtering, so ordinary unsorted values can turn this fast path into thousands of virtual calls and tiny copies per batch. Both added tests have a single contiguous survivor run. Please reserve/grow once and compact survivors into the destination in a bounded batch/span operation, and cover an alternating-match case.

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65921

Problem Summary: Generalize projected raw filter-and-project from PLAIN to eligible fixed-width encodings. BYTE_STREAM_SPLIT now supports identity-width INT32, INT64, FLOAT, and DOUBLE values; DELTA_BINARY_PACKED supports INT32 and INT64. Keep dictionary filtering on its existing specialized path, reject unsupported mixed page encodings before unsafe fallback, and extend the reader microbenchmark matrix across nullable predicate-only/projected scans and 1%, 10%, 50%, and 90% selectivity.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Manual test
    - 17 targeted ASAN unit tests covering encodings, nullable mapping, projection, fallback, mixed-page safety, and scenario matrix
    - Release microbenchmark matrix with 16 BSS/Delta predicate scenarios, 10 repetitions on CPU 8
    - Before/after Release microbench for projected 10% selectivity
    - clang-format 16, check-format, and git diff --check
- Behavior changed: No. This extends the existing raw predicate optimization to equivalent fixed-width decoder outputs.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (42/42) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.89% (24453/42242)
Line Coverage 41.92% (243505/580913)
Region Coverage 37.80% (193337/511436)
Branch Coverage 38.89% (86968/223642)

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39 Gabriel39 changed the title [improvement](be) Project PLAIN predicate survivors during raw filtering [improvement](be) Project fixed-width Parquet predicate survivors during raw filtering Jul 23, 2026
@Gabriel39 Gabriel39 changed the title [improvement](be) Project fixed-width Parquet predicate survivors during raw filtering [improvement](be) Optimize projected fixed-width Parquet predicate filtering Jul 23, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 30024 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit db67a99d19e74489b699f5c282897fba62327e3e, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17741	4276	4246	4246
q2	2071	333	200	200
q3	10248	1446	837	837
q4	4684	474	338	338
q5	7513	882	563	563
q6	193	181	143	143
q7	772	825	632	632
q8	9341	1686	1564	1564
q9	5613	4358	4341	4341
q10	6762	1748	1500	1500
q11	511	365	323	323
q12	756	592	476	476
q13	18159	3428	2717	2717
q14	276	275	252	252
q15	q16	790	781	705	705
q17	1041	1014	1042	1014
q18	7022	5693	5648	5648
q19	1311	1188	1094	1094
q20	845	726	601	601
q21	6018	2585	2525	2525
q22	433	374	305	305
Total cold run time: 102100 ms
Total hot run time: 30024 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4533	4479	4464	4464
q2	304	326	217	217
q3	4678	4972	4396	4396
q4	2095	2170	1495	1495
q5	4433	4304	4309	4304
q6	228	180	135	135
q7	1867	2099	1736	1736
q8	2623	2360	2356	2356
q9	7992	8015	7920	7920
q10	4695	4651	4220	4220
q11	565	404	417	404
q12	765	782	547	547
q13	3370	3600	2929	2929
q14	282	298	270	270
q15	q16	724	744	646	646
q17	1414	1381	1378	1378
q18	7958	7423	7242	7242
q19	1182	1154	1124	1124
q20	2243	2210	1923	1923
q21	5340	4719	4492	4492
q22	544	502	431	431
Total cold run time: 57835 ms
Total hot run time: 52629 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on the current head: the core fixed-width filter/project mapping and cursor lifecycle appear coherent, but four patch-scoped P2 issues remain in compatibility, count observability, end-to-end proof, and the benchmark registration contract. I did not duplicate the two existing threads for per-match-run appends and missing decode/materialization timers.

Critical checkpoint conclusions:

  • Goal and proof: the production path does evaluate and project eligible PLAIN/BSS/Delta survivors in one decoder pass, and static tracing found its row mapping/cursor behavior coherent. The added tests do not prove the newly accepted encoding/type/null/page matrix end to end; see inline.
  • Scope and clarity: the production change is focused around the scheduler/native reader/decoder interfaces, but the profile compatibility contract, current design text, and benchmark matrix contract were not updated consistently.
  • Concurrency: no new thread, shared mutable state, lock, or atomic behavior is introduced; reader scratch and the destination column remain scan-local and synchronous.
  • Lifecycle/static initialization: reader-owned scratch is bounded by the batch/Row Group lifecycle, and successful output ownership follows existing mutable-column/COW rules. No new static/global initialization or circular ownership is introduced.
  • Configuration: no configuration item is added or changed.
  • Compatibility: no storage/protocol/function-symbol rolling-upgrade issue was found, but removing the published PlainPredicateDirectBatches/Rows RuntimeProfile keys without aliases is an operational compatibility break; see inline.
  • Parallel paths: PLAIN, BYTE_STREAM_SPLIT, DELTA_BINARY_PACKED, ordinary materialization, dictionary filtering, predicate-only/projected columns, residual/delete predicates, lazy outputs, page/range gaps, and fallback/error paths were traced. The direct count bookkeeping remains incomplete compared with ordinary/dictionary selection; see inline.
  • Conditions and error handling: eligibility is limited to exact identity-width types, advertised encodings are prevalidated before cursor consumption, actual pages are rechecked, and late incompatibility returns corruption rather than retrying an advanced cursor. Status propagation is intact.
  • Tests and results: the changed C++ tests have coherent expected rows for the cases they cover, but the BSS/Delta fixture does not assert its emitted encoding and the accepted matrix lacks value-checked nullable/fragmented/page coverage. No generated .out file is changed. I did not run builds or tests because the governing review prompt prohibits them; validation was static against the authoritative bundled diff.
  • Observability: the newly routed direct traffic omits NativeReadCalls and ReaderSelectRows; the separate decode/materialization timer omission is already covered by an existing inline thread.
  • Transactions, persistence, data writes, and FE-BE propagation: none are involved.
  • Performance: the existing append-fragmentation thread covers the surviving hot-loop issue. The reported Delta comparison is directional under the benchmark guide's variance rule, as the PR already states. No additional nonduplicate CPU, memory, or lifetime issue survived review.
  • Other: the deduplicated reader matrix now contains 151 cases while the current smoke contract still requires 137; see inline.

No user-specific focus was supplied. Review completion status: complete and converged after Round 2; all suspicious points were either accepted below, deduplicated to the two existing discussions, or dismissed with concrete code-path evidence.

plain_predicate_direct_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "PlainPredicateDirectRows",
TUnit::UNIT, parquet_profile, 1);
fixed_width_predicate_direct_batches = ADD_CHILD_COUNTER_WITH_LEVEL(
profile, "FixedWidthPredicateDirectBatches", TUnit::UNIT, parquet_profile, 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the profile keys or update their compatibility contract

This removes the only PlainPredicateDirectBatches/Rows registrations, but the mandatory Parquet scan design still describes the predicate-only PLAIN path and tells operators/profile parsers to read those exact keys. After this change those lookups silently disappear, even for the existing PLAIN workload. Please preserve the old keys as deprecated aliases (or retain them with broadened semantics) and update the current-pipeline design to cover projected PLAIN/BSS/Delta behavior and the new names.

RETURN_IF_ERROR(_native_reader->read_plain_filter(
conjuncts, column_id, filter, static_cast<size_t>(rows - *rows_read), &loop_filter,
&loop_rows, &eof, &loop_used));
RETURN_IF_ERROR(_native_reader->read_fixed_width_filter(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the direct reader's count profile complete

read_with_filter() counts each native reader call in NativeReadCalls, and both ordinary and dictionary select() paths add their input selection to ReaderSelectRows. This fixed-width loop does neither, so a predicate-only direct scan can report zero native calls/selected rows for its only active reader; projected scans undercount them behind unrelated lazy-reader activity. This is separate from the existing decode/materialization timer thread. Please mirror both count updates here and assert them in the direct-path profile tests.


TEST(ParquetV2NativeDecoderTest, RawFixedFilterSupportsIdentityWidthEncodingTypes) {
using Reader = ColumnChunkReader<false, false>;
EXPECT_TRUE(Reader::supports_raw_fixed_filter_encoding(tparquet::Encoding::BYTE_STREAM_SPLIT,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Prove the newly accepted encoding/type matrix end to end

These assertions only test the whitelist. The new scan fixtures request BSS/Delta from Arrow but never verify the emitted footer/page encoding, and PLAIN increments the same direct counter, so both tests can stay green without exercising the new decoders. They also cover only required INT32 with one contiguous survivor run; the nullable benchmark checks row counts, not projected values. Please add value-checked filter/project coverage for every accepted BSS/Delta type (including nullable/fragmented and page transitions) and assert the fixture's actual encoding before treating the benchmark numbers as validation.

scenario.operation = ReaderOperation::PREDICATE_SCAN;
add(scenario);
}
for (const auto encoding : {Encoding::BYTE_STREAM_SPLIT, Encoding::DELTA_BINARY_PACKED}) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Update and assert the reader registration total

This loop contributes 14 unique registrations, not 16, because each encoding's 10%-selectivity projected case already comes from the preceding loop. The total therefore grows from 137 to 151, while be/benchmark/parquet/AGENTS.md still tells smoke reviewers to expect 137 and to reject a result with a different count. Please assert reader_scenarios().size() == 151 and update the current listing/matrix counts (leaving the historical validation record tied to its old commit unchanged).

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 177426 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit db67a99d19e74489b699f5c282897fba62327e3e, data reload: false

query5	4325	639	476	476
query6	491	235	222	222
query7	4863	591	349	349
query8	339	188	177	177
query9	8798	4026	4033	4026
query10	475	381	296	296
query11	5941	2341	2113	2113
query12	165	104	101	101
query13	1258	592	449	449
query14	6207	5245	4959	4959
query14_1	4260	4236	4260	4236
query15	224	205	174	174
query16	1028	495	488	488
query17	1136	754	577	577
query18	2446	485	359	359
query19	233	193	153	153
query20	121	111	104	104
query21	243	160	138	138
query22	13592	13575	13367	13367
query23	17402	16476	16124	16124
query23_1	16293	16158	16194	16158
query24	7510	1769	1287	1287
query24_1	1287	1277	1262	1262
query25	565	486	391	391
query26	1375	368	217	217
query27	2530	575	386	386
query28	4435	1992	1961	1961
query29	1073	599	450	450
query30	342	257	228	228
query31	1115	1085	971	971
query32	109	63	59	59
query33	521	312	238	238
query34	1205	1106	656	656
query35	749	766	661	661
query36	1189	1189	1073	1073
query37	152	109	98	98
query38	1873	1689	1660	1660
query39	868	894	838	838
query39_1	825	841	832	832
query40	242	163	136	136
query41	65	62	61	61
query42	98	96	92	92
query43	325	331	277	277
query44	1416	762	773	762
query45	195	184	173	173
query46	1080	1222	748	748
query47	2114	2104	1993	1993
query48	407	388	281	281
query49	568	419	307	307
query50	1066	413	347	347
query51	10687	10621	10460	10460
query52	83	86	76	76
query53	256	262	200	200
query54	268	231	221	221
query55	74	70	64	64
query56	303	288	291	288
query57	1332	1320	1209	1209
query58	299	288	251	251
query59	1549	1625	1412	1412
query60	310	276	259	259
query61	151	149	146	146
query62	541	498	432	432
query63	242	207	208	207
query64	2808	1050	834	834
query65	4758	4651	4633	4633
query66	1816	518	368	368
query67	29236	29382	29102	29102
query68	3166	1498	1027	1027
query69	394	309	264	264
query70	1037	985	948	948
query71	365	327	307	307
query72	2963	2668	2408	2408
query73	837	760	427	427
query74	5059	4936	4708	4708
query75	2530	2514	2128	2128
query76	2333	1157	744	744
query77	346	373	284	284
query78	11853	11892	11301	11301
query79	1264	1112	719	719
query80	649	555	479	479
query81	452	327	286	286
query82	563	156	120	120
query83	399	331	291	291
query84	275	163	132	132
query85	927	576	515	515
query86	361	303	278	278
query87	1840	1819	1764	1764
query88	3645	2769	2746	2746
query89	429	385	326	326
query90	1994	203	193	193
query91	203	194	161	161
query92	60	62	53	53
query93	1594	1517	980	980
query94	554	342	341	341
query95	796	514	567	514
query96	1112	768	357	357
query97	2629	2635	2537	2537
query98	220	208	203	203
query99	1099	1113	974	974
Total cold run time: 262205 ms
Total hot run time: 177426 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit db67a99d19e74489b699f5c282897fba62327e3e, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.06	0.05
query3	0.26	0.13	0.16
query4	1.61	0.15	0.14
query5	0.23	0.22	0.23
query6	1.29	1.06	1.03
query7	0.03	0.00	0.00
query8	0.06	0.04	0.04
query9	0.38	0.30	0.33
query10	0.54	0.59	0.58
query11	0.20	0.14	0.13
query12	0.18	0.15	0.15
query13	0.48	0.48	0.48
query14	1.02	1.01	0.99
query15	0.61	0.59	0.59
query16	0.32	0.32	0.33
query17	1.06	1.07	1.10
query18	0.23	0.21	0.20
query19	2.05	1.94	1.97
query20	0.02	0.01	0.01
query21	15.45	0.20	0.14
query22	4.96	0.05	0.06
query23	16.13	0.31	0.13
query24	2.96	0.42	0.30
query25	0.11	0.05	0.04
query26	0.73	0.20	0.16
query27	0.04	0.04	0.05
query28	3.53	0.90	0.51
query29	12.47	4.22	3.28
query30	0.27	0.16	0.15
query31	2.77	0.58	0.32
query32	3.25	0.58	0.48
query33	3.14	3.27	3.24
query34	15.72	4.24	3.51
query35	3.52	3.52	3.50
query36	0.56	0.44	0.44
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.16	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.82 s
Total hot run time: 25.05 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 92.52% (99/107) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.88% (24450/42243)
Line Coverage 41.89% (243343/580952)
Region Coverage 37.80% (193349/511450)
Branch Coverage 38.89% (86977/223668)

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2026
@Gabriel39
Gabriel39 merged commit 1871546 into apache:master Jul 23, 2026
34 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 23, 2026
…ltering (apache#65934)

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65921

Problem Summary:

When a fixed-width Parquet predicate column was also projected, Doris
materialized all selected predicate values and compacted the complete
column after filtering. The native decoder had already produced the
values needed by the raw predicate, so this added an avoidable
materialize-and-compact pass. The overhead is visible in TPC-DS Q88.

### What is changed?

Evaluate eligible predicates on decoded fixed-width values and append
only matching values to the projected column in the same decoder pass.

- Generalize the PLAIN-specific consumer and reader APIs into a
fixed-width raw filter-and-project path.
- Support PLAIN and BYTE_STREAM_SPLIT for identity-width INT32, INT64,
FLOAT, and DOUBLE values.
- Support DELTA_BINARY_PACKED for INT32 and INT64 values.
- Prevalidate advertised chunk encodings before consuming definition
levels.
- Reject an unexpected unsupported late-page encoding instead of
attempting a fallback after cursor progress.
- Keep the existing dictionary-id filter and dictionary materialization
path unchanged.

The raw decoder cannot rewind after predicate evaluation. Therefore,
when the predicate column is projected, survivors must be appended
before the encoded values and definition-level cursor are consumed.

Predicate-only columns retain their placeholder behavior. Nested
columns, converted logical types, residual/delete predicates,
unsupported expressions, and unsupported encodings continue to use the
existing materializing fallback.

### Microbenchmark

The reader microbenchmark from apache#65921 was run with a Release build, warm
fixture cache, CPU 8, a one-second minimum time, and 10 repetitions.

The matrix adds BYTE_STREAM_SPLIT and DELTA_BINARY_PACKED coverage with:

- 10% alternating NULLs;
- predicate-only and predicate-projected modes;
- 1%, 10%, 50%, and 90% selectivity.

All 16 combinations completed with the expected raw and selected row
counts.

Projected, 10% selectivity:

| Encoding | CPU ns/raw row, master | CPU ns/raw row, PR | Improvement |
CPU CV master / PR |
|---|---:|---:|---:|---:|
| BYTE_STREAM_SPLIT | 36.41 | 33.43 | 8.2% | 1.25% / 1.47% |
| DELTA_BINARY_PACKED | 38.75 | 36.84 | 4.9% | 5.40% / 2.94% |

The Delta baseline ran under sustained host load, so its result should
be treated as directional.

The original projected PLAIN results remain:

| Selectivity | CPU ns/raw row, master | CPU ns/raw row, PR |
Improvement |
|---:|---:|---:|---:|
| 1% | 32.80 | 30.80 | 6.1% |
| 10% | 33.95 | 31.82 | 6.3% |
| 50% | 37.76 | 35.05 | 7.2% |
| 90% | 42.06 | 37.66 | 10.5% |

### Release note

None

### Check List (For Author)

- Test: Unit Test and Manual test
- 17 targeted ASAN unit tests covering raw expression evaluation,
nullable mapping, projected PLAIN/BSS/Delta scans, unsupported-type
fallback, mixed-page safety, residual predicates, and benchmark matrix
constraints
    - Release benchmark target linked successfully
    - 16-case BSS/Delta reader microbenchmark matrix
    - Before/after projected-filter microbenchmarks
    - clang-format 16, check-format, and `git diff --check`
- clang-tidy was attempted but blocked by existing master/toolchain
errors: unmatched `NOLINTEND` in `be/src/core/types.h` and missing
`stddef.h` from the configured toolchain
- Behavior changed: No. This extends the existing raw predicate
optimization to equivalent fixed-width decoder output.
- Does this need documentation: No

(cherry picked from commit 1871546)
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 23, 2026
…ltering (apache#65934)

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65921

Problem Summary:

When a fixed-width Parquet predicate column was also projected, Doris
materialized all selected predicate values and compacted the complete
column after filtering. The native decoder had already produced the
values needed by the raw predicate, so this added an avoidable
materialize-and-compact pass. The overhead is visible in TPC-DS Q88.

### What is changed?

Evaluate eligible predicates on decoded fixed-width values and append
only matching values to the projected column in the same decoder pass.

- Generalize the PLAIN-specific consumer and reader APIs into a
fixed-width raw filter-and-project path.
- Support PLAIN and BYTE_STREAM_SPLIT for identity-width INT32, INT64,
FLOAT, and DOUBLE values.
- Support DELTA_BINARY_PACKED for INT32 and INT64 values.
- Prevalidate advertised chunk encodings before consuming definition
levels.
- Reject an unexpected unsupported late-page encoding instead of
attempting a fallback after cursor progress.
- Keep the existing dictionary-id filter and dictionary materialization
path unchanged.

The raw decoder cannot rewind after predicate evaluation. Therefore,
when the predicate column is projected, survivors must be appended
before the encoded values and definition-level cursor are consumed.

Predicate-only columns retain their placeholder behavior. Nested
columns, converted logical types, residual/delete predicates,
unsupported expressions, and unsupported encodings continue to use the
existing materializing fallback.

### Microbenchmark

The reader microbenchmark from apache#65921 was run with a Release build, warm
fixture cache, CPU 8, a one-second minimum time, and 10 repetitions.

The matrix adds BYTE_STREAM_SPLIT and DELTA_BINARY_PACKED coverage with:

- 10% alternating NULLs;
- predicate-only and predicate-projected modes;
- 1%, 10%, 50%, and 90% selectivity.

All 16 combinations completed with the expected raw and selected row
counts.

Projected, 10% selectivity:

| Encoding | CPU ns/raw row, master | CPU ns/raw row, PR | Improvement |
CPU CV master / PR |
|---|---:|---:|---:|---:|
| BYTE_STREAM_SPLIT | 36.41 | 33.43 | 8.2% | 1.25% / 1.47% |
| DELTA_BINARY_PACKED | 38.75 | 36.84 | 4.9% | 5.40% / 2.94% |

The Delta baseline ran under sustained host load, so its result should
be treated as directional.

The original projected PLAIN results remain:

| Selectivity | CPU ns/raw row, master | CPU ns/raw row, PR |
Improvement |
|---:|---:|---:|---:|
| 1% | 32.80 | 30.80 | 6.1% |
| 10% | 33.95 | 31.82 | 6.3% |
| 50% | 37.76 | 35.05 | 7.2% |
| 90% | 42.06 | 37.66 | 10.5% |

### Release note

None

### Check List (For Author)

- Test: Unit Test and Manual test
- 17 targeted ASAN unit tests covering raw expression evaluation,
nullable mapping, projected PLAIN/BSS/Delta scans, unsupported-type
fallback, mixed-page safety, residual predicates, and benchmark matrix
constraints
    - Release benchmark target linked successfully
    - 16-case BSS/Delta reader microbenchmark matrix
    - Before/after projected-filter microbenchmarks
    - clang-format 16, check-format, and `git diff --check`
- clang-tidy was attempted but blocked by existing master/toolchain
errors: unmatched `NOLINTEND` in `be/src/core/types.h` and missing
`stddef.h` from the configured toolchain
- Behavior changed: No. This extends the existing raw predicate
optimization to equivalent fixed-width decoder output.
- Does this need documentation: No

(cherry picked from commit 1871546)
yiguolei pushed a commit that referenced this pull request Jul 24, 2026
## Summary

Backport the requested Apache Doris PRs to `branch-4.1` in their master
merge order:

1. #63648
2. #64033
3. #64263
4. #64315
5. #63825
6. #65332
7. #65354
8. #65094
9. #65401
10. #65502
11. #65135
12. #65742
13. #65709
14. #65548
15. #65676
16. #65770
17. #65759
18. #65782
19. #65784
20. #65674
21. #65921
22. #65934
23. #65925
24. #65931

#64263, #65354, #65094, #65502, and #65770 are already effectively
present on current `branch-4.1`, so this branch does not duplicate those
commits. The remaining requested changes are represented by ordered
backport commits followed by branch-4.1 compatibility fixes.

## Compatibility notes

- Rebased onto the latest `upstream/branch-4.1` and retained both sides
of the Iceberg scan conflict resolution.
- Keep `be/src/format_v2` and `be/test/format_v2` exactly aligned with
current `upstream/master`, as requested.
- Apply branch-4.1 compatibility adaptations outside the format_v2
paths.
- Preserve the selected PR behavior while avoiding dependencies on
unrelated master-only changes.
- #65921 supplies the Parquet benchmark scenario header required by
`parquet_benchmark_scenarios_test.cpp`, fixing the BE UT compile
failure.
- Adapt branch-4.1 `VRuntimeFilterWrapper` at the master TableReader
boundary so FileScannerV2 residual predicates keep globally rewritten
slot indexes.
- Keep non-transactional JDBC V2 connections usable by avoiding the
unsupported auto-commit transition before Hikari invalidates the proxy.

## FE UT fix

- Widen the auto-profile test timing margins to prevent CI scheduling
delays from selecting the wrong threshold branch.
- Keep the table-filter correctness assertions and use broader
performance guardrails suitable for shared FE UT workers.

## Verification

- Full local FE+BE build: `Successfully build Doris`.
- ASAN BE UT build completed successfully; `test/doris_be_test` linked
with exit code 0.
- Related BE tests: 382/382 passed across 32 suites, including
FileScannerV2, runtime filters, Parquet, Iceberg readers, and format_v2
coverage.
- JDBC scanner UT: 3/3 passed; Maven reactor `BUILD SUCCESS`; Checkstyle
reports 0 violations.
- ClickHouse JDBC V2 regression: 1 suite, 0 failed suites, 0 fatal
scripts.
- Iceberg branch/tag regression (`iceberg_branch_complex_queries`): 1
suite, 0 failed suites, 0 fatal scripts.
- Repository clang-format 16 check: passed for 4,191 files.
- `git diff --check`: passed.
- `be/src/format_v2` and `be/test/format_v2` have zero diff from current
`upstream/master`.

---------

Signed-off-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: daidai <changyuwei@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.4-merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants