[fix](binlog) wrap Row Binlog AFTER columns as nullable before conversion - #66553
Open
luwei16 wants to merge 4 commits into
Open
[fix](binlog) wrap Row Binlog AFTER columns as nullable before conversion#66553luwei16 wants to merge 4 commits into
luwei16 wants to merge 4 commits into
Conversation
### What problem does this PR solve? Issue Number: close apache#65951 Related PR: apache#65810 Problem Summary: Row Binlog AFTER value columns are nullable even when their source columns are NOT NULL. Complex source columns then provide no top-level null map, so nullable ARRAY, MAP, and STRUCT writers wrote payloads without advancing their physical null child. Segment finalization aborted while building the empty child ordinal index. Normalize a missing source null map to an all-non-null map for nullable destinations and verify full and partial Row Binlog writes and readback. ### Release note Fix a BE crash when Row Binlog writes NOT NULL complex columns into nullable AFTER columns. ### Check List (For Author) - Test: Unit Test - `./run-be-ut.sh --run --filter=SegmentFlusherTransformFormatTest.RowBinlogWritesNotNullComplexColumnsToNullableAfterColumns -j 16` - `./build.sh --be -j96` - Behavior changed: Yes (nullable destinations now persist an all-non-null map when source data has no explicit null map) - Does this need documentation: No
luwei16
requested review from
airborne12,
csun5285,
eldenmoon,
gavinchou,
liaoxin01 and
yiguolei
as code owners
August 6, 2026 12:20
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29206 ms |
Contributor
TPC-DS: Total hot run time: 166383 ms |
Contributor
ClickBench: Total hot run time: 23.79 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
### What problem does this PR solve? Issue Number: close apache#65951 Related PR: apache#65810 Problem Summary: Row Binlog AFTER value columns are nullable even when their source columns are NOT NULL. Complex source columns then provide no top-level null map, so nullable ARRAY, MAP, and STRUCT writers can leave the physical null child empty and abort during segment finalization. Wrap only Row Binlog normal non-key AFTER inputs before OlapBlockDataConvertor, including missing columns filled from historical rows, while preserving the original shared input and full block. Restore the generic ColumnWriter behavior so other write paths are unaffected. ### Release note Fix a BE crash when Row Binlog writes NOT NULL complex columns into nullable AFTER columns. ### Check List (For Author) - Test: Unit Test - `./run-be-ut.sh --run --filter=SegmentFlusherTransformFormatTest.RowBinlogWritesNotNullComplexColumnsToNullableAfterColumns -j 16` - `./build.sh --be -j96` - `build-support/check-format.sh` - `build-support/run-clang-tidy.sh --base HEAD^ --build-dir be/ut_build_ASAN` - Behavior changed: Yes (limited to Row Binlog AFTER conversion) - Does this need documentation: No
luwei16
changed the title
luwei16
changed the title
HappenLee
reviewed
Aug 7, 2026
| _normal_column_ids.end(); | ||
| } | ||
|
|
||
| bool RowBinlogSourceDataWriter::is_after_column(uint32_t source_cid) const { |
Contributor
There was a problem hiding this comment.
这个名字感觉不是很正确，before也得加nullable，你这也能判断出来before
Contributor
Author
There was a problem hiding this comment.
确实，这个名字不太准确。这里拿到的是源表列，它本身没有 AFTER 或 BEFORE 的区别，所以我把判断改成 is_row_binlog_value_column，nullable 包装也改成 make_nullable_row_binlog_value_column。
prepare_by_source_block() 这条路径里，源表普通 value 列接下来用于写 AFTER，所以局部变量叫 source_column_for_after。BEFORE 不走这里，而是在 _fill_before_columns() 中从历史行单独构造，那里已经处理好了 nullable。
Resolve the Row Binlog test conflict after independent source and binlog tablets were introduced. Keep nullable AFTER conversion coverage on the new tablet-pair architecture.
HappenLee
reviewed
Aug 7, 2026
|
|
||
| namespace { | ||
|
|
||
| ColumnWithTypeAndName make_nullable_after_column(const ColumnWithTypeAndName& source) { |
Rename the source-column predicate and nullable wrapper so they describe Row Binlog value columns instead of implying the source cid identifies an AFTER column.
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28687 ms |
Contributor
TPC-DS: Total hot run time: 157467 ms |
Contributor
ClickBench: Total hot run time: 23.58 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #65951
Related PR: #65810
Problem Summary: Row Binlog makes normal non-key AFTER columns nullable even when their source columns are NOT NULL.
OlapBlockDataConvertorderives the null map from the runtime source column, so a non-nullable complex source provides no top-level null map. Nullable ARRAY, MAP, and STRUCT writers can then leave their physical null child empty and abort during segment finalization.This PR keeps the fix local to Row Binlog:
ColumnNullable<original>with an all-zero null map before conversion.full_block, and do not mutate the shared input Block.ColumnWriter::append()behavior and remove the implicit null-map state, avoiding behavior changes in Scalar, VARIANT, and other common write paths.The test covers full updates, explicit partial updates, and partial updates whose NOT NULL MAP/STRUCT values are filled from a historical Rowset. It verifies logical readback and the physical top-level null-child row count.
Release note
Fix a BE crash when Row Binlog writes NOT NULL complex columns into nullable AFTER columns.
Check List (For Author)
./run-be-ut.sh --run --filter=SegmentFlusherTransformFormatTest.RowBinlogWritesNotNullComplexColumnsToNullableAfterColumns -j 16./build.sh --be -j96build-support/check-format.shbuild-support/run-clang-tidy.sh --base HEAD~2 --build-dir be/ut_build_ASAN