Skip to content

[fix](be) Fix table stream change type predicate pushdown#65959

Open
zzzzwc wants to merge 1 commit into
apache:masterfrom
zzzzwc:zwc/fix-table-stream-delete
Open

[fix](be) Fix table stream change type predicate pushdown#65959
zzzzwc wants to merge 1 commit into
apache:masterfrom
zzzzwc:zwc/fix-table-stream-delete

Conversation

@zzzzwc

@zzzzwc zzzzwc commented Jul 23, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #65705

Related PR: None

Problem Summary: A MIN_DELTA table stream can expose DELETE events through a direct SELECT but lose them in INSERT SELECT when the query filters DORIS_STREAM_CHANGE_TYPE_COL. The stream-change column and the raw binlog operation column share the same physical slot but use different encodings rule. OlapScanner pushed the output-domain predicate down to storage, where it was evaluated against raw binlog operation values before BlockReader synthesized the final stream-change type. This filtered required rows too early. Keep binlog operation predicates as scanner residuals for MIN_DELTA and DETAIL scans so they are evaluated after BlockReader produces the stream-change values, while preserving predicate pushdown for other columns.

Release note

Fix Table Stream INSERT SELECT failing to emit DELETE events when filtering by stream change type.

Check List (For Author)

  • Test: No test run (existing regression coverage reproduces the issue)
  • Behavior changed: Yes, change-type predicates for MIN_DELTA and DETAIL scans are evaluated after BlockReader synthesizes the stream-change value
  • Does this need documentation: No

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzzwc

zzzzwc commented Jul 23, 2026

Copy link
Copy Markdown
Author

run buildall

### What problem does this PR solve?

Issue Number: close apache#65705

Related PR: None

Problem Summary: A MIN_DELTA table stream can expose DELETE events through a direct SELECT but lose them in INSERT SELECT when the query filters __DORIS_STREAM_CHANGE_TYPE_COL__. The stream-change column and the raw binlog operation column share the same physical slot but use different encodings. OlapScanner pushed the output-domain predicate down to storage, where it was evaluated against raw binlog operation values before BlockReader synthesized the final stream-change type. This filtered required rows too early. Keep binlog operation predicates as scanner residuals for MIN_DELTA and DETAIL scans so they are evaluated after BlockReader produces the stream-change values, while preserving predicate pushdown for other columns.

### Release note

Fix Table Stream INSERT SELECT failing to emit DELETE events when filtering by stream change type.

### Check List (For Author)

- Test: No test run (existing regression coverage reproduces the issue)
- Behavior changed: Yes, change-type predicates for MIN_DELTA and DETAIL scans are evaluated after BlockReader synthesizes the stream-change value
- Does this need documentation: No
@zzzzwc
zzzzwc force-pushed the zwc/fix-table-stream-delete branch from 7c1a945 to 2a0988d Compare July 23, 2026 10:54
@zzzzwc

zzzzwc commented Jul 23, 2026

Copy link
Copy Markdown
Author

/review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug](High) Table Stream DELETE events are visible in SELECT but not emitted by INSERT SELECT

2 participants