Skip to content

[feature](dbt) Complete Doris incremental strategies - #66361

Closed
xylaaaaa wants to merge 1 commit into
apache:masterfrom
xylaaaaa:agent/complete-doris-incremental-strategies
Closed

[feature](dbt) Complete Doris incremental strategies#66361
xylaaaaa wants to merge 1 commit into
apache:masterfrom
xylaaaaa:agent/complete-doris-incremental-strategies

Conversation

@xylaaaaa

@xylaaaaa xylaaaaa commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

dbt-doris previously materialized every incremental batch and exposed Unique Key upsert behavior under the standard insert_overwrite name. That made the public strategy semantics differ from dbt Core and prevented native Doris overwrite behavior.

This PR implements the dbt Core 1.12 incremental strategy contract for:

  • append on Duplicate Key tables;
  • merge as a full-row Merge-on-Write Unique Key upsert;
  • delete+insert, using a single MOW upsert where equivalent and a staged transactional DELETE USING plus INSERT path for Merge-on-Read targets;
  • native whole-table, static-partition, and dynamic-partition insert_overwrite.

It also adds early source/target key validation, duplicate-key protection, on_schema_change handling with Doris schema-change polling, atomic full-refresh table exchange, failed View-to-Table recovery, and cleanup of dbt helper relations.

The adapter now targets dbt Core 1.12.x and Python 3.10 or newer. microbatch, Doris 4.1 native MERGE INTO, partial merge columns, and incremental predicates remain out of scope.

Release note

Incremental strategy names now match dbt semantics. Projects that previously used insert_overwrite for Doris Unique Key upserts must migrate to merge. Native whole-table and partition INSERT OVERWRITE are now supported.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
      • python -m pytest -q test/unit: 61 passed on Python 3.12
      • 20 Doris incremental functional tests passed on the original implementation snapshot before this branch was rebased onto current master; they were not rerun for this draft PR.
      • flake8 passed for every Python file changed by this PR.
      • git diff --check origin/master...HEAD passed.
      • A full-tree flake8 invocation still reports the pre-existing unused pytest import in test/functional/adapter/test_basic.py, which this PR does not modify.
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. insert_overwrite now performs a real overwrite; Unique Key upsert uses merge.
  • Does this need documentation?

    • No.
    • Yes. Usage and migration notes are included in extension/dbt-doris/docs/incremental.zh-CN.md.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: dbt-doris previously materialized every incremental batch and used insert_overwrite as the name for Unique Key upsert semantics. Implement the dbt Core 1.12 strategy contract for append, merge, delete+insert, and native insert_overwrite; avoid physical staging for single-statement paths; add safe transactional staging for Merge-on-Read delete+insert; wait for asynchronous schema changes; and recover failed View-to-Table full refresh replacements.

### Release note

Incremental strategy names now match dbt semantics. Projects that used insert_overwrite for Unique Key upsert must migrate to merge. Native whole-table and partition INSERT OVERWRITE are now supported.

### Check List (For Author)

- Test:
    - Unit Test: 61 unit tests passed on Python 3.12
    - Functional Test: 20 Doris incremental functional tests passed before rebasing this change; not rerun for this draft PR
    - Static check: flake8 passed for changed Python files and git diff --check passed
- Behavior changed: Yes; insert_overwrite now performs native overwrite, while Unique Key upsert uses merge
- Does this need documentation: Yes; extension/dbt-doris/docs/incremental.zh-CN.md
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xylaaaaa

xylaaaaa commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Closing because this draft was opened against the upstream repository by mistake. The intended review is an internal PR in the xylaaaaa/doris fork.

@xylaaaaa xylaaaaa closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants