[feature](dbt) Complete Doris incremental strategies - #66361
Closed
xylaaaaa wants to merge 1 commit into
Closed
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: dbt-doris previously materialized every incremental batch and used insert_overwrite as the name for Unique Key upsert semantics. Implement the dbt Core 1.12 strategy contract for append, merge, delete+insert, and native insert_overwrite; avoid physical staging for single-statement paths; add safe transactional staging for Merge-on-Read delete+insert; wait for asynchronous schema changes; and recover failed View-to-Table full refresh replacements.
### Release note
Incremental strategy names now match dbt semantics. Projects that used insert_overwrite for Unique Key upsert must migrate to merge. Native whole-table and partition INSERT OVERWRITE are now supported.
### Check List (For Author)
- Test:
- Unit Test: 61 unit tests passed on Python 3.12
- Functional Test: 20 Doris incremental functional tests passed before rebasing this change; not rerun for this draft PR
- Static check: flake8 passed for changed Python files and git diff --check passed
- Behavior changed: Yes; insert_overwrite now performs native overwrite, while Unique Key upsert uses merge
- Does this need documentation: Yes; extension/dbt-doris/docs/incremental.zh-CN.md
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
Closing because this draft was opened against the upstream repository by mistake. The intended review is an internal PR in the xylaaaaa/doris fork. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
dbt-doris previously materialized every incremental batch and exposed Unique Key upsert behavior under the standard
insert_overwritename. That made the public strategy semantics differ from dbt Core and prevented native Doris overwrite behavior.This PR implements the dbt Core 1.12 incremental strategy contract for:
appendon Duplicate Key tables;mergeas a full-row Merge-on-Write Unique Key upsert;delete+insert, using a single MOW upsert where equivalent and a staged transactionalDELETE USINGplusINSERTpath for Merge-on-Read targets;insert_overwrite.It also adds early source/target key validation, duplicate-key protection,
on_schema_changehandling with Doris schema-change polling, atomic full-refresh table exchange, failed View-to-Table recovery, and cleanup of dbt helper relations.The adapter now targets dbt Core 1.12.x and Python 3.10 or newer.
microbatch, Doris 4.1 nativeMERGE INTO, partial merge columns, and incremental predicates remain out of scope.Release note
Incremental strategy names now match dbt semantics. Projects that previously used
insert_overwritefor Doris Unique Key upserts must migrate tomerge. Native whole-table and partitionINSERT OVERWRITEare now supported.Check List (For Author)
Test
python -m pytest -q test/unit: 61 passed on Python 3.12git diff --check origin/master...HEADpassed.pytestimport intest/functional/adapter/test_basic.py, which this PR does not modify.Behavior changed:
insert_overwritenow performs a real overwrite; Unique Key upsert usesmerge.Does this need documentation?
extension/dbt-doris/docs/incremental.zh-CN.md.Check List (For Reviewer who merge this PR)