[fix](streamingjob) Support precreated streaming job targets - #66950
Open
JNSimba wants to merge 5 commits into
Open
[fix](streamingjob) Support precreated streaming job targets#66950JNSimba wants to merge 5 commits into
JNSimba wants to merge 5 commits into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-20T03:35:00Z. Please trigger /review again after that time. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes FE streaming job creation for FROM <source> TO DATABASE by allowing precreated Doris target tables to bypass JDBC source-column type mapping/DDL generation, avoiding failures when the source contains types unsupported by the automatic mapper.
Changes:
- Update streaming job table-creation planning to detect existing target tables early and skip source-to-Doris column conversion/CREATE TABLE generation for those targets.
- Adjust the streaming job creation flow to handle “maybe-create” semantics via
Optional<CreateTableCommand>. - Add a Postgres CDC regression suite verifying CREATE JOB succeeds and data lands in a precreated target table even when the source uses a problematic type (ENUM).
Review Checkpoints (per skill guidance)
- Goal & correctness: Largely achieved (precreated targets can bypass automatic type mapping), but there is one behavior regression to address (see stored PR comment) regarding
exclude_columnsvalidation when the target already exists. - Scope/focus: Changes are localized to streaming job utilities + job init path; the signature change is contained (single caller).
- Concurrency: No new shared-state concurrency patterns introduced; logic runs during job initialization.
- Config/compatibility: No new configs; API change is internal (method signature only used in one place).
- Tests: New regression test and
.outare added (external-docker PG); author notes it wasn’t run.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| regression-test/suites/job_p0/streaming_job/cdc/test_streaming_postgres_job_precreated_target.groovy | New regression suite covering precreated target behavior with a PG ENUM source column. |
| regression-test/data/job_p0/streaming_job/cdc/test_streaming_postgres_job_precreated_target.out | Expected output for the new regression query. |
| fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java | Return optional CREATE TABLE commands; early-detect existing targets to skip column type conversion; refactor exclude-column validation helper. |
| fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java | Consume the new optional CREATE TABLE command mapping during job initialization. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
JNSimba
changed the title
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 17477 ms |
Contributor
TPC-DS: Total hot run time: 84670 ms |
Contributor
ClickBench: Total hot run time: 14.76 s |
Member
Author
|
run feut |
Member
Author
|
run p0 |
Contributor
FE UT Coverage ReportIncrement line coverage |
Member
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Review status: complete — converged after 3 rounds.
I am requesting changes for three substantiated issues: the existing-target shortcut still needs downstream CDC type-safety checks, generated target-existence states must be revalidated when consumed, and the committed regression result must be produced by the prescribed runner. The previously reported exclude_columns validation gap is fixed at this head and was treated as a duplicate fence.
Critical checkpoint conclusions
- Goal and correctness: The focused change proves that a precreated compatible target can accept a supported non-key PostgreSQL enum without FE DDL mapping, but the broad bypass also admits schemas that fail snapshot splitting or silently omit values; see the first inline comment.
- Scope and user focus: The four-file change is otherwise localized to generation/consumption of target DDL plus one regression suite. No additional user focus was supplied.
- Concurrency and thread safety: Job initialization can overlap catalog create/drop operations. No new locks, lock-order, or deadlock issue was introduced, but the Optional protocol preserves unlocked catalog observations across remote JDBC work; both stale directions are covered by the second inline comment.
- Lifecycle, error handling, and memory safety: CREATE-time reader setup does not classify the affected source types, so failures can be deferred until scheduled snapshot work after job registration. Local Java Optionals add no ownership, native-memory, C++ static-lifecycle, or BE nullable hazard; the material lifecycle/error issues are the first two comments.
- Configuration and dynamic behavior: No configuration item or dynamic-reload contract is added. Doris also does not currently set or expose Debezium's unknown-type option, which is part of the first finding.
- Compatibility, parallel paths, and conditions: The signature has one production caller and local test callers; generic erasure is unchanged, and no storage, wire, FE/BE, or rolling-upgrade format changes. Existing/missing, default/mapped/fan-in targets and all offset modes were traced. The new existence condition needs the execution-time recheck described inline.
- Tests and generated results: The positive snapshot and invalid-exclusion cases cover the intended branch, and an extra incremental enum assertion was dismissed because snapshot and incremental records share the same deserializer/mapped writer while nearby suites cover incremental routing. Coverage is still missing for the accepted enum-key, unknown custom type, and stale target-state boundaries. The checked-in
.outalso lacks the mandated generated provenance; see the third comment. - Observability: Existing job/task diagnostics cover ordinary failures, but logging cannot make a silently omitted custom field correct; reject or handle that schema losslessly. No separate metrics issue was found.
- Transactions, persistence, data writes, and FE/BE variables: There is no new EditLog schema, persisted field, transaction protocol, or FE/BE variable. Stream-load writes retain their existing transaction path, but source-value omission is a data-correctness failure and the non-atomic catalog observation can register a job against a missing target.
- Performance: Skipping unnecessary FE type conversion for existing targets is the intended optimization, and raw schema fetch occurs only for non-empty exclusions. The only substantiated redundant work is the duplicate target DDL execution covered by the second comment.
- Other issues: No additional distinct issue remained after the final changed-file, downstream-path, and live-thread sweep.
At the reviewed head, Compile, CheckStyle, FE/BE/Cloud UT, External Regression, NonConcurrent Regression, cloud regression, coverage, and performance checks are green; P0 Regression is failed, while FE coverage and the code-review workflow are pending. No local build or test was run, as required by this automated review environment.
Member
Author
|
run p0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Streaming Job converted every source JDBC column and generated target DDL before checking whether the mapped Doris target table already existed. An unsupported source type therefore prevented CREATE JOB even when the user had already created a compatible target table. This change resolves the mapped target table first and skips source-to-Doris type conversion only when that target already exists. Source table identity, primary-key validation, excluded-primary-key validation, and automatic table creation for missing targets remain unchanged.
Release note
Allow FROM source TO DATABASE streaming jobs to use precreated target tables without requiring FE automatic type mapping.
Check List (For Author)