-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature][Transform] Transform supports multi-table scenarios #5646
base: dev
Are you sure you want to change the base?
Conversation
# Conflicts: # seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/execution/ExecutionPlanGenerator.java
check ci error |
inputCatalogTables.forEach( | ||
inputCatalogTable -> { | ||
String tableId = inputCatalogTable.getTableId().toTablePath().toString(); | ||
transformMap.put(tableId, buildTransform(inputCatalogTable, config)); | ||
}); | ||
|
||
this.outputCatalogTables = | ||
inputCatalogTables.stream() | ||
.map( | ||
inputCatalogTable -> { | ||
String tableName = | ||
inputCatalogTable.getTableId().toTablePath().toString(); | ||
return transformMap.get(tableName).getProducedCatalogTable(); | ||
}) | ||
.collect(Collectors.toList()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This processing method contains serious potential problems. If there are 10 tables in the upstream, but only one of the tables is need be transformed, the job will report an error because the transform of the remaining 9 tables cannot be constructed normally, there is no corresponding configuration, and it stands to reason that the transforms of the remaining 9 tables should not be constructed, because no modifications have been made at all. We should not foreach the upstream catalogtables, but foreach the user's transform configuration.
Purpose of this pull request
1、supports multi-table for transform
2、supports multi-table for assert sink
Does this PR introduce any user-facing change?
Multi-table transfrom is supported. But the original logic is perfectly compatible
How was this patch tested?
add some e2e test
Check list
New License Guide
release-note
.