[flink-cdc-composer] Base interfaces and models for composer and pipeline definition#2656
Conversation
…r and pipeline definition
Note that Flink composer doesn't have any solid implementation now.
| public PipelineExecution compose(PipelineDef pipelineDef) { | ||
| return new FlinkPipelineExecution(env, "CDC Job", isBlocking); |
There was a problem hiding this comment.
pipelineDef seems useless here.
Do we need a hook passing configurations to StreamExecutionenvironment?
There was a problem hiding this comment.
Currently FlinkPipelineComposer doesn't implement any solid functionality because it depends on common and runtime module, so there's no usage for pipelineDef now. The implementation of compose would be like:
@Override
public PipelineExecution compose(PipelineDef pipelineDef) {
// Translate routers and other operators ...
Source flinkSource = translateToSource(pipelineDef.getSource());
Sink flinkSink = translateToSink(pipelineDef.getSink())
// Build Flink DataStream job
env.fromSource(flinkSource)
.addTransformation(router)
.sinkTo(flinkSink);
// Construct execution
return new FlinkPipelineExecution(env, jobName, isBlocking);
}Then the pipelineDef will be used.
I don't quite get the "passing configuration by hook" part. Could you elaborate more about it?
There was a problem hiding this comment.
I got it , @PatrickRen Can u add todo to remind the uncomplete logic.
| private final SinkDef sink; | ||
| private final List<RouteDef> routes; | ||
| private final List<TransformDef> transforms; | ||
| private final Configuration config; |
There was a problem hiding this comment.
Configuration is optional is ok , some fixed common configuration key should add as a constant (default value,desc,type etc.) for easy to access.
There was a problem hiding this comment.
I got it , com.ververica.cdc.composer.flink.FlinkPipelineComposer#compose is not complete.
There was a problem hiding this comment.
Yeah I can't tell what configurations are needed right now, but will do as we implement the composer later. Thanks!
…line definition (apache#2656) * [flink-cdc-composer] Base interfaces and models for Flink CDC composer and pipeline definition * [flink-cdc-composer] Flink implementation of composer and execution. Note that Flink composer doesn't have any solid implementation now.
3 participants
This pull request implements #2608, which introduces base interfaces exposed to upper-level CLI, including:
PipelineComposer: interface for composerPipelineDef: Java object abstracting a pipeline definitionPipelineExecution: Abstraction for an execution of the pipelineAnd a Flink implementation (just a framework without solid composing logic) is made as a proof of concept.