[Roadmap][Volunteer-Wanted] SeaTunnel 2.0 Roadmap Task

[CalvinKirs]

Hello community, we have recently made a 2.0 roadmap, welcome everyone to discuss and supplement.

Update some latest messages:
Hi guys, this is the roadmap draft according to the mindmap, for the following 3 months, please feel free to share your ideas and welcome to contribute to SeaTunnel and join the community.

Roadmap for 12/2021 ~ 03/2022:

• Official website construction
  • offical website @wolfboys
  • CN/EN User Documentation #1146
  • [Website]CN/EN API & Developer Documentation #980
• Project Structure @simon824
  • Refine module, codes structure [Feature][plugins] refine plugin module #775 [Feature][Plugin] Refactor seatunnel-connector structure #824
• Core Framework
  • execution
    • [DISCUSS]decide whether to have a intermediate layer to unify spark, flink and other execution engines. #962
  • Connectors Plugin Framework
    • refine plugin Framework related codes and APIs（due to breaking changes from spark2.x to spark3.x, whereas flink1.9 to flink1.14）
    • Multi-Version Plugin Dependency with Java ClassLoader @wntp [Feature] Plugin system design #823
    • Do not package in a assembly jar (for multi-version plugins) to avoid Plugin Dependency override @CalvinKirs
    • [Feature]support CDC in framework (not only flink cdc, also for spark) #963
    • auto-generate docs from plugin implementation code, especially from config.
    • (if possible) write once connector, can run both in spark and flink
  • Flink Backend Integration
    • [Umbrella][flink] Upgrade flink version #788
    • refine the codes to support integrate with Flink Connector with Zero coding #871
    • [Refactor][Connector] Put source and sink into combined connector #868
  • Spark Backend Integration
    • [Feature]support latest version of Spark 3.2.0 #875
  • Configuration Management Framework
    • replace typesafe config as pom dependency, other than copying codes to SeaTunnel codebase @leo65535 [Umbrella] optimize the config module  #903 [SeaTunnel #904] [config] improve the config module by shade plugin #906
    • ability to define variables @simon824
    • [DISCUSS]Unified DSL Design(alternatives HOCON, SQL, JSON, YAML ... #964
• Code Quality & Developer Cooperation #
  • add CI to SeaTunnel codebase on github [CI]Run UT #708 [CI]deadlink check #777 Add SkyWalking-Eyes to check License. #677 [CI]Building and testing Java with Maven #673
  • add codestyle check [CI]code style check #714
  • Change Scala to Java as the main program language
  • force check pull request title and commit log starting with issue id, such as '[ISSUE-194] xxx', just like flink community.
• plugins @wuchunfu @wolfboys @simon824
  • add more sources: Hive, Clickhouse, Elasticsearch, MongDB(CDC), Kafka, HDFS,Druid
  • add more sink: Hive, Clickhouse, Elasticsearch, Kafka, HDFS
  • add more transform: ?
  • [Feature][Plugin] Add Plugin Flink Source Clickhouse #1062
  • [Feature][Plugin] Add Plugin Flink Sink Clickhouse #1063
  • [Feature][Plugin] Add Plugin Flink Source Elasticsearch #1064
  • [Feature][Plugin] Add Plugin Flink Sink Elasticsearch #1065
  • [Feature][Plugin] Add Plugin Flink Source Mongodb CDC  #1066
  • [SeaTunnel #1037] Add InfluxDB Flink Sink #1038
  • [Feature] [connector] Support Spark Batch Source/Sink of Datalake delta  #1039
  • [Feature#800][Plugin] Add Spark Sink Doris #816
  • [Feature #772][Plugin] Add plugin-spark-sink-hudi #773
  • [Feature][Plugin] Add plugin-spark-source-redis #752
  • [Feature #739] Add plugin-spark-sink-kafka #740
  • [Seatunnel #691] Add plugin-spark-source-file #692
  • [Feature][plugins] Add plugin-flink-sink-doris  #809
  • Add Druid Flink Source plugin #923
  • [SeaTunnel #919] Add spark-hudi source plugin  #920
  • [SeaTunnel #910] Support spark-hive sink #911
• Installation & Deployment @kezhenxu94 Add Docker images and quick start compose #815
  • refine start-seatunnel.sh related codes to give more convenience for SeaTunnel job management(install, deploy, undeploy..)
  • support install and try SeaTunnel by docker container.

[xleoken]

The roadmap 2.0 is excellent 👍.

I want to add a new feature to roadmap, it is very important for data integration. We use flume long long ago, but it's not friendly for us to write the agent file. Then we use flinksql replace the flume, it have provided great convenience to insert、etl data, etc.

project	weakness
flume	1) need to learn how to write agent file 2) can not work with resource manager like yarn, k8s
sqoop	1) don't have rich connectors 2) based on mapreduce

flinksql

• has rich connectors, cdc
• friendly for us, we can use sql grammar easily, like +, -, etc
• easy to work with resource manager, yarn, k8s

This point of this feature is introduce a new workflow template, show you the workflow demo.

The origin workflow

env {
  execution.parallelism = 1
}

source {
    FakeSourceStream {
      result_table_name = "fake"
      field_name = "name,age"
    }
}

transform {
    sql {
      sql = "select name,age from fake"
    }
}

sink {
  ConsoleSink {}
}

The feature workflow

CREATE TABLE fake_source (
  name string,
  age int
) with (
  'connector.type' = 'fakestream',
  'format.type' = 'json'
);

CREATE TABLE print_sink (
  name string,
  age int
) with (
  'connector.type' = 'print',
);

INSERT INTO print_sink
SELECT * FROM fake_source;

[wntp]

The roadmap 2.0 is excellent 👍.
I wanted After Process and Per Process of source/sink/transform is also in, Let's do it together, come on!

[charlesy6]

The roadmap 2.0 is excellent 👍.

I have a suggestion, it seems we can focus on one type underline engine. It will be mord hard to maintain flink or spark in further.

[kalencaya]

The roadmap 2.0 is excellent 👍.
Configurable is programmable, why not provide useful config dsl assemble source、transform、sink?

[chenhu]

The roadmap 2.0 is excellent 👍
The new configuration is easier to accept .

[chenhu]

We could add the scheduler info like this :
CREATE TABLE fake_source (
name string,
age int
) with (
'connector.type' = 'fakestream',
'format.type' = 'json'
'scheduler.cron' = '* * * * *'
);

CREATE TABLE print_sink (
name string,
age int
) with (
'connector.type' = 'print',
);

INSERT INTO print_sink
SELECT * FROM fake_source;

[wolfboys]

I want to contribute to the Core Framework and Plugin Framework part, I am interested in Flink Backend Integration and Plugin Framework, I have relevant implementation experience.

[davidzollo]

I want to contribute to the Core Framework and Plugin Framework part, I am interested in Flink Backend Integration and Plugin Framework, I have relevant implementation experience.

good job, Do you want to implement the entire parts of Flink Backend Integration and Plugin Framework?

[davidzollo]

@garyelephant good news. great work

[davidzollo]

Everyone is welcome to share your opinions and suggestions if you have any suggestions about the roadmap, welcome to join the open source community, thx

[wuchunfu]

I want to contribute to the plugins part, I am interested in plugins.

[simon824]

I can contribute Project Structure ,Plugins,ability to define variables

[calvinjiang]

I'm mainly interested in the Flink Backend Integration. I'd like to contribute more features for this.

[kezhenxu94]

• support install and try SeaTunnel by docker container.

This one is finished in #815

[davidzollo]

I'm mainly interested in the Flink Backend Integration. I'd like to contribute more features for this.

good job, please create an related issue first

[davidzollo]

if anybody who want to do some contributions, please leave a message.

[yuangjiang]

the roadmap 2.0
It is recommended to disassemble flink into two different execution modes. From the current test, the flink datastream api and table api cannot be unified. Then we need to support the sql connector that can be submitted to the community. The plug-ins that have been developed by themselves are not two Different modes of execution

[yuangjiang]

It is recommended to support the submission of spark and flink scripts. The plugin can be used to submit the extended data of spark hive.

[BruceWong96]

I want to contribute to the Plugin Framework part, I am interested in Flink Backend Integration and Plugin Framework, I have relevant implementation experience.

[Yves-yuan]

The roadmap 2.0 is excellent 👍
I'm interested in [Flink Backend Integration],[Configuration Management Framework] and [plugins] modules,and i will try to make some contributions.