[Test][Plugin]Plugin Test #1326

CalvinKirs · 2022-02-24T10:01:38Z

Hi, everyone, at present we have added many new plugins to the dev branch. We plan to release a version in March. Community contributors are welcome to test it. If you are interested in this, please leave a message in the issue (if you are interested plugin), I will assign it to you, please leave a message in this issue when you are finished, (if you have additional information, you can submit an issue and indicate it in the comment)

You can find the documentation of the plugin on the official website: https://seatunnel.apache.org/docs/introduction

Spark	Plugin	Assignees	Status
Source	TiDB	@zixi0825
Sink	TiDB	@zixi0825
Source	Socket	@wuchunfu	done
Sink	Redis	@wuchunfu	done
Source	Redis	@wuchunfu	done
Sink	Phoneix	@PotatoDream
Source	Phoneix	@PotatoDream
Source	Neo4J	selectbook
Sink	MongoDB	@wuchunfu	done
Source	MongoDB	@wuchunfu	done
Sink	Kudu	@simon824	done
Source	Kudu	@simon824	done
Sink	Kafka	@ZhangchengHu0923	done
Source	Kafka	@ZhangchengHu0923	done
Sink	Jdbc	@zhaomin1423 @wuchunfu	done
Source	Jdbc	@zhaomin1423 @wuchunfu	done
Sink	Hudi	@leo65535
Source	Hudi	@leo65535
Sink	Hive	@zhaomin1423
Source	Hive	@zhaomin1423
Sink	Hbase	@PotatoDream @simon824	done
Source	Hbase	@PotatoDream @simon824	done
Sink	File	@simon824	done
Source	File	@simon824	done
Source	Fake
Sink	Email	@lvshaokang
Source	Elasticsearch	@ITzhangqiang
Sink	Elasticsearch	@ITzhangqiang
Sink	Doris	@taikongren365	done
Sink	Console	@dailidong
Sink	ClickHouse	@xsaffable	done
Transform	Split	@denglingang @simon824 @wuchunfu	done
Transform	Sql	@denglingang @simon824	done
Transform	Json	@denglingang

Flink	Plugin	Assignees	Status
Source	Socket	@JesseAtSZ	done
Sink	Kafka	@gaodayu1209
Source	Kafka	@gaodayu1209
Sink	JDBC	@kalencaya
Source	JDBC	@kalencaya
Sink	InfluxDB	@asdf2014
Source	InfluxDB	@asdf2014
Source	File	@simon824 @kalencaya	done
Sink	File	@simon824 @kalencaya	done
Source	Fake	@JesseAtSZ	done
Sink	Elasticsearch	@ITzhangqiang
Source	Druid	@asdf2014
Sink	Druid	@asdf2014
Sink	Doris	@taikongren365	done
Source	Console	@dailidong
Transform	Split	@simon824	done
Transform	Sql	@simon824	done
Transform	table2datastream
Transform	datastream2table

asdf2014 · 2022-02-24T12:43:27Z

Hi @CalvinKirs, I would like to pick up the plugins of Druid and InfluxDB since I contributed them. BTW, you have missed the source plugin of InfluxDB I believe, please add it as well, thanks.

zhaomin1423 · 2022-02-24T16:58:02Z

Hello, I would like to pick up the plugins of spark-jdbc and spark-hive.

kalencaya · 2022-02-25T01:37:56Z

Hi, I'm willing to pick up flink jdbc and file plugins, for I'm working on them, which resoling unexpected behavior caused by flink FileSink json format.

gaodayu1209 · 2022-02-25T03:14:20Z

Hello, I would like to pick up the plugins of flink-kafka

wuchunfu · 2022-02-25T03:21:50Z

@CalvinKirs Hi, I will test with redis and mongodb

denglingang · 2022-02-25T05:48:48Z

Hello, I would like to pick up the plugins of Transform-json , Transform-split, and Transform-sql

taikongren365 · 2022-02-25T06:34:19Z

Dear @CalvinKirs, Im willing to pick up Spark-Sink-Doris and Flink-Sink-Doris. For now, im doing jobs with Doris.

zixi0825 · 2022-03-01T03:16:34Z

@CalvinKirs I would like to pick up the plugins of spark-tidb source and sink

ITzhangqiang · 2022-03-01T03:29:10Z

@CalvinKirs I would like to pick up Elasticsearch-source and Elasticsearch-sink, for I'm good at them.

suxinshuo · 2022-03-01T03:35:23Z

@CalvinKirs I would like to pick up the plugins of ClickHouse-sink.

PotatoDream · 2022-03-01T03:36:21Z

@CalvinKirs I would like to pick up the plugins of spark-hbase/spark-phoenix source and sink.

JesseAtSZ · 2022-03-01T03:43:12Z

@CalvinKirs No similar tests have been done before. Is this test rigorous? Is there any guidance? If I can, I'd like to try fake source.

CalvinKirs · 2022-03-01T03:48:11Z

@CalvinKirs No similar tests have been done before. Is this test rigorous? Is there any guidance? If I can, I'd like to try fake source.

We need to test whether the corresponding plug-in is working properly, following the plug-in documentation will do. For now, just make sure it works functionally

kyle-cx91 · 2022-03-01T05:52:38Z

@CalvinKirs I would like to pick up the plugins of spark-kafka source and sink，and I would test it in local mode, if u don't mind

CalvinKirs · 2022-03-02T13:52:29Z

hi，guys,Please update the status of this issue as soon as you finish the test

kyle-cx91 · 2022-03-03T07:26:27Z

@CalvinKirs I would like to pick up the plugins of spark-kafka source and sink，and I would test it in local mode, if u don't mind

I functionally tested this and everything works，but I didn't do any performance testing
@CalvinKirs

kyle-cx91 · 2022-03-03T07:31:19Z

@CalvinKirs I would like to pick up the plugins of spark-kafka source and sink，and I would test it in local mode, if u don't mind

I functionally tested this and everything works，but I didn't do any performance testing @CalvinKirs

the doc has nothing write about spark-kafka source plugin,I'd like to add what I learn while doing the test to the document,could u tell me the step to do it
@CalvinKirs

CalvinKirs · 2022-03-03T07:35:21Z

@CalvinKirs I would like to pick up the plugins of spark-kafka source and sink，and I would test it in local mode, if u don't mind

I functionally tested this and everything works，but I didn't do any performance testing @CalvinKirs

the doc has nothing write about spark-kafka source plugin,I'd like to add what I learn while doing the test to the document,could u tell me the step to do it @CalvinKirs

is here https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/spark/configuration/sink-plugins/Kafka.md,
You can see if there is anything to add to this, and again, deeply thanks for your contribution. 👍

kyle-cx91 · 2022-03-03T08:46:21Z

@CalvinKirs I would like to pick up the plugins of spark-kafka source and sink，and I would test it in local mode, if u don't mind

I functionally tested this and everything works，but I didn't do any performance testing @CalvinKirs

the doc has nothing write about spark-kafka source plugin,I'd like to add what I learn while doing the test to the document,could u tell me the step to do it @CalvinKirs

is here https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/spark/configuration/sink-plugins/Kafka.md, You can see if there is anything to add to this, and again, deeply thanks for your contribution. 👍

I found that we are just missing a link in the readme file, need I add and submit a PR ?

CalvinKirs · 2022-03-03T08:52:21Z

@CalvinKirs I would like to pick up the plugins of spark-kafka source and sink，and I would test it in local mode, if u don't mind

I functionally tested this and everything works，but I didn't do any performance testing @CalvinKirs

the doc has nothing write about spark-kafka source plugin,I'd like to add what I learn while doing the test to the document,could u tell me the step to do it @CalvinKirs

is here https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/spark/configuration/sink-plugins/Kafka.md, You can see if there is anything to add to this, and again, deeply thanks for your contribution. 👍

I found that we are just missing a link in the readme file, need I add and submit a PR ?

This is the best 👍

suxinshuo · 2022-03-03T10:13:14Z

@CalvinKirs I would like to pick up the plugins of ClickHouse-sink.

I tested it in local mode and yarn mode, everything works fine.

CalvinKirs · 2022-03-03T10:15:33Z

@CalvinKirs I would like to pick up the plugins of ClickHouse-sink.

I tested it in local mode and yarn mode, everything works fine.

well done, deeply thanks. 👍

taikongren365 · 2022-03-03T13:00:20Z

Dear @CalvinKirs, Im willing to pick up Spark-Sink-Doris and Flink-Sink-Doris. For now, im doing jobs with Doris.

I tested both Spark-Sink-Doris and Flink-Sink-Doris plugin. Both of them works functionally well with simple cases in local mode. Laterly, I will update result which is tested on my Company's Cluster Environment.

CalvinKirs · 2022-03-03T13:10:28Z

Dear @CalvinKirs, Im willing to pick up Spark-Sink-Doris and Flink-Sink-Doris. For now, im doing jobs with Doris.

I tested both Spark-Sink-Doris and Flink-Sink-Doris plugin. Both of them works functionally well with simple cases in local mode. Laterly, I will update result which is tested on my Company's Cluster Environment.

well done, deeply thanks. 👍

CalvinKirs · 2022-03-04T02:41:32Z

@zixi0825, @wuchunfu, @PotatoDream, @zhaomin1423, @leo65535, @PotatoDream, @ITzhangqiang, @denglingang, @gaodayu1209, @asdf2014, @kalencaya, @taikongren365

hi, everyone, if you finish the test, remember to leave a message in this issue, thank you.

simon824 · 2022-03-04T06:58:47Z

These Plugins have been done.
spark：file-sink、file-source、transform-split、transform-sql
flink：file-sink、file-source、transform-split、transform-sql

taikongren365 · 2022-03-04T07:46:20Z

Dear @CalvinKirs, Im willing to pick up Spark-Sink-Doris and Flink-Sink-Doris. For now, im doing jobs with Doris.

I tested both Spark-Sink-Doris and Flink-Sink-Doris plugin. Both of them works functionally well with simple cases in local mode. Laterly, I will update result which is tested on my Company's Cluster Environment.

Flink-Sink-Doris.conf.txt
Spark-Sink-Doris.conf.txt

well done, deeply thanks. 👍

I've finished tests on Company's Cluster Environment. They all ran quite well.
Below is my config file for anyone who is interested in Spark-Sink-Doris and Flink-Sink-Doris plugin.
Dear @CalvinKirs I thought tests for Spark-Sink-Doris and Flink-Sink-Doris plugin are done.

wuchunfu · 2022-03-04T08:28:22Z

These Plugins have been done. spark：file-sink、file-source、transform-split、transform-sql flink：file-sink、file-source、transform-split、transform-sql

well done, thanks

wuchunfu · 2022-03-04T08:28:32Z

Dear @CalvinKirs, Im willing to pick up Spark-Sink-Doris and Flink-Sink-Doris. For now, im doing jobs with Doris.

I tested both Spark-Sink-Doris and Flink-Sink-Doris plugin. Both of them works functionally well with simple cases in local mode. Laterly, I will update result which is tested on my Company's Cluster Environment.

Flink-Sink-Doris.conf.txt Spark-Sink-Doris.conf.txt

well done, deeply thanks. 👍

I've finished tests on Company's Cluster Environment. They all ran quite well. Below is my config file for anyone who is interested in Spark-Sink-Doris and Flink-Sink-Doris plugin. Dear @CalvinKirs I thought tests for Spark-Sink-Doris and Flink-Sink-Doris plugin are done.

well done, thanks

CalvinKirs · 2022-03-06T14:53:09Z

@zixi0825, @PotatoDream, @zhaomin1423, @leo65535, @PotatoDream, @ITzhangqiang, @denglingang, @gaodayu1209, @asdf2014, @kalencaya

hi, everyone, if you finish the test, remember to leave a message in this issue, thank you.

kalencaya · 2022-03-07T08:13:54Z

@zixi0825, @PotatoDream, @zhaomin1423, @leo65535, @PotatoDream, @ITzhangqiang, @denglingang, @gaodayu1209, @asdf2014, @kalencaya

hi, everyone, if you finish the test, remember to leave a message in this issue, thank you.

of course I will do.
Actually, I'm resolving flink JdbcSink plugin exception which caused by not implementing FlinkStreamSink<Row, Row> interface and pull my request recently.

JesseAtSZ · 2022-03-08T03:16:09Z

Do Flink socket and fake source need to be tested? I've been using these two sources for debugging recently. There's no problem.

simon824 · 2022-03-09T09:55:24Z

hi, I have tested spark-kudu source & sink, spark-hbase source & sink.

wuchunfu · 2022-03-10T02:24:48Z

@CalvinKirs The following modules have been tested and the tests passed

spark-connector-source-socket
spark-connector-source-redis
spark-connector-sink-redis
spark-connector-source-mongodb
spark-connector-sink-mongodb
spark-connector-source-jdbc
spark-connector-sink-jdbc
spark-transform-split

zhaomin1423 · 2022-03-10T02:26:51Z

@CalvinKirs spark-connector-source-hive and spark-connector-sink-hive work well. spark-connector-source-jdbc work well.

zhaomin1423 · 2022-03-10T02:27:22Z

@CalvinKirs The following modules have been tested and the tests passed

spark-connector-source-socket
spark-connector-source-redis
spark-connector-sink-redis
spark-connector-source-mongodb
spark-connector-sink-mongodb
spark-connector-source-jdbc
spark-connector-sink-jdbc
spark-transform-split

spark-connector-sink-jdbc has a bug when save mode is update.

wuchunfu · 2022-03-10T03:09:42Z

@CalvinKirs The following modules have been tested and the tests passed
spark-connector-source-socket
spark-connector-source-redis
spark-connector-sink-redis
spark-connector-source-mongodb
spark-connector-sink-mongodb
spark-connector-source-jdbc
spark-connector-sink-jdbc
spark-transform-split
spark-connector-sink-jdbc has a bug when save mode is update.

I'll merge in later

zhaomin1423 · 2022-03-10T03:10:49Z

@CalvinKirs The following modules have been tested and the tests passed
spark-connector-source-socket
spark-connector-source-redis
spark-connector-sink-redis
spark-connector-source-mongodb
spark-connector-sink-mongodb
spark-connector-source-jdbc
spark-connector-sink-jdbc
spark-transform-split
spark-connector-sink-jdbc has a bug when save mode is update.
I'll merge in later

Thanks.

kyle-cx91 · 2022-03-11T02:32:04Z

@CalvinKirs when I tested spark-kafka plugin, I had used spark-console-sink ,it works well

ITzhangqiang · 2022-03-25T06:28:09Z

@CalvinKirs
Hi，I tested the components related Elasticsearch
SPARK：Elasticsearch-source and Elasticsearch-sink doesn't support ES 7.x version
FLINK： Elasticsearch-sink doesn't support ES 7.x version
Does the community plan to support multiple versions? Maybe I can give it a try

selectbook · 2022-03-25T10:30:47Z

@CalvinKirs Hello, I would like to pick up the plugins of Spark-Neo4J

lvshaokang · 2022-04-01T09:18:05Z

@CalvinKirs Hello, I will like to test the email plugin sink

CalvinKirs · 2022-04-01T09:21:24Z

@CalvinKirs Hi，I tested the components related Elasticsearch SPARK：Elasticsearch-source and Elasticsearch-sink doesn't support ES 7.x version FLINK： Elasticsearch-sink doesn't support ES 7.x version Does the community plan to support multiple versions? Maybe I can give it a try

Hi,#1631 is doing this.

lvshaokang · 2022-04-01T16:22:50Z

@CalvinKirs
I tested the email sink，i found some enhancement points，like enable ssl/tsl option、support ArrayType data etc. I will submit a pr later.

CalvinKirs · 2022-04-02T02:00:55Z

@CalvinKirs I tested the email sink，i found some enhancement points，like enable ssl/tsl option、support ArrayType data etc. I will submit a pr later.

well.that's good 🤣

lvshaokang · 2022-04-13T06:14:59Z

@CalvinKirs
PTAL. #1686

lucifer1024 · 2022-11-01T09:40:03Z

Hello, I would like to test the transform of table2datastream and datastream2table

2000liux · 2023-01-18T03:53:09Z

nice

CalvinKirs added good first issue good first issue test volunteer wanted labels Feb 24, 2022

CalvinKirs mentioned this issue Feb 25, 2022

[OnlineMeeting]SeaTunnel community meeting Topic collection #1335

Closed

CalvinKirs mentioned this issue Mar 1, 2022

[Easy to fix list] good first contributor #785

Open

1 task

debrachena mentioned this issue May 11, 2022

SeaTunnel贡献指南 weopenprojects/WeOpen-Star#29

Open

wuzhenhua01 mentioned this issue May 16, 2022

[Test][Plugin]Add kafka plugins test #1887

Closed

3 tasks

[Test][Plugin]Plugin Test #1326

[Test][Plugin]Plugin Test #1326

Comments

CalvinKirs commented Feb 24, 2022 • edited Loading

asdf2014 commented Feb 24, 2022

zhaomin1423 commented Feb 24, 2022

kalencaya commented Feb 25, 2022 • edited Loading

gaodayu1209 commented Feb 25, 2022

wuchunfu commented Feb 25, 2022

denglingang commented Feb 25, 2022

taikongren365 commented Feb 25, 2022

zixi0825 commented Mar 1, 2022

ITzhangqiang commented Mar 1, 2022

suxinshuo commented Mar 1, 2022

PotatoDream commented Mar 1, 2022

JesseAtSZ commented Mar 1, 2022

CalvinKirs commented Mar 1, 2022

kyle-cx91 commented Mar 1, 2022 • edited Loading

CalvinKirs commented Mar 2, 2022

kyle-cx91 commented Mar 3, 2022

kyle-cx91 commented Mar 3, 2022 • edited Loading

CalvinKirs commented Mar 3, 2022 • edited Loading

kyle-cx91 commented Mar 3, 2022

CalvinKirs commented Mar 3, 2022

suxinshuo commented Mar 3, 2022

CalvinKirs commented Mar 3, 2022

taikongren365 commented Mar 3, 2022

CalvinKirs commented Mar 3, 2022

CalvinKirs commented Mar 4, 2022

simon824 commented Mar 4, 2022 • edited Loading

taikongren365 commented Mar 4, 2022

wuchunfu commented Mar 4, 2022

wuchunfu commented Mar 4, 2022

CalvinKirs commented Mar 6, 2022

kalencaya commented Mar 7, 2022

JesseAtSZ commented Mar 8, 2022

simon824 commented Mar 9, 2022

wuchunfu commented Mar 10, 2022

zhaomin1423 commented Mar 10, 2022

zhaomin1423 commented Mar 10, 2022

wuchunfu commented Mar 10, 2022

zhaomin1423 commented Mar 10, 2022

kyle-cx91 commented Mar 11, 2022

ITzhangqiang commented Mar 25, 2022

selectbook commented Mar 25, 2022

lvshaokang commented Apr 1, 2022

CalvinKirs commented Apr 1, 2022

lvshaokang commented Apr 1, 2022

CalvinKirs commented Apr 2, 2022

lvshaokang commented Apr 13, 2022

lucifer1024 commented Nov 1, 2022

2000liux commented Jan 18, 2023

CalvinKirs commented Feb 24, 2022 •

edited

Loading

kalencaya commented Feb 25, 2022 •

edited

Loading

kyle-cx91 commented Mar 1, 2022 •

edited

Loading

kyle-cx91 commented Mar 3, 2022 •

edited

Loading

CalvinKirs commented Mar 3, 2022 •

edited

Loading

simon824 commented Mar 4, 2022 •

edited

Loading