[Api draft][connector] Add Kudu source and sink connector #2250

2013650523 · 2022-07-23T08:30:35Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

[API-DRAFT] [MERGE] Merge api-draft to dev branch

This reverts commit 770bade.

* Ignore some dead-link check * Update .dlc.json

* unify connector v2 modules name * add codeql check timeout from 30min to 60min

…ger (apache#2118) * remove biginteger

…che#2054) Co-authored-by: smallhibiscus <844981280>

* [Connector-V2]Hive Source * Unified dep version * Unified dep version * Add License and Notice

upgrade beanutils to 1.9.4 upgrade cli to 1.4

* add file sink connector v2 * Update docs/en/connector-v2/sink/File.mdx Co-authored-by: Hisoka <fanjiaeminem@qq.com> * Update docs/en/connector-v2/sink/File.mdx Co-authored-by: Hisoka <fanjiaeminem@qq.com>

…er (apache#2112) * [seatunnel-1947][seatunnel-server] init & add interface for script/script_param/user * [seatunnel-1947][seatunnel-server] update mysql connection * [seatunnel-1947][seatunnel-server] adjust for reviews. Co-authored-by: dijie <nj18652727118@gmail.com>

* [Core][Plugin] fix same plugin can't create twice error. * [Core][Plugin] change create plugin instance method name to createPluginInstance.

apache#2085) * Fix the data output exception when accessing Hive using Spark JDBC Source * add License * format code Co-authored-by: zhaobingzhi <zhaobingzhi@gitv.cn>

The simple example of contribution location on the official website has not been updated

* [doc][connector-v2] pulsar source options doc Simplify option's name and add detailed documentation * [doc][connector-v2] Standardized Pulsar document

* tmp commit * add hadoop2 and hadoop3 shade jar * add hadoop2 and hadoop3 shade jar * add license head * change know denpendencies * tmp commit * tmp commit * change hadoop dependency scope to provide * back pom * fix checkstyle * add example * fix example bug * remove file connector from example and e2e because hadoop2 can not compile with jdk11 * no need jdk8 and jdk11 profile because we don't use hadoop shade jar * change hadoop jar dependency scope to provided * back * file connector can not build in jdk11 * drop hadoop shade * add gitignore item * add hadoop and local file sink * fix pom error * fix pom error * fix pom error * implement new interface * fix UT error * fix e2e error * update build timeout from 30min to 40min * fix e2e error * remove auto service * fix e2e error * fix e2e error * fix e2e error * found e2e error * fix e2e error * fix e2e error * fix e2e error * merge from upstream * merge from upstream * merge from upstream * merge from upstream * merge from upstream * add mvn jvm option * add mvn jvm option * add license * add licnese * add licnese * fix dependency * fix build jvm oom * fix build jvm oom * fix build jvm oom * fix dependency * fix dependency * fix e2e error * add codeql check timeout from 30min to 60min * merge from dev * merge from dev * fix ci error * fix checkstyle * fix ci * fix ci * aa * aa * aa * add .idea * del .idea * del .idea * del .idea * del .idea * remove no use license * remove no use before and after method in test * fix license; remove dependency * fix review * fix build order * fix license * fix license * fix review * fix review * fix review * fix review * fix review * fix review * fix review * fix review * fix review * add code-analysys timeout to 120 * retry ci * update license and remove no use jar from LICENSE file * retry ci Co-authored-by: Hisoka <fanjiaeminem@qq.com>

1. class name of SeaTunnelRuntimeEnvironment changes to SeaTunnelContextAware 2. Serializer doesn't need to extend Serializable 3. unify Enumerator concept

…pache#2151)

* Update new-license.md * Update new-license.md * Update new-license.md

* [Connector-V2]Add Hudi Source * fix dependency licenses * add hudi souce v2 doc * fix doc * fix doc links checked

…xception was encountered (apache#2160)

) * [Bug] [seatunnel-core] Failed to get APP_DIR path bug fixed * [Bug] [seatunnel-core] Failed to get APP_DIR path bug fixed * [Bug] [seatunnel-core] Failed to get APP_DIR path bug fixed Co-authored-by: Gerry <youyang.kou@relx.com>

…2218)

Flink 1.13.6 version is compatible with 1.12, but not applicable to below 1.12

…2225)

…#2229) (apache#2230)

Currently, url https://seatunnel.apache.org/docs/category/source will expand two parent sidebar with both source and source-v2. This is because we're using same slug in our sidebars.js.

) * [Core][Starter] Fix connector v2 can't deserialize on spark * [Core][Starter] Add SerializationUtils Unit Test * [Core][Starter] Add SerializationUtils Unit Test * [Core][Flink] Fixed FlinkEnvironment registerPlugin logic both old and new api

… (apache#2236)

* [Improvement][core] StateT of SeaTunnelSource should extend `Serializable` ,so that `org.apache.seatunnel.api.source.SeaTunnelSource.getEnumeratorStateSerializer` can support a default implementation. This will be useful to each SeaTunnelSource subclass implementation. * repetitive dependency repetitive dependency * [Improvement][connector-v2] postgre jar should be contained in container like mysql-java, so it should be provided, not compile * [Improvement][connector-v2] remove the code block in the implementation class to keep code clean. * [Improvement][connector-v2] remove unused import * [Improvement][connector-v2] modify import order Co-authored-by: bjyflihongyu <lihongyuinfo@jd.com>

add kudu dependency

add kudu config

add kudu

* [CI]Fix License Check not found snapshot jar * Update backend.yml * Update backend.yml

CalvinKirs · 2022-07-23T12:47:36Z

Hi, could you submit it to the DEV branch?

2013650523 · 2022-07-24T00:11:16Z

Hi, I have submitted the PR to Dev branch at #2254,But this is the first time FOR me to submit PR. Could you please check whether there is anything wrong with this? Will all subsequent connectors be submitted to the dev branch?

CalvinKirs · 2022-07-24T09:56:36Z

Hi, I have submitted the PR to Dev branch at #2254,But this is the first time FOR me to submit PR. Could you please check whether there is anything wrong with this?

Thanks,I'll take a look later.

Will all subsequent connectors be submitted to the dev branch?

Yup:)

CalvinKirs and others added 30 commits July 1, 2022 16:24

Update .asf.yaml (apache#2103)

770bade

Merge pull request apache#2083 from apache/api-draft

df251bf

[API-DRAFT] [MERGE] Merge api-draft to dev branch

Revert "Update .asf.yaml (apache#2103)" (apache#2104)

19867ee

This reverts commit 770bade.

Ignore some dead-link check (apache#2108)

a505d50

* Ignore some dead-link check * Update .dlc.json

[Connector-V2]unify connector v2 modules name (apache#2106)

cd34f1d

* unify connector v2 modules name * add codeql check timeout from 30min to 60min

[bugfix] Check isOpen before closing (apache#2107)

7ec0ada

[Feat][UI] Add multi-language related configuration. (apache#2116)

66f8161

[Bug] [seatunnel-connector-flink-fake] Type is not supported: BigInte…

528c6fd

…ger (apache#2118) * remove biginteger

Fix the bug Unrecognized field "TwoPhaseCommit" after doris 0.15 (apa…

418e408

…che#2054) Co-authored-by: smallhibiscus <844981280>

ignore backend ci for ui update (apache#2121)

da4ad17

[Connector-V2]Hive Source (apache#2123)

ffcf3f5

* [Connector-V2]Hive Source * Unified dep version * Unified dep version * Add License and Notice

[SeaTunnel API] [Sink] remove useless context field (apache#2124)

a31fdee

[Bump]Upgrade commons version (apache#2131)

e807b34

upgrade beanutils to 1.9.4 upgrade cli to 1.4

[Connector-V2][Doc] Add File Sink Connector V2 document (apache#2120)

e7cfa2b

* add file sink connector v2 * Update docs/en/connector-v2/sink/File.mdx Co-authored-by: Hisoka <fanjiaeminem@qq.com> * Update docs/en/connector-v2/sink/File.mdx Co-authored-by: Hisoka <fanjiaeminem@qq.com>

[Core][Plugin] Fix same plugin can't create twice error. (apache#2132)

3dd7dc0

* [Core][Plugin] fix same plugin can't create twice error. * [Core][Plugin] change create plugin instance method name to createPluginInstance.

Fix the data output exception when accessing Hive using Spark JDBC So… (

67613d1

apache#2085) * Fix the data output exception when accessing Hive using Spark JDBC Source * add License * format code Co-authored-by: zhaobingzhi <zhaobingzhi@gitv.cn>

Update setup.md (apache#2139)

6d29e3e

The simple example of contribution location on the official website has not been updated

[Feat][UI] Add theme-related configuration. (apache#2136)

bc82e91

fix the doc error (apache#2143)

b0e4204

[doc][connector-v2] pulsar source options doc (apache#2128)

59ce8a2

* [doc][connector-v2] pulsar source options doc Simplify option's name and add detailed documentation * [doc][connector-v2] Standardized Pulsar document

[Improvement][new api] refer to apache#2127 (apache#2144)

e19660a

1. class name of SeaTunnelRuntimeEnvironment changes to SeaTunnelContextAware 2. Serializer doesn't need to extend Serializable 3. unify Enumerator concept

[Core][Starter] Fix same source and sink register plugin set twice (a…

eb179cf

…pache#2151)

Update license.yml (apache#2153)

fd2c075

Update new-license.md (apache#2155)

509add5

* Update new-license.md * Update new-license.md * Update new-license.md

[Connector-V2]Add Hudi Source (apache#2147)

eaedc0a

* [Connector-V2]Add Hudi Source * fix dependency licenses * add hudi souce v2 doc * fix doc * fix doc links checked

[Bug] [connector-v2] When outputting data to clickhouse, a ClassCastE…

a3a2b5d

…xception was encountered (apache#2160)

fix KafkaTableStream source json parse issue (apache#2168)

dde91db

EricJoy2048 and others added 22 commits July 19, 2022 18:41

[Fix][CI] Add remove jar from /tmp/seatunnel-dependencies before run

35f72cd

[Feat][UI] Add dashboard default router. (apache#2216)

99d7cea

[Feat][UI] Add the header component in the dashboard layout. (apache#…

b8c9bb2

…2218)

[Core][Starter] Change jar connector load logic (apache#2193)

2dd3476

[Docs]Fix Flink engine version requirements (apache#2220)

9dbace9

Flink 1.13.6 version is compatible with 1.12, but not applicable to below 1.12

[Feat][UI] Add the setting dropdown in the dashboard layout. (apache#…

a41b0d1

…2225)

[Feat][UI] Add the user dropdown in the dashboard layout. (apache#2228)

e96fbd8

[Bug][hive-connector-v2] Resolve the schema inconsistency bug (apache…

62ca075

…#2229) (apache#2230)

[doc] Correct v2 connector avoid duplicate slug (apache#2231)

5fc205b

Currently, url https://seatunnel.apache.org/docs/category/source will expand two parent sidebar with both source and source-v2. This is because we're using same slug in our sidebars.js.

[Build]Optimize license check (apache#2232)

8250fc8

[Bug][connector-hive] filter '_SUCCESS' file in file list (apache#2235)…

db04651

… (apache#2236)

[Feat][UI] Add the table in the user manage. (apache#2234)

31d43f3

0

1737798

[Feat][UI] Add the table in the data pipes. (apache#2247)

8801c60

Update pom.xml

eedef28

add kudu dependency

Update plugin-mapping.properties

146fa24

add kudu config

Update pom.xml

9b1f10b

add kudu

add email sink connector

8c28e80

[CI]Fix License Check not found snapshot jar (apache#2242)

7414bcc

* [CI]Fix License Check not found snapshot jar * Update backend.yml * Update backend.yml

using constants replace the hard code string (apache#2251)

b71c993

2013650523 added 3 commits July 24, 2022 07:10

Delete seatunnel-connectors-v2/connector-email directory

ed790c8

Update plugin-mapping.properties

5ba8049

Merge branch 'dev' into api-draft

48018fa

CalvinKirs closed this Jul 24, 2022

CalvinKirs mentioned this pull request Aug 28, 2022

[Feature][Connector-V2] oracle connector #2550

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Api draft][connector] Add Kudu source and sink connector #2250

[Api draft][connector] Add Kudu source and sink connector #2250

2013650523 commented Jul 23, 2022 •

edited

CalvinKirs commented Jul 23, 2022

2013650523 commented Jul 24, 2022 •

edited

CalvinKirs commented Jul 24, 2022 •

edited

[Api draft][connector] Add Kudu source and sink connector #2250

[Api draft][connector] Add Kudu source and sink connector #2250

Conversation

2013650523 commented Jul 23, 2022 • edited

Purpose of this pull request

Check list

CalvinKirs commented Jul 23, 2022

2013650523 commented Jul 24, 2022 • edited

CalvinKirs commented Jul 24, 2022 • edited

2013650523 commented Jul 23, 2022 •

edited

2013650523 commented Jul 24, 2022 •

edited

CalvinKirs commented Jul 24, 2022 •

edited