[Feature][Connector] support avro format by liunaijie · Pull Request #4976 · apache/seatunnel

liunaijie · 2023-06-26T14:20:40Z

Purpose of this pull request

close #4821

added avro format and UT
kafka support avro format and e2e test

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

- added avro format and UT - kafka support avro format and e2e test

TyrantLucifer · 2023-06-27T02:19:52Z

Good pr, let's waiting CI/CD

liunaijie · 2023-06-27T09:00:45Z

pls re-trigger the CI.

liugddx · 2023-06-27T10:39:35Z

Can you improve the relevant documents?

https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/kafka.md
https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/kafka.md

liunaijie · 2023-06-30T07:38:34Z

@TyrantLucifer @liugddx PTAL

TaoZex

LGTM

...seatunnel-format-avro/src/main/java/org/apache/seatunnel/format/avro/RowToAvroConverter.java

...nnel-connector-v2-e2e/connector-kafka-e2e/src/test/resources/avro/kafka_avro_to_console.conf

seatunnel-formats/seatunnel-format-avro/pom.xml

liunaijie · 2023-07-07T02:36:32Z

can we re-trigger the tests, i check the error, it's failed on PaimonIT. get NPE when write data to paimon. but the col name and value all are not null.

liugddx · 2023-07-07T02:42:05Z

can we re-trigger the tests, i check the error, it's failed on PaimonIT. get NPE when write data to paimon. but the col name and value all are not null.

This looks like a bug

liunaijie · 2023-07-07T07:18:36Z

I remember when use avro 1.10.2 it can pass the actions. so i will increase the avro version to verify is this related.

liunaijie · 2023-07-07T07:32:21Z

increase the avro version same with paimon

Signed-off-by: Jarvis <liunaijie1996@163.com>

liunaijie · 2023-07-10T01:53:17Z

updated to 1.10.2. now i think it can pass the CI/CD

liunaijie · 2023-07-10T06:33:54Z

It's weird, in this action I updated avro version to 1.10.2, but forgot to change the license.txt. this action pass all tests, exclude dependency check.

and when i updated the licences.txt and ru-run. fail on other stage.

liugddx · 2023-07-10T06:51:35Z

It's weird, in this action I updated avro version to 1.10.2, but forgot to change the license.txt. this action pass all tests, exclude dependency check.

and when i updated the licences.txt and ru-run. fail on other stage.

Can you see why?

liunaijie · 2023-07-11T06:51:31Z

i run the actions on myself repo. it run successful without any change

https://github.com/liunaijie/seatunnel/actions/runs/5507761324
https://github.com/liunaijie/seatunnel/actions/runs/5507761323
https://github.com/liunaijie/seatunnel/actions/runs/5507761322

here is the code campare, you can see i don't have any change exclude the action config.
liunaijie/seatunnel@dev...liunaijie:seatunnel:avro-1.10.2

pls re-trigger the ci, let's see the result.

Co-authored-by: wsstony <tonymao777@163.com>

Co-authored-by: lightzhao <zhaolianyong777@gmail.com>

… redundant code (apache#4982) Co-authored-by: chenzy15 <chenzy15@ziroom.com>

…he#4964) * [Improve][Connector-V2][Jindo-Oss] Optimize jindo-oss connector * [Improve][Connector-V2][Jindo-Oss] Update module name * [Hotfix][Connector-V2][StarRocks] Fix code style

…apache#4969)

… SQL Transform plugin (apache#4991) Co-authored-by: mcy <rewrma@163.com>

…he#4965)

Co-authored-by: chenqqq11 <chenzy15@ziroom.com>

apache#4999)

…apache#4997)

…e#4921)

…ache#149) (apache#5031)

…ker image (apache#5022) Co-authored-by: yctan <1417983443@qq.com>

…ng (apache#5034)

…e#5054)

…#5033) (apache#5035) * [bugfix] change MySQL CDC serverId from int to long (apache#5033) * style: 🎨 optimize code style

… zone (apache#5030)

…ient reuse (apache#5011)

) --------- Co-authored-by: silenceland <silenceland23@163.com> Co-authored-by: changhuyan <877018069@qq.com>

…t file cannot be generated apache#4985 (apache#5051) * fix after the savepoint job is restored, the checkpoint file cannot be generated

…#5065)

[Fea] support avro format

71d4733

- added avro format and UT - kafka support avro format and e2e test

liunaijie force-pushed the dev branch 2 times, most recently from a3aca9e to 2b37d49 Compare June 27, 2023 08:59

liunaijie force-pushed the dev branch 2 times, most recently from 9b59440 to a9f4d1f Compare June 27, 2023 13:01

TaoZex approved these changes Jul 4, 2023

View reviewed changes

...seatunnel-format-avro/src/main/java/org/apache/seatunnel/format/avro/RowToAvroConverter.java Outdated Show resolved Hide resolved

github-actions bot added the reviewed label Jul 4, 2023

liunaijie force-pushed the dev branch from f1626af to c2fe9d6 Compare July 5, 2023 00:26

liugddx reviewed Jul 6, 2023

View reviewed changes

...nnel-connector-v2-e2e/connector-kafka-e2e/src/test/resources/avro/kafka_avro_to_console.conf Show resolved Hide resolved

seatunnel-formats/seatunnel-format-avro/pom.xml Show resolved Hide resolved

liunaijie force-pushed the dev branch 2 times, most recently from f68f382 to 309d293 Compare July 6, 2023 15:31

liugddx closed this Jul 7, 2023

liugddx reopened this Jul 7, 2023

[Feature] fix code style and update avro version

07eb66d

Signed-off-by: Jarvis <liunaijie1996@163.com>

liunaijie force-pushed the dev branch from bbdba1f to 07eb66d Compare July 7, 2023 14:31

liugddx closed this Jul 11, 2023

liugddx reopened this Jul 11, 2023

[Bug] [zeta][starter]fix bug (apache#4983) (apache#4984)

159055d

Co-authored-by: wsstony <tonymao777@163.com>

lightzhao and others added 29 commits July 13, 2023 16:59

fix redis nodes format error. (apache#4981)

4642229

Co-authored-by: lightzhao <zhaolianyong777@gmail.com>

[Improve][CDC]Remove driver for cdc connector (apache#4952)

ba92a14

[Hotfix][Connector-V2][Mongodb] Fix document error content and remove…

495f59b

… redundant code (apache#4982) Co-authored-by: chenzy15 <chenzy15@ziroom.com>

[Improve][Connector-V2][OSS-Jindo] Optimize jindo oss connector (apac…

24c36f9

…he#4964) * [Improve][Connector-V2][Jindo-Oss] Optimize jindo-oss connector * [Improve][Connector-V2][Jindo-Oss] Update module name * [Hotfix][Connector-V2][StarRocks] Fix code style

[bugfix] Upgrade the key log output level(apache#4993)

a9b4265

[Feature][Zeta] Configuration files support user variable replacement (…

fec8cba

…apache#4969)

[Feature][Transform-V2][SQL] Support 'select *' and 'like' clause for…

8fe6bde

… SQL Transform plugin (apache#4991) Co-authored-by: mcy <rewrma@163.com>

[Improve][CDC]change driver scope to provider (apache#5002)

bbb0da1

[Hotfix][Connector-V2][Hive] Support user-defined hive-site.xml (apac…

aa5f86a

…he#4965)

[Improve][Connector-v2][Mongodb]Optimize reading logic (apache#5001)

c46cd40

Co-authored-by: chenqqq11 <chenzy15@ziroom.com>

[Feature][Connector-V2][Clickhouse] clickhouse writes with checkpoints (

279e947

apache#4999)

[Hotfix][Connector-V2][Mongodb] Compatible with historical parameters (…

4363fb6

…apache#4997)

Split updated modules integration test for part 4 (apache#5028)

86a5f36

[Hotfix] Fix the CI Job name error (apache#5032)

b54a8f1

[Feature][CDC] Support disable/enable exactly once for INITIAL (apach…

549b77e

…e#4921)

[bugfix][zeta] Fixed multi-table job data loss and latency issues (ap…

d73dd0e

…ache#149) (apache#5031)

[Hotfix][CDC] Fix jdbc connection leak for mysql (apache#5037)

589f2e1

[Bugfix][zeta] Fix cdc connection does not close (apache#4922)

8710629

Fix XA Transaction bug (apache#5020)

14f712b

Set Up with Kubernetes, dockerfile document error in constructing doc…

c48c3d3

…ker image (apache#5022) Co-authored-by: yctan <1417983443@qq.com>

[Improve][Connector-v2][Mongodb]sink support transaction update/writi…

9df49f2

…ng (apache#5034)

fix:the HdfsStorage can not delete checkpoint file apache#5046 (apach…

9c8a8c8

…e#5054)

[BugFix] [Connector-V2] [MySQL-CDC] serverId from int to long (apache…

bacb26e

…#5033) (apache#5035) * [bugfix] change MySQL CDC serverId from int to long (apache#5033) * style: 🎨 optimize code style

[Feature][Connector-V2][cdc] Change the time zone to the default time…

124f604

… zone (apache#5030)

[Bugfix][connector-cdc-mysql] Fix listener not released when BinlogCl…

9991aa1

…ient reuse (apache#5011)

[Feature][Connector-V2][Jdbc] Add oceanbase dialect factory (apache#4989

9ac43e8

) --------- Co-authored-by: silenceland <silenceland23@163.com> Co-authored-by: changhuyan <877018069@qq.com>

[HotFix][Zeta] fix after the savepoint job is restored, the checkpoin…

8cc3c0f

…t file cannot be generated apache#4985 (apache#5051) * fix after the savepoint job is restored, the checkpoint file cannot be generated

fix class not found exception (apache#5063)

24e528a

[Feature] update action config to support run CI on fork repo (apache…

6320bba

…#5065)

liunaijie closed this Jul 13, 2023

Conversation

liunaijie commented Jun 26, 2023

Purpose of this pull request

Check list

Uh oh!

TyrantLucifer commented Jun 27, 2023

Uh oh!

liunaijie commented Jun 27, 2023

Uh oh!

liugddx commented Jun 27, 2023

Uh oh!

liunaijie commented Jun 30, 2023

Uh oh!

TaoZex left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liunaijie commented Jul 7, 2023

Uh oh!

liugddx commented Jul 7, 2023

Uh oh!

liunaijie commented Jul 7, 2023

Uh oh!

liunaijie commented Jul 7, 2023

Uh oh!

liunaijie commented Jul 10, 2023

Uh oh!

liunaijie commented Jul 10, 2023

Uh oh!

liugddx commented Jul 10, 2023

Uh oh!

liunaijie commented Jul 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants