Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs][Connector-V2][Mysql] Refactor connector-v2 docs using unified format Mysql #4590

Merged
merged 18 commits into from May 15, 2023

Conversation

zhilinli123
Copy link
Contributor

@zhilinli123 zhilinli123 commented Apr 17, 2023

subtask of #4572

@zhilinli123 zhilinli123 changed the title Feature docs improve mysql [Umbrella][Feature][Connector-V2][Docs][Mysql] Refactor connector-v2 docs using unified format Mysql Apr 17, 2023
@zhilinli123
Copy link
Contributor Author

Let's see if we have any questions thanks`

@@ -0,0 +1,170 @@
# JDBC
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update JDBC to MySQL ?

| user | String | No | - | Connection instance user name |
| password | String | No | - | Connection instance password |
| query | String | No | - | Use this sql write upstream input datas to database. e.g `INSERT ...` |
| database | String | No | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

query have the higher priority.

| password | String | No | - | Connection instance password |
| query | String | No | - | Use this sql write upstream input datas to database. e.g `INSERT ...` |
| database | String | No | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
| table | String | No | - | Use database and this table-name auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with query and has a higher priority. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| table | String | No | - | Use database and this table-name auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with query and has a higher priority. |
| table | String | No | - | Use database and this table-name auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |


### simple:

> This is a simple example of Mysql via Jdbc Sink
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add more information about this example. For example:

This example defines a SeaTunnel synchronization task that automatically generates data through FakeSource and sends it to JDBC Sink. FakeSource generates a total of 16 rows of data (row.num=16), with each row having two fields, name (string type) and age (int type). The final target table is test_table will also be 16 rows of data in the table.

Before run this job, you need create database test and table test_table in your mysql. And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in Install SeaTunnel to install and deploy SeaTunnel. And then follow the instructions in Quick Start With SeaTunnel Engine to run this job.


### CDC(Change data capture) event

> CDC change data is also supported by us
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some examples we need to tell the user which options is needed. For example:

In this case, you need config database, table and primary_keys.

@@ -0,0 +1,152 @@
# JDBC
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# JDBC
# MySQL

| password | String | No | - | Connection instance password |
| query | String | Yes | - | Query statement |
| connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete |
| partition_column | String | No | - | The column name for parallelism's partition, only support numeric type. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only support numeric type primary key, and only can config one column.


### simple:

> This is to access your data source according to your query parameter We can use this if we don't have a speed requirement
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, we need let user know what does this example do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

zhilinli added 3 commits April 17, 2023 17:43
…r to specify a default change to the first partition
…r to specify a default change to the first partition

transform {
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/category/transform
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# please go to https://seatunnel.apache.org/docs/category/transform
# please go to https://seatunnel.apache.org/docs/category/transform-v2

image

The transform has been removed, use transform-v2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Please wait for CI.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…r to specify a default change to the first partition
@EricJoy2048 EricJoy2048 changed the title [Umbrella][Feature][Connector-V2][Docs][Mysql] Refactor connector-v2 docs using unified format Mysql [Docs][Connector-V2][Mysql] Refactor connector-v2 docs using unified format Mysql Apr 18, 2023
EricJoy2048
EricJoy2048 previously approved these changes Apr 19, 2023
Copy link
Member

@EricJoy2048 EricJoy2048 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

TaoZex
TaoZex previously approved these changes Apr 19, 2023
Copy link
Contributor

@TaoZex TaoZex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment on lines 29 to 30
|------------|-----------------------------------------------|--------------------------|---------------------------------------|---------------------------------------------------------------------------|
| mysql | Depending on your Database dependency version | com.mysql.cj.jdbc.Driver | jdbc:mysql://localhost:3306:3306/test | [Download](https://mvnrepository.com/artifact/mysql/mysql-connector-java) |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Different dependency version has different driver class.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

…r to specify a default change to the first partition
EricJoy2048
EricJoy2048 previously approved these changes Apr 25, 2023
Copy link
Member

@EricJoy2048 EricJoy2048 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

This is a good job. Thanks for you contribution.

docs/en/connector-v2/sink/Mysql.md Outdated Show resolved Hide resolved
## Key features

- [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [ ] [stream](../../concept/connector-v2-features.md)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hailin0 Please review

Comment on lines 138 to 150
jdbc {
url = "jdbc:mysql://localhost:3306/test"
driver = "com.mysql.cj.jdbc.Driver"

max_retries = 0
user = "root"
password = "123456"
query = "insert into test_table(name,age) values(?,?)"

is_exactly_once = "true"

xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource"
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
jdbc {
url = "jdbc:mysql://localhost:3306/test"
driver = "com.mysql.cj.jdbc.Driver"
max_retries = 0
user = "root"
password = "123456"
query = "insert into test_table(name,age) values(?,?)"
is_exactly_once = "true"
xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource"
}
sink {
jdbc {
url = "jdbc:mysql://localhost:3306/test"
driver = "com.mysql.cj.jdbc.Driver"
max_retries = 0
user = "root"
password = "123456"
query = "insert into test_table(name,age) values(?,?)"
is_exactly_once = "true"
xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource"
}
}

Comment on lines 158 to 168
jdbc {
url = "jdbc:mysql://localhost:3306/test"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "123456"

# You need to configure both database and table
database = test
table = sink_table
primary_keys = ["id","name"]
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
jdbc {
url = "jdbc:mysql://localhost:3306/test"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "123456"
# You need to configure both database and table
database = test
table = sink_table
primary_keys = ["id","name"]
}
sink {
jdbc {
url = "jdbc:mysql://localhost:3306/test"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "123456"
# You need to configure both database and table
database = test
table = sink_table
primary_keys = ["id","name"]
}
}

## Key features

- [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove stream

}

sink {
Console {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Console {}
Console {}

Comment on lines 117 to 129
Jdbc {
url = "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "123456"
# Define query logic as required
query = "select * from type_bin"
# Parallel sharding reads fields
partition_column = "id"
# Number of fragments
partition_num = 10
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Jdbc {
url = "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "123456"
# Define query logic as required
query = "select * from type_bin"
# Parallel sharding reads fields
partition_column = "id"
# Number of fragments
partition_num = 10
}
source {
Jdbc {
url = "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "123456"
# Define query logic as required
query = "select * from type_bin"
# Parallel sharding reads fields
partition_column = "id"
# Number of fragments
partition_num = 10
}
}

Comment on lines 137 to 151
Jdbc {
url = "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "123456"
# Define query logic as required
query = "select * from type_bin"
partition_column = "id"
# Read start boundary
partition_lower_bound = 1
# Read end boundary
partition_upper_bound = 500
partition_num = 10
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Jdbc {
url = "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "123456"
# Define query logic as required
query = "select * from type_bin"
partition_column = "id"
# Read start boundary
partition_lower_bound = 1
# Read end boundary
partition_upper_bound = 500
partition_num = 10
}
source {
Jdbc {
url = "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "123456"
# Define query logic as required
query = "select * from type_bin"
partition_column = "id"
# Read start boundary
partition_lower_bound = 1
# Read end boundary
partition_upper_bound = 500
partition_num = 10
}
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your careful review. I have corrected it
@hailin0

@TyrantLucifer
Copy link
Member

waiting CI/CD

Copy link
Contributor

@TaoZex TaoZex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@hailin0 hailin0 merged commit 28f2314 into apache:dev May 15, 2023
4 checks passed
zhilinli123 added a commit to zhilinli123/incubator-seatunnel that referenced this pull request May 17, 2023
zhilinli123 added a commit to zhilinli123/incubator-seatunnel that referenced this pull request May 18, 2023
zhilinli123 added a commit to zhilinli123/incubator-seatunnel that referenced this pull request May 18, 2023
zhilinli123 added a commit to zhilinli123/incubator-seatunnel that referenced this pull request May 23, 2023
zhilinli123 added a commit to zhilinli123/incubator-seatunnel that referenced this pull request Jun 7, 2023
zhilinli123 added a commit to zhilinli123/incubator-seatunnel that referenced this pull request Jun 20, 2023
ic4y pushed a commit that referenced this pull request Aug 6, 2023
…fied format PostgreSQL #4590 (#4757)

* [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL
liunaijie pushed a commit to liunaijie/seatunnel that referenced this pull request Aug 12, 2023
…fied format PostgreSQL apache#4590 (apache#4757)

* [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL
ruanwenjun added a commit to ruanwenjun/incubator-seatunnel that referenced this pull request Nov 6, 2023
* [Feature][Connector V2] expose configurable options in Cassandra (apache#3681)

* [Connector-V2][Paimon] Introduce paimon connector (apache#4178)

* [Improve][Zeta] Improve Zeta operation max count and ignore NPE (apache#4787)

* [Improve][Zeta] Improve Zeta operation max count and ignore NPE

* [Improve][Zeta] Improve Zeta operation max count and ignore NPE

* [Improve][Zeta] Cancel pipeline add retry to avoid cancel failed. (apache#4792)

* [Hotfix][CDC] Fix chunk start/end parameter type error (apache#4777)

Incorrect wrapping as Array<Array> types, but only Array type required

* [Feature][Zeta] Add OSS support for Imap storage to cluster-mode type (apache#4683)

* Add OSS/S3 to cluster-mode type apache#4621

* fixed bug & add e2e test

* Wait for the node to start before scheduling  & Move jar to parent pom & optimize writer

* update LICENSE

* [Hotfix][CI] Fix error repository name in ci config files (apache#4795)

* [Feature][Json-format] support read format for pulsar (apache#4111)

* [Improve][Connector-V2][Jdbc-Sink][Doc] Add the generate sink sql par… (apache#4797)

* [Improve][Connector-V2][Jdbc-Sink][Doc] Add the generate sink sql parameter for the jdbc sinj document

* [Docs][Connector-V2][Mysql] fix Mysql sink format doc (apache#4800)

* [Hotfix][Connector][Jdbc] Fix sqlserver system table case sensitivity (apache#4806)

* [Hotfix][Connector][Jdbc] Fix reconnect throw close statement exception (apache#4801)

* [Hotfix][Connector-V2][Jdbc] Fix the error of extracting primary key column in sink (apache#4815)

* [Feature][Connector-v2] Add Snowflake Source&Sink connector (apache#4470)


---------

Co-authored-by: Eric <gaojun2048@gmail.com>
Co-authored-by: hailin0 <wanghailin@apache.org>

* [Hotfix][CI] Fix redundant modules run e2e tests when change jdbc module (apache#4824)

* fix pom.xml code style (apache#4836)

* [Chore] Format the.conf file using the same style (apache#4830)

* [Hotfix][Zeta] Fix cpu load problem (apache#4828)

* [Improve][Zeta] Reduce the number of IMAPs used by checkpointIdCounter (apache#4832)

* [Bugfix][connector-v2][rabbitmq] Fix reduplicate ack msg bug and code style (apache#4842)

---------

Co-authored-by: 毕博 <bibo@mafengwo.com>

* [Improve][Zeta] async execute checkpoint trigger and other block method (apache#4846)

* [Improve][Zeta] async execute checkpoint trigger

* [Bug][Zeta] Fix zeta cannot normally recycle thread belong to abnormal tasks

* [Improve][Zeta] Move `restoreState` add `addSplitsBack` execute by TaskExecuteService

* [Improve][Zeta] Move `receivedReader` execute by TaskExecuteService

* [Bug][Zeta] Fix task `notifyTaskStatusToMaster` failed when job not running or failed before run (apache#4847)

* [Bug][Zeta] Fix task repeat notify failed when job not running

* [Bug][Zeta] Fix notifyTaskStatusToMaster not release lock and NPE

* [Improve][Zeta] Reduce the frequency of fetching data from imap (apache#4851)

* [Improve][Zeta] Add Metaspace size default value to config file (apache#4848)

* [Improve][Zeta] Speed up listAllJob function (apache#4852)

* [Bug][Zeta] Fix TaskGroupContext always hold classloader so classloader can't recycle (apache#4849)

* [Improve][Zeta] Fix engine runtime error (apache#4850)

* [Hotfix][Zeta] Fix completePendingCheckpoint concurrent action (apache#4854)

This operation does not allow concurrent execution

* [Hotfix][Zeta] Fix master active bug (apache#4855)

* [Bugfix][DAG] Fix the incorrect setting of transform parallelism (apache#4814)

* [Hotfix][Zeta] fix pipeline state not right bug (apache#4823)

* [BUG][Doris] Add a jobId to the doris label to distinguish between tasks (apache#4853)

Co-authored-by: zhouyao <yao.zhou@marketingforce.com>

* [Improve] Add a jobId to the doris label to distinguish between tasks (apache#4839)

Co-authored-by: zhouyao <yao.zhou@marketingforce.com>

* [Hotfix][Zeta] Fix IMap operation timeout bug (apache#4859)

* [Bug][Zeta] Fix restoreComplete Future can't be completed when cancel task (apache#4863)

* [Feature][SQL Transform]Add catalog support for SQL Transform plugin (apache#4819)

* [improve][SelectDB] Add a jobId to the selectDB label to distinguish between tasks (apache#4864)

Co-authored-by: zhouyao <yao.zhou@marketingforce.com>

* [Hotfix][Connector-v2][kafka] Fix the short interval of pull data settings and revise the format (apache#4875)

* [Bug][Connector-V2][Doris] update last checkpoint id when doing snapshot (apache#4881)

* [Hotfix][Zeta] Fix deploy operation timeout but task already finished bug (apache#4867)

* [Core][Docs]Remove incubator in README file (apache#4882)

* [Bugfix][CDC Base] Solving the ConcurrentModificationException caused by snapshotState being modified concurrently. (apache#4877)

* [improve][CDC base] Implement Sample-based Sharding Strategy with Configurable Sampling Rate (apache#4856)

* [Improve][Zeta] Reduce the operation count of imap_running_job_metrics (apache#4861)

* [Bug][Zeta] Fix TaskExecutionService will return not active ExecutionContext (apache#4869)

* [Hotfix][Jdbc] Fix XA DataSource crash(Oracle/Dameng/SqlServer) (apache#4866)

* [Bugfix] [Connector-V2] [File] Fix read temp file (apache#4876)

Co-authored-by: wantao <wantao@inmyshow.com>

* [Bug][Zeta] Fix TaskExecutionService synchronized lock will not release (apache#4886)

* [Improve][Zeta] Move driver into lib directory and change operation count (apache#4845)

* [hotfix][kafka] Fix the problem that the partition information cannot be obtained when kafka is restored (apache#4764)

* [Bugfix][zeta] Fix the deadlock issue with JDBC driver loading (apache#4878)

* [Chore] update 2.3.2 release-note.md (apache#4892)

* [Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys  (apache#4634)

* [Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(20) used as a partition_column apache#4634

Co-authored-by: zhilinli <lzl15844876351@163.com>

* [Bug][connector-v2][doris] add streamload Content-type for doris URLdecode error (apache#4880)

* [Chore] Change repository name from incubator-seatunnel to seatunnel  (apache#4868)


---------

Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

* [Improve][connector-V2-Neo4j]Supports neo4j sink batch write and update docs (apache#4841)

* [Hotfix][connector-v2][e2e] Fix maven scope (apache#4901)

* quick-start-seatunnel-engine.md (apache#4943)

* fix error (apache#4888)

* [Hotfix][Connector-V2][ClickhouseFile] Fix ClickhouseFile write file failed when field value is null (apache#4937)

* Update ClickhouseFileSinkWriter.java

Bug fix: When ClikchouseFileSinkerWriter writes to a temporary file, it does not check whether the field value is empty, so an exception will be thrown.

Modified to write an empty string when a null value is encountered

* Update ClickhouseFileSinkWriter.java

repair code style

* Update ClickhouseFileSinkWriter.java

code style

* [Improve][Zeta] Add an interface for batch retrieval of JobMetrics (apache#4576)

* [Improve] Documentation and partial word optimization. (apache#4936)

* code format

* add cdc feature

* fix cdc can not get driver error

---------

Co-authored-by: gdliu3 <gdliu3@iflytek.com>

* [Doc][Connector-V2] StarRocks `nodeUrls` property name fix (apache#4951)

node_urls -> nodeUrls
node_urls doesn't work

* [Feature][E2E][FtpFile] add ftp file e2e test case (apache#4647)

* [WIP][Feature][Connector-e2e] add ftp e2e test

* Let e2e barely execute by excluding the commons-net jar package.

* Resolve the maven conflict

---------

Co-authored-by: hailin0 <wanghailin@apache.org>

* [Hotfix][Connector-V2][StarRocks] Fix code style (apache#4966)

* [Hotfix][Connector-v2][HbaseSink]Fix default timestamp (apache#4958)

* [Doc]Change the transform website url (apache#4954)

* [Docs][Connector-V2][Http]Reconstruct the Http connector document (apache#4962)

Co-authored-by: chenzy15 <chenzy15@ziroom.com>

* [Feature][connector-v2][mongodb] mongodb support cdc sink (apache#4833)

* [Bug] [zeta][starter]fix bug (apache#4983) (apache#4984)

Co-authored-by: wsstony <tonymao777@163.com>

* fix redis nodes format error. (apache#4981)

Co-authored-by: lightzhao <zhaolianyong777@gmail.com>

* [Improve][CDC]Remove  driver for cdc connector (apache#4952)

* [Hotfix][Connector-V2][Mongodb] Fix document error content and remove redundant code (apache#4982)

Co-authored-by: chenzy15 <chenzy15@ziroom.com>

* [Improve][Connector-V2][OSS-Jindo] Optimize jindo oss connector (apache#4964)

* [Improve][Connector-V2][Jindo-Oss] Optimize jindo-oss connector

* [Improve][Connector-V2][Jindo-Oss] Update module name

* [Hotfix][Connector-V2][StarRocks] Fix code style

* [bugfix] Upgrade the key log output level(apache#4993)

* [Feature][Zeta] Configuration files support user variable replacement (apache#4969)

* [Feature][Transform-V2][SQL] Support 'select *' and 'like' clause for SQL Transform plugin (apache#4991)

Co-authored-by: mcy <rewrma@163.com>

* [Improve][CDC]change driver scope to provider (apache#5002)

* [Hotfix][Connector-V2][Hive] Support user-defined hive-site.xml (apache#4965)

* [Improve][Connector-v2][Mongodb]Optimize reading logic (apache#5001)

Co-authored-by: chenqqq11 <chenzy15@ziroom.com>

* [Feature][Connector-V2][Clickhouse] clickhouse writes with checkpoints (apache#4999)

* [Hotfix][Connector-V2][Mongodb] Compatible with historical parameters (apache#4997)

* Split updated modules integration test for part 4 (apache#5028)

* [Hotfix] Fix the CI Job name error (apache#5032)

* [Feature][CDC] Support disable/enable exactly once for INITIAL (apache#4921)

* [bugfix][zeta] Fixed multi-table job data loss and latency issues (apache#149) (apache#5031)

* [Hotfix][CDC] Fix jdbc connection leak for mysql (apache#5037)

* [Bugfix][zeta] Fix cdc connection does not close (apache#4922)

* Fix XA Transaction bug (apache#5020)

* Set Up with Kubernetes, dockerfile document error in constructing docker image (apache#5022)

Co-authored-by: yctan <1417983443@qq.com>

* [Improve][Connector-v2][Mongodb]sink support transaction update/writing (apache#5034)

* fix:the HdfsStorage can not delete checkpoint file apache#5046 (apache#5054)

* [BugFix] [Connector-V2] [MySQL-CDC] serverId from int to long (apache#5033) (apache#5035)

* [bugfix] change MySQL CDC serverId from int to long (apache#5033)

* style: 🎨 optimize code style

* [Feature][Connector-V2][cdc] Change the time zone to the default time zone (apache#5030)

* [Bugfix][connector-cdc-mysql] Fix listener not released when BinlogClient reuse (apache#5011)

* [Feature][Connector-V2][Jdbc] Add oceanbase dialect factory (apache#4989)


---------

Co-authored-by: silenceland <silenceland23@163.com>
Co-authored-by: changhuyan <877018069@qq.com>

* [HotFix][Zeta] fix after the savepoint job is restored, the checkpoint file cannot be generated apache#4985 (apache#5051)

* fix after the savepoint job is restored, the checkpoint file cannot be generated

* fix class not found exception (apache#5063)

* [Feature] update action config to support run CI on fork repo (apache#5065)

* [Bugfix]fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (apache#5080)

* [Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone (apache#5078)

* Add clickhouse connector time zone key,default system time zone

* Modify the document and add clickhouse server_time_zone configuration

* [Chore] Modify repeat des (apache#5088)

Co-authored-by: 80597928 <Lzl@qq.com>

* [Docs] Add Value types in Java to Schema feature (apache#5087)

* [Feature][Connector-V2] JDBC source support string type as partition key (apache#4947)

* [HotFix] Fix code style (apache#5092)

* [Docs][Zeta] Add savepoint doc (apache#5081)

* [Feature][connector-v2][mongodbcdc]Support source mongodb cdc (apache#4923)

* [Improve] Improve savemode api (apache#4767)

* [Doc] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (apache#5102)

* [Improve][Docs][Clickhouse] Reconstruct the clickhouse connector doc (apache#5085)



---------

Co-authored-by: chenzy15 <chenzy15@ziroom.com>

* [Pom]update version to 2.3.3-SNAPSHOT (apache#5043)

* update version to 2.3.3-SNAPSHOT

* update dependency version in know dependencies file

* Add logs to find job restore from master active switch error

* [Feature][Connector-V2][mysql cdc] Conversion of tinyint(1) to bool is supported (apache#5105)

Co-authored-by: zhouyao <yao.zhou@marketingforce.com>

* [Improve][Zeta] Add sleep for Task to reduce CPU cost (apache#5117)

* [Feature][JDBC Sink] Add DM upsert support (apache#5073)



---------

Co-authored-by: David Zollo <davidzollo365@gmail.com>

* [Hotfix][Connector][Jdbc] Fix the problem of JdbcOutputFormat database connection leak (apache#4802)

[Hotfix][Connector][Jdbc] Fix the problem of JdbcOutputFormat database connection leak

* [Hotfix]Fix mongodb cdc e2e instability (apache#5128)

Co-authored-by: chenzy15 <chenzy15@ziroom.com>

* [Hotfix][Zeta] Fix task state memory leak (apache#5139)

* [Hotfix][Zeta] Fix checkpoint error report without msg (apache#5137)

* [Improve][Zeta] Improve CheckpointCoordinator notify complete when restore (apache#5136)

* [Improve] Improve CheckpointCoordinator notify complete when restore

* update

* [Improve][Zeta] Improve CheckpointCoordinator log error when report error from task (apache#178) (apache#5134)

* [Hotfix][Zeta] Fix MultipleTableJobConfigParser ignore env option (apache#5067)

* [Fix][Zeta] Fix MultipleTableJobConfigParser ignore env option

* update

* [Improve][Connector[File] Optimize files commit order (apache#5045)

Before using `HashMap` store files path, so every checkpoint file commit is out of order.

Now switch to using `LinkedHashMap` to ensure that files are commit in the generated order

* [Hotfix][Mongodb cdc] Solve startup resume token is negative (apache#5143)


---------

Co-authored-by: chenzy15 <chenzy15@ziroom.com>

* [Feature][connector][kafka] Support read debezium format message from kafka (apache#5066)

* [Feature][CDC] Support tables without primary keys (with unique keys) (apache#163) (apache#5150)

* [Feature][Connector-V2][CDC] Support string type shard fields. (apache#5147)

* [feature][CDC base] Supports string type shard fields

* Delete invalid code

* [Feature][Connector-V2][File] Add cos source&sink (apache#4979)

* [Feature][Connector-V2][File] Add cos sink

* update doc&e2e and add pom file header

* add e2e file header and config

* add file-cos module into dist pom.xml

* [Feature][Connector-V2][File] Add cos source

---------

Co-authored-by: dengd1937 <dengd1803@gmail.com>

* [Fix][Zeta] Fix SinkFlowLifeCycle without init lastCommitInfo (apache#5152)

* [Hotfix][MongodbCDC]Refine data format to adapt to universal logic (apache#5162)

Co-authored-by: chenzy15 <chenzy15@ziroom.com>

* [Chore] Update bug-report.yml (apache#5160)

* [Improve][CDC] support exactly-once of cdc and fix the BinlogOffset comparing bug (apache#5057)

* [Improve][CDC] support exactly-once of cdc, fix the BinlogOffset comparing bug

* [Improve][CDC] adjust code style

* [Improve][CDC] fix ci error

---------

Co-authored-by: happyboy1024 <296442618@qq.com>

* [Docs][Connector-V2][Hudi] Reconstruct the Hudi connector document (apache#4905)

* [Docs][Connector-V2][Hudi] Reconstruct the Hudi connector document


---------

Co-authored-by: zhouyao <yao.zhou@marketingforce.com>

* [Docs][Connector-V2][Doris] Reconstruct the Doris connector document (apache#4903)

* [Docs][Connector-V2][Doris] Reconstruct the Doris connector document

---------

Co-authored-by: zhouyao <yao.zhou@marketingforce.com>

* [improve] [CDC Base] Add some split parameters to the optionRule (apache#5161)

* [bugfix] [File Base] Fix Hadoop Kerberos authentication related issues. (apache#5171)

* [CI] add code style check when docs changed (apache#5183)

* [Bug][Translation][Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type. (apache#5170)

* [Improve][Zeta] Move checkpoint notify complete in checkpoint stage (apache#5185)

* [Feature][Catalog] Add JDBC Catalog auto create table (apache#4917)

* [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files. (apache#5153)

* [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

* [Improve][Connector-v2][Jdbc]  check url not null throw friendly message (apache#5097)

* check url not null throw friendly message

* check jdbc source config

* modify jdbc validate method

---------

Co-authored-by: 80597928 <Lzl@qq.com>
Co-authored-by: 80597928 <673421862@qq.com>

* [bugfix][zeta] Fix the issue of two identical IDs appearing when executing seatunnel.sh -l as the job resumes (apache#5191)

* [Improve][Docs][Kafka]Reconstruct the kafka connector document (apache#4778)

* [Docs][Connector-V2][Kafka]Reconstruct the kafka connector document

---------

Co-authored-by: chenzy15 <chenzy15@ziroom.com>

* [Bug][Improve][LocalFileSink]Fix LocalFile Sink file_format_type. (apache#5118)

* [Bug] [connector-v2] PostgreSQL versions below 9.5 are compatible use cdc sync problem (apache#5120)

* [e2e] kafka e2e error (apache#5200)

* [Hotfix][Connector-V2][JindoOssFile] Fix plugin-mapping.properties (apache#5215)

Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com>

* [Improve][Zeta] Don't trigger handleSaveMode when restore (apache#5192)

* move imap storage file dependency packages to submodules (apache#5218)

* [Hotfix][CI]Declare files that will always have UNIX line endings on checkout. (apache#5221)

* [Hotfix][Connector-V2][Paimon] Bump paimon-bundle version to 0.4.0-incubating (apache#5219)

* [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL apache#4590 (apache#4757)

* [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL

* [Docs] Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (apache#4793)

* [Docs] update seatunnel-flink.yaml and Dockerfile to help the demo work

* [Docs] update release-note apache#4788

---------

Co-authored-by: flynnxue <flynnxue@lilithgames.com>
Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com>

* [feature][doris] Doris factory type (apache#5061)

* [feature][doris] Web need factory and data type convertor

* [Fix] Update the Readme (apache#4968)

Use the better description for the SeaTunnel Project

* [CI] Split updated modules integration test for part 5 (apache#5208)

* [CI] Split updated modules integration test for part 5

* [CI] Split updated modules integration test for part 5

* Split e2e

* update json-smart

* fix dm error

* fix dm error

* fix dm error

* fix dm error

* fix dm error

* fix dm error

* fix dm error

* revert code

* revert code

* revert code

---------

Co-authored-by: gdliu3 <gdliu3@iflytek.com>

* [Feature][CDC][Zeta] Support schema evolution framework(DDL) (apache#5125)

* Fixed IMap file storage e2e bug (apache#5237)

* [Improve] [Connector-V2] Remove scheduler in JDBC sink apache#4736 (apache#5168)


---------

Co-authored-by: gdliu3 <gdliu3@iflytek.com>

* [Doc] [JDBC Oracle] Add JDBC Oracle Documentation (apache#5239)

* [Feature][Zeta][REST-API]Add REST API To Submit Job (apache#5107)

* [Fix] Update the project description (apache#4967)

* Update the project description

* [Feature][Zeta] Support history service record job execute error (apache#5114)

* fix:hdfs Checkpoint Storage management fails to delete historical files

* fix:hdfs Checkpoint Storage management fails to delete historical files

* fix after the savepoint job is restored, the checkpoint file cannot be generated

* [Feature][Zeta] Support history service record job execute error

* Improve Jobstate-related class additions add serialVersionUID

* add e2e test

* [hotfix]Update .asf.yaml (apache#5242)

* Update .asf.yaml

* [Hotfix]Fix array index anomalies caused by apache#5057 (apache#5195)

* [bugfix] [savepoint test] Turn on the testSavepoint test. (apache#5199)

* [BUG][Connector-V2][Jdbc] support postgresql json type  (apache#5194)

* add Postgresql json type
Co-authored-by: 80597928 <673421862@qq.com>

* [Bugfix][cdc] Fix mysql bit column to java byte (apache#4817)

* [Bugfix][AmazonDynamoDB] Fix the problem that all table data cannot be obtained (apache#5146)

* [Docs][Connector][Source][jdbc]Change the line boundary store value type to BigDecimal (apache#4900)

* [bug][jdbc][oracle]Fix the Oracle number type mapping problem (apache#5209)

* [Bugfix][zeta] Fix the serialization issue of GetMetricsOperation during multi-node operation. (apache#5206)

* [Hotfix][Zeta] Avoid Redundant Job Submissions by Checking Job Status (apache#5229)

* [Bugfix][zeta] Fixed the issue of duplicated metrics caused by job fault tolerance or restore. (apache#5214)

* [Imporve] [CDC Base] Add a fast sampling method that supports character types (apache#5179)

* fixed zeta ci error (apache#5254)

* [Doc][README] Remove useless github workflow, and adjust description of 'engineering structure'. (apache#4305)

* [Feature][Zeta]The expiration time of a historical Job can be config (apache#5180)

* fix:hdfs Checkpoint Storage management fails to delete historical files
Co-authored-by: hailin0 <wanghailin@apache.org>

* [bugfix] [e2e] Fixed a minor bug (apache#5274)

* [Improve][SQL] Support use catalogTableName as SQL expression (apache#5273)

* [Doc] Improve S3File Source & S3File Sink document (apache#5101)

* Improve S3File Source & S3File Sink document

* Fix style error (apache#5280)

* Fix StarRocksJsonSerializer will transform array/map/row to string (apache#5281)

* [Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document (apache#5129)

* [Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document

* fix format

* fix format

* [Improve][API & Zeta] Using connector custom serializer encode/decode states (apache#5238)

* API: Using DefaultSerializer as connector sink default serializer
* Zeta: Using connector custom serializer encode/decode states

* [Feature][Connector-V2] connector-kafka source support data conversion extracted by kafka connect source (apache#4516)

* Compatible kafka connect json apache#4137

* [Improve][CI/CD] Remove 'paths-ignore', enable the code style check for markdown files. (apache#5286)

* [Bugfix][zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0. (apache#5263)

* [Bugfix][zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0.

* remove max-concurrent

* [Feature][Connector-v2][RedisSink]Support redis to set expiration time. (apache#4975)

* Support redis to set expiration time.

* Set redis expire default value.

* add e2e test.

* add e2e test.

* modify config file name.

---------

Co-authored-by: lightzhao <zhaolianyong777@gmail.com>

* [bugfix] Fix testGetErrorInfo case error (apache#5282)

* [Feature][Zeta] Checkpoint support hdfs ha mode (apache#4942)

* fix browser long type intercept (apache#5267)

Co-authored-by: 80597928 <673421862@qq.com>

* [Docs] remove `incubating` keyword in document (apache#5257)

* [feature][web] hive add option because web need (apache#5154)

* [feature][web] hive add option because web need

* [feature][web] hive add option read_columns

* [feature][web] required update optional

* [bugfix] mvn spotless

* fix conf

* fix conf

---------

Co-authored-by: liuli <m_liuli@163.com>

* [Bug][flink-runtime][connectors-v2] Flink register table Environment  The running mode is set to`job.mode` (apache#4826)

* [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document (apache#5132)

* [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document

* [Improve][Connector-v2][HiveSink]remove drop partition when abort. (apache#4940)

Co-authored-by: lightzhao <zhaolianyong777@gmail.com>
Co-authored-by: liuli <m_liuli@163.com>
Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com>

* [Docs][Connector-V2][SelectDB-Cloud]Reconstruct the SelectDB-Cloud connector document (apache#5130)

* [Docs][Connector-V2][SelectDB-Cloud]Reconstruct the SelectDB-Cloud connector document

* fix codestyle

---------

Co-authored-by: liuli <m_liuli@163.com>

* [Docs][Connector-V2][HDFS]Refactor connector-v2 docs using unified format HDFS. (apache#4871)

* Refactor connector-v2 docs using unified format HDFS.

* add data type.

* update.

* add key feature.

* add hdfs_site_path

* 1.add data type.
2.add hdfs_site_path conf.

* add data type.

* add hdfs site conf.

---------

Co-authored-by: lightzhao <zhaolianyong777@gmail.com>
Co-authored-by: liuli <m_liuli@163.com>

* [Improve] [Connector-V2] Remove scheduler in Tablestore sink (apache#5272)


---------

Co-authored-by: gdliu3 <gdliu3@iflytek.com>

* [BUG][Connector-V2][Mongo-cdc] Incremental data kind error in snapshot phase (apache#5184)

* [BUG][Connector-V2][Mongo-cdc] Incremental data kind error in snapshot phase

* [Hotfix] Fix com.google.common.base.Preconditions to seatunnel shade one (apache#5284)

* [Merge] Fix merge conflict and fix jdbc fieldIde with compatibleMode confusion

---------

Co-authored-by: Cason-ACE <35160064+cason0126@users.noreply.github.com>
Co-authored-by: Tyrantlucifer <TyrantLucifer@gmail.com>
Co-authored-by: hailin0 <wanghailin@apache.org>
Co-authored-by: Xiaojian Sun <sunxiaojian926@163.com>
Co-authored-by: Laglangyue <35491928+laglangyue@users.noreply.github.com>
Co-authored-by: ZhilinLi <zhilinli0706@gmail.com>
Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com>
Co-authored-by: Hao Xu <sduxuhao@gmail.com>
Co-authored-by: Eric <gaojun2048@gmail.com>
Co-authored-by: Bibo <33744252+531651225@users.noreply.github.com>
Co-authored-by: 毕博 <bibo@mafengwo.com>
Co-authored-by: Carl-Zhou-CN <67902676+Carl-Zhou-CN@users.noreply.github.com>
Co-authored-by: zhouyao <yao.zhou@marketingforce.com>
Co-authored-by: Marvin <29311598@qq.com>
Co-authored-by: monster <60029759+MonsterChenzhuo@users.noreply.github.com>
Co-authored-by: gnehil <adamlee489@gmail.com>
Co-authored-by: TaoZex <45089228+TaoZex@users.noreply.github.com>
Co-authored-by: xiaofan2012 <41982310+xiaofan2022@users.noreply.github.com>
Co-authored-by: wantao <wantao@inmyshow.com>
Co-authored-by: Guangdong Liu <804167098@qq.com>
Co-authored-by: zhilinli <lzl15844876351@163.com>
Co-authored-by: zhaifengbing <ainizfb@126.com>
Co-authored-by: dalong <60906603+alibabaMapengfei@users.noreply.github.com>
Co-authored-by: FuYouJ <1247908487@qq.com>
Co-authored-by: davidfans <136911434+davidfans@users.noreply.github.com>
Co-authored-by: Fan Donglai <ddna_1022@163.com>
Co-authored-by: gdliu3 <gdliu3@iflytek.com>
Co-authored-by: DismalSnail <yepenghaobit@gmail.com>
Co-authored-by: lightzhao <40714172+lightzhao@users.noreply.github.com>
Co-authored-by: chenzy15 <chenzy15@ziroom.com>
Co-authored-by: wssmao <39487209+wssmao@users.noreply.github.com>
Co-authored-by: wsstony <tonymao777@163.com>
Co-authored-by: lightzhao <zhaolianyong777@gmail.com>
Co-authored-by: XiaoJiang521 <131635688+XiaoJiang521@users.noreply.github.com>
Co-authored-by: mcy <rewrma@163.com>
Co-authored-by: yctanGmail <138592845+yctanGmail@users.noreply.github.com>
Co-authored-by: yctan <1417983443@qq.com>
Co-authored-by: wu-a-ge <wfygowxf1@163.com>
Co-authored-by: 司马琦昂 <bruce.maqiang@gmail.com>
Co-authored-by: happyboy1024 <137260654+happyboy1024@users.noreply.github.com>
Co-authored-by: He Wang <wanghechn@qq.com>
Co-authored-by: silenceland <silenceland23@163.com>
Co-authored-by: changhuyan <877018069@qq.com>
Co-authored-by: Jarvis <liunaijie1996@163.com>
Co-authored-by: 阿丙 <50567478+gaopeng666@users.noreply.github.com>
Co-authored-by: jackyyyyyssss <127465317+jackyyyyyssss@users.noreply.github.com>
Co-authored-by: 80597928 <Lzl@qq.com>
Co-authored-by: Chengyu Yan <cheneyyin@hotmail.com>
Co-authored-by: zhangchengming601 <86779821+zhangchengming601@users.noreply.github.com>
Co-authored-by: lihjChina <237206177@qq.com>
Co-authored-by: David Zollo <davidzollo365@gmail.com>
Co-authored-by: EchoLee5 <39044001+EchoLee5@users.noreply.github.com>
Co-authored-by: dengdi <114273849+dengd1937@users.noreply.github.com>
Co-authored-by: dengd1937 <dengd1803@gmail.com>
Co-authored-by: happyboy1024 <296442618@qq.com>
Co-authored-by: FlechazoW <35768015+FlechazoW@users.noreply.github.com>
Co-authored-by: 80597928 <673421862@qq.com>
Co-authored-by: kun <66303359+Lifu12@users.noreply.github.com>
Co-authored-by: Volodymyr <volodymyrduke@gmail.com>
Co-authored-by: javalover123 <javalover123@foxmail.com>
Co-authored-by: Volodymyr <770925351@qq.com>
Co-authored-by: kksxf <flynnxue02@gmail.com>
Co-authored-by: flynnxue <flynnxue@lilithgames.com>
Co-authored-by: fang <56808812+zhibinF@users.noreply.github.com>
Co-authored-by: gejinxin <844156709@qq.com>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: Koyfin <1040080742@qq.com>
Co-authored-by: liuli <m_liuli@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants