Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][connector-jdbc] Support PostgreSQL based databases to use Copy statements to accelerate data writing. #6443

Merged
merged 1 commit into from Mar 11, 2024

Conversation

mosence
Copy link
Contributor

@mosence mosence commented Mar 5, 2024

Purpose of this pull request

Support the COPY table1 from STDIN method for fast data writing,
[Feature] #6372

Does this PR introduce any user-facing change?

Add Connector-Jdbc Options [use_copy_statement] For PostgreSQL Driver.

How was this patch tested?

Add e2e test.
/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-3/src/test/resources/jdbc_postgres_source_and_sink_copy_stmt.conf

Check list

@Hisoka-X
Copy link
Member

Hisoka-X commented Mar 5, 2024

cc @hailin0 @Carl-Zhou-CN

@Hisoka-X
Copy link
Member

Hisoka-X commented Mar 5, 2024

Please fix CI. Thanks

@mosence
Copy link
Contributor Author

mosence commented Mar 6, 2024

@Hisoka-X Ci. success.
I often encounter Ci stopping in Kafka or RocketMQ e2e testing.

pom.xml Show resolved Hide resolved
@mosence mosence force-pushed the feature/pg_copy branch 2 times, most recently from 613dd5a to 2fd8072 Compare March 7, 2024 06:01
@mosence
Copy link
Contributor Author

mosence commented Mar 8, 2024

@Hisoka-X
I added it to knowledge-dependencies.txt and checked the debug log. The dependency:copy-dependencies command skipped connector-jdbc, which is a dependency in the connector-jdbc component.

Logs:

2024-03-07T06:54:28.7284111Z ##[debug]Code: 0
2024-03-07T06:54:28.7285606Z ##[debug]Signal: null
2024-03-07T06:54:29.6886880Z Command completed after 1 attempt(s).
2024-03-07T06:54:29.6922824Z ##[debug]Node Action run completed with exit code 0
2024-03-07T06:54:29.6927130Z ##[debug]Set output total_attempts = 1
2024-03-07T06:54:29.6927664Z ##[debug]Set output exit_code = 0
2024-03-07T06:54:29.6928412Z ##[debug]Finishing: Install
2024-03-07T06:54:29.6937509Z ##[debug]Evaluating condition for step: 'Check Dependencies Licenses'
2024-03-07T06:54:29.6939150Z ##[debug]Evaluating: success()
2024-03-07T06:54:29.6939655Z ##[debug]Evaluating success:
2024-03-07T06:54:29.6940152Z ##[debug]=> true
2024-03-07T06:54:29.6940621Z ##[debug]Result: true
2024-03-07T06:54:29.6941269Z ##[debug]Starting: Check Dependencies Licenses
2024-03-07T06:54:29.6951031Z ##[debug]Loading inputs
2024-03-07T06:54:29.6951867Z ##[debug]Loading env
2024-03-07T06:54:29.6994139Z ##[group]Run tools/dependencies/checkLicense.sh
2024-03-07T06:54:29.6994643Z �[36;1mtools/dependencies/checkLicense.sh�[0m
2024-03-07T06:54:29.7059615Z shell: /usr/bin/bash -e {0}
2024-03-07T06:54:29.7059956Z env:
2024-03-07T06:54:29.7060366Z   JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.402-6/x64
2024-03-07T06:54:29.7061028Z   JAVA_HOME_8_X64: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.402-6/x64
2024-03-07T06:54:29.7061550Z ##[endgroup]
2024-03-07T06:54:29.7193306Z ##[debug]/usr/bin/bash -e /home/runner/work/_temp/b429e46b-370e-4878-a7cd-f3659645b8ec.sh
...
...
2024-03-07T06:54:35.4433398Z [INFO] ----------------< org.apache.seatunnel:connector-jdbc >-----------------
2024-03-07T06:54:35.4434552Z [INFO] Building SeaTunnel : Connectors V2 : Jdbc 2.3.5-SNAPSHOT        [65/194]
2024-03-07T06:54:35.4435672Z [INFO] --------------------------------[ jar ]---------------------------------
2024-03-07T06:54:35.8255340Z [INFO] 
2024-03-07T06:54:35.8256438Z [INFO] --- maven-clean-plugin:3.3.2:clean (default-clean) @ connector-jdbc ---
2024-03-07T06:54:35.8267264Z [INFO] Deleting /home/runner/work/incubator-seatunnel/incubator-seatunnel/seatunnel-connectors-v2/connector-jdbc/target
2024-03-07T06:54:35.8472265Z [INFO] 
2024-03-07T06:54:35.8477947Z [INFO] --- flatten-maven-plugin:1.3.0:clean (flatten.clean) @ connector-jdbc ---
2024-03-07T06:54:35.8480451Z [INFO] Deleting /home/runner/work/incubator-seatunnel/incubator-seatunnel/seatunnel-connectors-v2/connector-jdbc/.flattened-pom.xml
2024-03-07T06:54:35.8481960Z [INFO] 
2024-03-07T06:54:35.8483001Z [INFO] --- maven-dependency-plugin:3.1.1:copy-dependencies (default-cli) @ connector-jdbc ---
2024-03-07T06:54:35.8484125Z [INFO] Skipping plugin execution
...
...

@Hisoka-X
Copy link
Member

Hisoka-X commented Mar 8, 2024

Please fix CI

@mosence mosence force-pushed the feature/pg_copy branch 6 times, most recently from 82c556c to cbd0bfc Compare March 8, 2024 14:53
@hailin0
Copy link
Contributor

hailin0 commented Mar 11, 2024

waiting for ci passed

@mosence
Copy link
Contributor Author

mosence commented Mar 11, 2024

@Hisoka-X
@hailin0

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hailin0 hailin0 merged commit ca4a65f into apache:dev Mar 11, 2024
6 checks passed
@mosence mosence deleted the feature/pg_copy branch March 11, 2024 12:41
@mosence mosence restored the feature/pg_copy branch March 11, 2024 12:41
@mosence mosence deleted the feature/pg_copy branch March 11, 2024 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants