New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][connector-jdbc] Support PostgreSQL based databases to use Copy statements to accelerate data writing. #6443
Conversation
Please fix CI. Thanks |
@Hisoka-X Ci. success. |
9fe7c80
to
89d31a4
Compare
...or-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcOptions.java
Outdated
Show resolved
Hide resolved
...seatunnel/connectors/seatunnel/jdbc/internal/executor/CopyManagerBatchStatementExecutor.java
Show resolved
Hide resolved
...seatunnel/connectors/seatunnel/jdbc/internal/executor/CopyManagerBatchStatementExecutor.java
Show resolved
Hide resolved
.../java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/executor/CopyManagerProxy.java
Outdated
Show resolved
Hide resolved
...in/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/JdbcOutputFormatBuilder.java
Show resolved
Hide resolved
613dd5a
to
2fd8072
Compare
@Hisoka-X Logs: 2024-03-07T06:54:28.7284111Z ##[debug]Code: 0
2024-03-07T06:54:28.7285606Z ##[debug]Signal: null
2024-03-07T06:54:29.6886880Z Command completed after 1 attempt(s).
2024-03-07T06:54:29.6922824Z ##[debug]Node Action run completed with exit code 0
2024-03-07T06:54:29.6927130Z ##[debug]Set output total_attempts = 1
2024-03-07T06:54:29.6927664Z ##[debug]Set output exit_code = 0
2024-03-07T06:54:29.6928412Z ##[debug]Finishing: Install
2024-03-07T06:54:29.6937509Z ##[debug]Evaluating condition for step: 'Check Dependencies Licenses'
2024-03-07T06:54:29.6939150Z ##[debug]Evaluating: success()
2024-03-07T06:54:29.6939655Z ##[debug]Evaluating success:
2024-03-07T06:54:29.6940152Z ##[debug]=> true
2024-03-07T06:54:29.6940621Z ##[debug]Result: true
2024-03-07T06:54:29.6941269Z ##[debug]Starting: Check Dependencies Licenses
2024-03-07T06:54:29.6951031Z ##[debug]Loading inputs
2024-03-07T06:54:29.6951867Z ##[debug]Loading env
2024-03-07T06:54:29.6994139Z ##[group]Run tools/dependencies/checkLicense.sh
2024-03-07T06:54:29.6994643Z �[36;1mtools/dependencies/checkLicense.sh�[0m
2024-03-07T06:54:29.7059615Z shell: /usr/bin/bash -e {0}
2024-03-07T06:54:29.7059956Z env:
2024-03-07T06:54:29.7060366Z JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.402-6/x64
2024-03-07T06:54:29.7061028Z JAVA_HOME_8_X64: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.402-6/x64
2024-03-07T06:54:29.7061550Z ##[endgroup]
2024-03-07T06:54:29.7193306Z ##[debug]/usr/bin/bash -e /home/runner/work/_temp/b429e46b-370e-4878-a7cd-f3659645b8ec.sh
...
...
2024-03-07T06:54:35.4433398Z [INFO] ----------------< org.apache.seatunnel:connector-jdbc >-----------------
2024-03-07T06:54:35.4434552Z [INFO] Building SeaTunnel : Connectors V2 : Jdbc 2.3.5-SNAPSHOT [65/194]
2024-03-07T06:54:35.4435672Z [INFO] --------------------------------[ jar ]---------------------------------
2024-03-07T06:54:35.8255340Z [INFO]
2024-03-07T06:54:35.8256438Z [INFO] --- maven-clean-plugin:3.3.2:clean (default-clean) @ connector-jdbc ---
2024-03-07T06:54:35.8267264Z [INFO] Deleting /home/runner/work/incubator-seatunnel/incubator-seatunnel/seatunnel-connectors-v2/connector-jdbc/target
2024-03-07T06:54:35.8472265Z [INFO]
2024-03-07T06:54:35.8477947Z [INFO] --- flatten-maven-plugin:1.3.0:clean (flatten.clean) @ connector-jdbc ---
2024-03-07T06:54:35.8480451Z [INFO] Deleting /home/runner/work/incubator-seatunnel/incubator-seatunnel/seatunnel-connectors-v2/connector-jdbc/.flattened-pom.xml
2024-03-07T06:54:35.8481960Z [INFO]
2024-03-07T06:54:35.8483001Z [INFO] --- maven-dependency-plugin:3.1.1:copy-dependencies (default-cli) @ connector-jdbc ---
2024-03-07T06:54:35.8484125Z [INFO] Skipping plugin execution
...
... |
Please fix CI |
82c556c
to
cbd0bfc
Compare
waiting for ci passed |
cbd0bfc
to
bb31156
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Purpose of this pull request
Support the COPY table1 from STDIN method for fast data writing,
[Feature] #6372
Does this PR introduce any user-facing change?
Add Connector-Jdbc Options [use_copy_statement] For PostgreSQL Driver.
How was this patch tested?
Add e2e test.
/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-3/src/test/resources/jdbc_postgres_source_and_sink_copy_stmt.conf
Check list
New License Guide
release-note
.