-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JDBC connections to support checkpointing #24604
Conversation
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|
- See "Actionable Items" below for how to resolve warnings and errors.
❌ Destinations (22)
Connector | Version | Changelog | Publish |
---|---|---|---|
destination-azure-blob-storage |
0.2.0 |
✅ | ✅ |
destination-clickhouse |
0.2.3 |
✅ | ❌ (diff seed version) |
destination-clickhouse-strict-encrypt |
0.2.3 |
🔵 (ignored) |
🔵 (ignored) |
destination-databricks |
1.0.0 |
✅ | ✅ |
destination-dynamodb |
0.1.7 |
✅ | ✅ |
destination-exasol |
0.1.1 |
✅ | ✅ |
destination-gcs |
0.2.16 |
✅ | ✅ |
destination-mariadb-columnstore |
0.1.7 |
✅ | ✅ |
destination-mssql |
0.1.23 |
✅ | ❌ (diff seed version) |
destination-mssql-strict-encrypt |
0.1.23 |
🔵 (ignored) |
🔵 (ignored) |
destination-mysql |
0.1.20 |
✅ | ✅ |
destination-mysql-strict-encrypt |
❌ 0.1.21 (mismatch: 0.1.20 ) |
🔵 (ignored) |
🔵 (ignored) |
destination-oracle |
0.1.19 |
✅ | ✅ |
destination-oracle-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
destination-postgres |
0.3.27 |
✅ | ❌ (diff seed version) |
destination-postgres-strict-encrypt |
0.3.27 |
🔵 (ignored) |
🔵 (ignored) |
destination-redshift |
0.4.5 |
✅ | ✅ |
destination-rockset |
0.1.4 |
✅ | ✅ |
destination-snowflake |
0.4.61 |
✅ | ✅ |
destination-teradata |
0.1.1 |
✅ | ✅ |
destination-tidb |
0.1.1 |
✅ | ❌ (diff seed version) |
destination-yugabytedb |
0.1.1 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
👀 Other Modules (1)
- base-normalization
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml ), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
/test connector=connectors/destination-snowflake
Build PassedTest summary info:
|
This comment was marked as outdated.
This comment was marked as outdated.
/test connector=connectors/destination-mysql
Build FailedTest summary info:
|
This comment was marked as outdated.
This comment was marked as outdated.
/test connector=connectors/destination-postgres
Build PassedTest summary info:
|
/test connector=connectors/destination-mssql
Build PassedTest summary info:
|
This comment was marked as outdated.
This comment was marked as outdated.
/test connector=connectors/destination-oracle
|
695dc2f
to
5ce1aaf
Compare
/test connector=connectors/destination-clickhouse
Build PassedTest summary info:
|
/publish connector=connectors/destination-clickhouse run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/destination-postgres run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/destination-postgres-strict-encrypt run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/destination-mssql run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/destination-mssql-strict-encrypt run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/destination-tidb run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
117b601
to
a18f9fa
Compare
/publish connector=connectors/destination-clickhouse-strict-encrypt run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
What
Closes #23678
Change is only isolated to changing the functionality of table creation, record writing, and tear down process
Separate PRs will follow that rely on this change but those PRs include:
tmpTableName
no longer being usedReason for separate PRs is to keep changes as simple as possible to review. Original changes include bloat that would be nice to haves but not functionally relevant
Affected destination connectors and tested manually (check if tables are written to and
tmp
tables aren't created):How
onStartFunction
tmp
table creationairbyte_raw
table creationrecordWriterFunction
tmp
tableairbyte_raw
tableonCloseFunction
tmp
table cleanupairbyte_raw
table creation (moved toonStartFunction
)Recommended reading order
JdbcBufferedConsumerFactory.java
StagingConsumerFactory.java
🚨 User Impact 🚨
No user breaking changes, all data will still be written to
airbyte_raw
table as expected, going forward this will also remove the usage oftmp
table creationsFor connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.
If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.0.0.1
Dockerfile
has version0.0.1
README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog with an entry for the initial version. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with
airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.Code reviews completed
Connector version has been incremented
Dockerfile
has updated versionDocumentation updated
README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
with an entry for the new version. See changelog examplePR name follows PR naming conventions
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changes