-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Async Code V0: Flush Worker Improvements. #26384
Conversation
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|---|---|---|
source-alloydb |
2.0.28 |
✅ | ✅ |
source-alloydb-strict-encrypt |
2.0.28 |
🔵 (ignored) |
🔵 (ignored) |
source-azure-blob-storage |
0.1.0 |
✅ | ✅ |
source-bigquery |
0.2.3 |
✅ | ✅ |
source-clickhouse |
0.1.17 |
✅ | ✅ |
source-clickhouse-strict-encrypt |
0.1.17 |
🔵 (ignored) |
🔵 (ignored) |
source-cockroachdb |
0.1.22 |
✅ | ✅ |
source-cockroachdb-strict-encrypt |
0.1.22 |
🔵 (ignored) |
🔵 (ignored) |
source-db2 |
0.1.19 |
✅ | ✅ |
source-db2-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
source-dynamodb |
0.1.2 |
✅ | ✅ |
source-e2e-test |
2.1.4 |
✅ | ✅ |
source-e2e-test-cloud |
2.1.4 |
🔵 (ignored) |
🔵 (ignored) |
source-elasticsearch |
0.1.1 |
✅ | ✅ |
source-jdbc |
0.3.5 |
🔵 (ignored) |
🔵 (ignored) |
source-kafka |
0.2.3 |
✅ | ✅ |
source-mongodb-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
source-mongodb-v2 |
0.1.19 |
✅ | ✅ |
source-mssql |
1.0.16 |
✅ | ✅ |
source-mssql-strict-encrypt |
1.0.16 |
🔵 (ignored) |
🔵 (ignored) |
source-mysql |
2.0.21 |
✅ | ✅ |
source-mysql-strict-encrypt |
2.0.21 |
🔵 (ignored) |
🔵 (ignored) |
source-oracle |
0.3.24 |
✅ | ✅ |
source-oracle-strict-encrypt |
0.3.24 |
🔵 (ignored) |
🔵 (ignored) |
source-postgres |
2.0.29 |
✅ | ✅ |
source-postgres-strict-encrypt |
2.0.29 |
🔵 (ignored) |
🔵 (ignored) |
source-redshift |
0.3.16 |
✅ | ✅ |
source-relational-db |
0.3.1 |
🔵 (ignored) |
🔵 (ignored) |
source-scaffold-java-jdbc |
0.1.0 |
🔵 (ignored) |
🔵 (ignored) |
source-sftp |
0.1.2 |
✅ | ✅ |
source-snowflake |
0.1.34 |
✅ | ✅ |
source-teradata |
0.1.0 |
✅ | ✅ |
source-tidb |
0.2.4 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
❌ Destinations (49)
Connector | Version | Changelog | Publish |
---|---|---|---|
destination-azure-blob-storage |
0.2.0 |
✅ | ✅ |
destination-bigquery |
1.4.3 |
✅ | ✅ |
destination-bigquery-denormalized |
1.4.1 |
✅ | ✅ |
destination-cassandra |
0.1.4 |
✅ | ✅ |
destination-clickhouse |
0.2.3 |
✅ | ✅ |
destination-clickhouse-strict-encrypt |
0.2.3 |
🔵 (ignored) |
🔵 (ignored) |
destination-csv |
1.0.0 |
✅ | ✅ |
destination-databricks |
1.0.2 |
✅ | ✅ |
destination-dev-null |
0.3.0 |
🔵 (ignored) |
🔵 (ignored) |
destination-doris |
0.1.0 |
✅ | ✅ |
destination-dynamodb |
0.1.7 |
✅ | ✅ |
destination-e2e-test |
0.3.0 |
✅ | ✅ |
destination-elasticsearch |
0.1.6 |
✅ | ✅ |
destination-elasticsearch-strict-encrypt |
0.1.6 |
🔵 (ignored) |
🔵 (ignored) |
destination-exasol |
0.1.1 |
✅ | ✅ |
destination-gcs |
0.3.0 |
✅ | ✅ |
destination-iceberg |
0.1.0 |
✅ | ✅ |
destination-kafka |
0.1.10 |
✅ | ✅ |
destination-keen |
0.2.4 |
✅ | ✅ |
destination-kinesis |
0.1.5 |
✅ | ✅ |
destination-local-json |
0.2.11 |
✅ | ✅ |
destination-mariadb-columnstore |
0.1.7 |
✅ | ✅ |
destination-mongodb |
0.1.9 |
✅ | ✅ |
destination-mongodb-strict-encrypt |
0.1.9 |
🔵 (ignored) |
🔵 (ignored) |
destination-mqtt |
0.1.3 |
✅ | ✅ |
destination-mssql |
0.1.23 |
✅ | ✅ |
destination-mssql-strict-encrypt |
0.1.23 |
🔵 (ignored) |
🔵 (ignored) |
destination-mysql |
0.1.20 |
✅ | ✅ |
destination-mysql-strict-encrypt |
❌ 0.1.21 (mismatch: 0.1.20 ) |
🔵 (ignored) |
🔵 (ignored) |
destination-oracle |
0.1.19 |
✅ | ✅ |
destination-oracle-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
destination-postgres |
0.3.27 |
✅ | ✅ |
destination-postgres-strict-encrypt |
0.3.27 |
🔵 (ignored) |
🔵 (ignored) |
destination-pubsub |
0.2.0 |
✅ | ✅ |
destination-pulsar |
0.1.3 |
✅ | ✅ |
destination-r2 |
0.1.0 |
✅ | ✅ |
destination-redis |
0.1.4 |
✅ | ✅ |
destination-redpanda |
0.1.0 |
✅ | ✅ |
destination-redshift |
0.4.7 |
✅ | ✅ |
destination-rockset |
0.1.4 |
✅ | ✅ |
destination-s3 |
0.4.1 |
✅ | ❌ (diff seed version) |
destination-s3-glue |
0.1.7 |
✅ | ✅ |
destination-scylla |
0.1.3 |
✅ | ✅ |
destination-selectdb |
0.1.0 |
✅ | ✅ |
destination-snowflake |
1.0.4 |
✅ | ✅ |
destination-starburst-galaxy |
0.0.1 |
✅ | ✅ |
destination-teradata |
0.1.1 |
✅ | ✅ |
destination-tidb |
0.1.1 |
✅ | ✅ |
destination-yugabytedb |
0.1.1 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled. |
@@ -80,6 +81,16 @@ public MemoryBoundedLinkedBlockingQueue.MemoryItem<E> poll() { | |||
return null; | |||
} | |||
|
|||
@Override |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it make sense to merge into this branch and have it all merged as one since both branches update this method
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'm ok either way. i'll merge that branch in right after
Incorporate the changes from #26178 . - Update queue method from setMaxMemory to addMaxMemory. - Update work retrieval logic to only assign work - if there are free worker threads - to account for in-progress worker threads
Incorporate the changes from airbytehq#26178 . - Update queue method from setMaxMemory to addMaxMemory. - Update work retrieval logic to only assign work - if there are free worker threads - to account for in-progress worker threads
What
Incorporate the changes from #26178 .
How
Recommended reading order
All files.
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user?
For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.
If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Actions
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.0.0.1
Dockerfile
has version0.0.1
README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog with an entry for the initial version. See changelog exampledocs/integrations/README.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
Updating a connector
Community member or Airbyter
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
Connector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changes