-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
source-mysql : chunking queries impl #29109
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
/legacy-test connector=connectors/source-mysql |
source-mysql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
source-mysql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
/legacy-test connector=connectors/source-mysql |
source-mysql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
source-mysql-strict-encrypt test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql-strict-encrypt:integrationTest | ❌ |
Acceptance tests | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql-strict-encrypt test
source-mysql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
/legacy-test connector=connectors/source-mysql |
source-mysql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
source-mysql-strict-encrypt test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql-strict-encrypt:integrationTest | ❌ |
Acceptance tests | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql-strict-encrypt test
source-mysql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
/legacy-test connector=connectors/source-mysql |
source-mysql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
source-mysql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ❌ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
/approve-and-merge reason="mysql tests passing" |
Closes #28186
Reading order + major changes :
MySqlInitialLoadRecordIterator.java
: New record that implements the chunking logic. Continuously makes queries to the database with the configured chunk size and previously processed high watermark of PK value evaluated. Previous logic of creating prepared statements has been moved to this class fromMySqlInitialLoadHandler
. e.g.MySqlQueryUtils.java
: Queries the size and average row length of each table in initial load.MySqlInitialLoadHandler.java
: Logic to calculate the chunk (limit size) used in the above queries. Each chunk should correspond to about ~1GB of data.MySqlInitialSyncStateIterator.java
,MySqlLoadGlobalStateManager.java
: Adds interface to update the primary key watermark processed. This is needed to dynamically calculate pk_max_1, pk_max_2 in the example above.MySqlInitialLoadSourceOperations.java
: Small bug fix to process null values. In line with the parent class atAbstractjdbcSourceOperations.java
.Some other changes :
Sync logs from local testing :
26af7a74_0437_42a9_9b17_981b4c659fff_logs_57_txt.txt