Migrate connectors to use our python base image (Round 1) #31543

alafanechere · 2023-10-18T10:03:01Z

What

Migrate the following connectors to use our base image:

source-sendgrid
source-azure-blob-storage
source-salesforce
source-google-analytics-data-api

(Ignoring source-s3 because we have to implement a build_customization.py following @flash1293 recent nltk changes to the Dockerfile.)

How

Run

airbyte-ci connectors --name=source-sendgrid --name=source-azure-blob-storage --name=source-salesforce --name=source-google-analytics-data-api migrate-to-base-image 31543

🚨 User Impact 🚨

None if the tests pass.

vercel · 2023-10-18T10:03:02Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 18, 2023 3:43pm

alafanechere · 2023-10-18T10:03:14Z

Current dependencies on/for this PR:

master
- PR python-connector-base: install socat #31541
  - PR Migrate connectors to use our python base image (Round 1) #31543 👈

This comment was auto-generated by Graphite.

github-actions · 2023-10-18T10:03:30Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

airbyte-oss-build-runner · 2023-10-18T10:21:08Z

source-sendgrid test report (commit `ae3c6214bc`) - ✅

⏲️ Total pipeline duration: 04mn42s

Step	Result
Build source-sendgrid docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-sendgrid	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-sendgrid test

bazarnov · 2023-10-18T10:20:11Z

docs/integrations/sources/azure-blob-storage.md

@@ -178,5 +178,6 @@ There are currently no options for JSONL parsing.

 | Version | Date       | Pull Request                                    | Subject                                                                 |
 |:--------|:-----------|:------------------------------------------------|:------------------------------------------------------------------------|
+| 0.2.1 | 2023-10-18 | [31543](https://github.com/airbytehq/airbyte/pull/31543) | Use our base image and remove Dockerfile |


nit: "Replace Dockerfile with the python-connector-base image"?

airbyte-oss-build-runner · 2023-10-18T10:22:01Z

source-azure-blob-storage test report (commit `ae3c6214bc`) - ✅

⏲️ Total pipeline duration: 05mn35s

Step	Result
Build source-azure-blob-storage docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-azure-blob-storage	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-azure-blob-storage test

bazarnov · 2023-10-18T10:23:42Z

@alafanechere You'll probably face the rate-limits for the Google Analytics Data API very shortly, it's recommended to have a separate PR for it, in order not to be blocked by the single connector and be able to merge timely.

alafanechere · 2023-10-18T10:24:21Z

@alafanechere You'll probably face the rate-limits for the Google Analytics Data API very shortly, it's recommended to have a separate PR for it, in order not to be blocked by the single connector and be able to merge timely.

Thanks for the tips, will do if it gets annoying.

bazarnov

LGTM, with the small change to the changelog.

airbyte-oss-build-runner · 2023-10-18T11:38:22Z

source-salesforce test report (commit `ae3c6214bc`) - ✅

⏲️ Total pipeline duration: 81mn55s

Step	Result
Build source-salesforce docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-salesforce	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-salesforce test

airbyte-oss-build-runner · 2023-10-18T11:41:26Z

source-google-analytics-data-api test report (commit `ae3c6214bc`) - ✅

⏲️ Total pipeline duration: 85mn00s

Step	Result
Build source-google-analytics-data-api docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-google-analytics-data-api	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-analytics-data-api test

alafanechere · 2023-10-18T11:51:02Z

@artem1205 the CI time for these connector did not get degraded. The image build duration remains around 30s.

alafanechere · 2023-10-18T11:51:42Z

I will merge this PR once #31541 is approved and on master.

airbyte-oss-build-runner · 2023-10-18T13:53:05Z

source-sendgrid test report (commit `328b0d1067`) - ❌

⏲️ Total pipeline duration: 04mn25s

Step	Result
Build source-sendgrid docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-sendgrid	❌
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-sendgrid test

airbyte-oss-build-runner · 2023-10-18T13:53:31Z

source-azure-blob-storage test report (commit `328b0d1067`) - ✅

⏲️ Total pipeline duration: 05mn15s

Step	Result
Build source-azure-blob-storage docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-azure-blob-storage	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-azure-blob-storage test

airbyte-oss-build-runner · 2023-10-18T15:15:21Z

source-google-analytics-data-api test report (commit `328b0d1067`) - ❌

⏲️ Total pipeline duration: 87mn04s

Step	Result
Build source-google-analytics-data-api docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-google-analytics-data-api	❌
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-analytics-data-api test

airbyte-oss-build-runner · 2023-10-18T15:26:17Z

source-salesforce test report (commit `328b0d1067`) - ❌

⏲️ Total pipeline duration: 98mn00s

Step	Result
Build source-salesforce docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-salesforce	❌
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-salesforce test

…ur_python_base_image_Round_1_

airbyte-oss-build-runner · 2023-10-18T15:44:51Z

source-azure-blob-storage test report (commit `d46cd29166`) - ✅

⏲️ Total pipeline duration: 02mn41s

Step	Result
Build source-azure-blob-storage docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-azure-blob-storage	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-azure-blob-storage test

airbyte-oss-build-runner · 2023-10-18T15:45:02Z

source-sendgrid test report (commit `d46cd29166`) - ✅

⏲️ Total pipeline duration: 02mn54s

Step	Result
Build source-sendgrid docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-sendgrid	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-sendgrid test

airbyte-oss-build-runner · 2023-10-18T16:03:53Z

source-google-analytics-data-api test report (commit `d46cd29166`) - ❌

⏲️ Total pipeline duration: 21mn43s

Step	Result
Build source-google-analytics-data-api docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌
Code format checks	✅
Validate metadata for source-google-analytics-data-api	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-analytics-data-api test

airbyte-oss-build-runner · 2023-10-18T16:08:51Z

source-salesforce test report (commit `d46cd29166`) - ✅

⏲️ Total pipeline duration: 26mn42s

Step	Result
Build source-salesforce docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-salesforce	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-salesforce test

alafanechere · 2023-10-18T18:05:53Z

Merging as source-google-analytics-data-api CAT failure is transient (probably due to a rate limiting issue): it passed two times in a row and failed on the third run (without any code change).

…31543)

alafanechere mentioned this pull request Oct 18, 2023

python-connector-base: install socat #31541

Merged

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/azure-blob-storage labels Oct 18, 2023

vercel bot deployed to Preview October 18, 2023 10:05 View deployment

alafanechere force-pushed the augustin/10-18-Migrate_connectors_to_use_our_python_base_image_Round_1_ branch from 08ff807 to ae3c621 Compare October 18, 2023 10:11

octavia-squidington-iii added connectors/source/salesforce connectors/source/sendgrid connectors/source/google-analytics-data-api labels Oct 18, 2023

alafanechere marked this pull request as ready for review October 18, 2023 10:14

vercel bot deployed to Preview October 18, 2023 10:15 View deployment

alafanechere requested review from bazarnov and artem1205 October 18, 2023 10:15

octavia-squidington-iv requested review from a team October 18, 2023 10:16

bazarnov reviewed Oct 18, 2023

View reviewed changes

octavia-squidington-iv requested a review from a team October 18, 2023 10:22

bazarnov approved these changes Oct 18, 2023

View reviewed changes

vercel bot deployed to Preview October 18, 2023 11:49 View deployment

artem1205 approved these changes Oct 18, 2023

View reviewed changes

Base automatically changed from augustin/10-17-python-connector-base_install_socat to master October 18, 2023 13:43

alafanechere added 5 commits October 18, 2023 15:44

source-azure-blob-storage migration

48984bf

source-google-analytics-data-api migration

e9bed94

source-salesforce migration

73c9677

source-sendgrid migration

23429da

[skip ci] reword the changelog entry

0383876

alafanechere force-pushed the augustin/10-18-Migrate_connectors_to_use_our_python_base_image_Round_1_ branch from 728060c to 0383876 Compare October 18, 2023 13:45

Trigger CI

328b0d1

Merge branch 'master' into augustin/10-18-Migrate_connectors_to_use_o…

d46cd29

…ur_python_base_image_Round_1_

vercel bot deployed to Preview October 18, 2023 15:43 View deployment

alafanechere merged commit c544183 into master Oct 18, 2023
26 of 28 checks passed

alafanechere deleted the augustin/10-18-Migrate_connectors_to_use_our_python_base_image_Round_1_ branch October 18, 2023 18:06

alafanechere mentioned this pull request Oct 20, 2023

Remove dockerfiles for certified connectors and build them using our new build process #30243

Closed

ariesgun pushed a commit to ariesgun/airbyte that referenced this pull request Oct 20, 2023

Migrate connectors to use our python base image (Round 1) (airbytehq#…

62873d3

…31543)

ariesgun pushed a commit to ariesgun/airbyte that referenced this pull request Oct 23, 2023

Migrate connectors to use our python base image (Round 1) (airbytehq#…

b272368

…31543)

aaronsteers mentioned this pull request Oct 24, 2023

✨ Destination DuckDB: Use latest DuckDB 0.9.1 libraries 🚨(breaking change)🚨 #31744

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate connectors to use our python base image (Round 1) #31543

Migrate connectors to use our python base image (Round 1) #31543

alafanechere commented Oct 18, 2023 •

edited

Loading

vercel bot commented Oct 18, 2023 •

edited

Loading

alafanechere commented Oct 18, 2023

github-actions bot commented Oct 18, 2023 •

edited by alafanechere

Loading

airbyte-oss-build-runner commented Oct 18, 2023

bazarnov Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

bazarnov commented Oct 18, 2023

alafanechere commented Oct 18, 2023

bazarnov left a comment

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

alafanechere commented Oct 18, 2023

alafanechere commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

alafanechere commented Oct 18, 2023

Migrate connectors to use our python base image (Round 1) #31543

Migrate connectors to use our python base image (Round 1) #31543

Conversation

alafanechere commented Oct 18, 2023 • edited Loading

What

How

🚨 User Impact 🚨

vercel bot commented Oct 18, 2023 • edited Loading

alafanechere commented Oct 18, 2023

github-actions bot commented Oct 18, 2023 • edited by alafanechere Loading

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Oct 18, 2023

source-sendgrid test report (commit ae3c6214bc) - ✅

bazarnov Oct 18, 2023

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 18, 2023

source-azure-blob-storage test report (commit ae3c6214bc) - ✅

bazarnov commented Oct 18, 2023

alafanechere commented Oct 18, 2023

bazarnov left a comment

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 18, 2023

source-salesforce test report (commit ae3c6214bc) - ✅

airbyte-oss-build-runner commented Oct 18, 2023

source-google-analytics-data-api test report (commit ae3c6214bc) - ✅

alafanechere commented Oct 18, 2023

alafanechere commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

source-sendgrid test report (commit 328b0d1067) - ❌

airbyte-oss-build-runner commented Oct 18, 2023

source-azure-blob-storage test report (commit 328b0d1067) - ✅

airbyte-oss-build-runner commented Oct 18, 2023

source-google-analytics-data-api test report (commit 328b0d1067) - ❌

airbyte-oss-build-runner commented Oct 18, 2023

source-salesforce test report (commit 328b0d1067) - ❌

airbyte-oss-build-runner commented Oct 18, 2023

source-azure-blob-storage test report (commit d46cd29166) - ✅

airbyte-oss-build-runner commented Oct 18, 2023

source-sendgrid test report (commit d46cd29166) - ✅

airbyte-oss-build-runner commented Oct 18, 2023

source-google-analytics-data-api test report (commit d46cd29166) - ❌

airbyte-oss-build-runner commented Oct 18, 2023

source-salesforce test report (commit d46cd29166) - ✅

alafanechere commented Oct 18, 2023

alafanechere commented Oct 18, 2023 •

edited

Loading

vercel bot commented Oct 18, 2023 •

edited

Loading

github-actions bot commented Oct 18, 2023 •

edited by alafanechere

Loading

source-sendgrid test report (commit `ae3c6214bc`) - ✅

source-azure-blob-storage test report (commit `ae3c6214bc`) - ✅

source-salesforce test report (commit `ae3c6214bc`) - ✅

source-google-analytics-data-api test report (commit `ae3c6214bc`) - ✅

source-sendgrid test report (commit `328b0d1067`) - ❌

source-azure-blob-storage test report (commit `328b0d1067`) - ✅

source-google-analytics-data-api test report (commit `328b0d1067`) - ❌

source-salesforce test report (commit `328b0d1067`) - ❌

source-azure-blob-storage test report (commit `d46cd29166`) - ✅

source-sendgrid test report (commit `d46cd29166`) - ✅

source-google-analytics-data-api test report (commit `d46cd29166`) - ❌

source-salesforce test report (commit `d46cd29166`) - ✅