Source Salesforce: use concurrent CDK for full refresh syncs by clnoll · Pull Request #32535 · airbytehq/airbyte

Catherine Noll (clnoll) · 2023-11-15T02:14:55Z

Adds concurrency to the Source Salesforce connector, for streams that are configured for full refresh.

vercel · 2023-11-15T02:15:01Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Dec 4, 2023 6:39pm

github-actions · 2023-11-15T02:15:16Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Augustin (alafanechere) · 2023-11-15T08:02:43Z

Catherine Noll (@clnoll) I'm updating your branch so that you get the fresh CI workflow for format

Alexandre Girard (girarda)

looks great !

Alexandre Girard (girarda) · 2023-11-16T16:58:08Z

+        ),
+    ),
+)
+def test_unspecified_and_incremental_streams_are_not_concurrent(stream_config, stream_names, catalog_stream_names) -> None:


what's an unspecified stream?

This just means the catalog hasn't been set. It shouldn't happen during reads but I thought it was worth testing just in case.

Maxime Carbonneau-Leclerc (maxi297)

Awesome! 🎉

Maxime Carbonneau-Leclerc (maxi297) · 2023-11-17T14:11:11Z

+
+    def _get_sync_mode_from_catalog(self, stream: Stream) -> Optional[SyncMode]:
+        if self.catalog:
+            for catalog_stream in self.catalog.streams:


Outside of the scope of this PR: I see this looping logic more and more. Would there be a way to have a ConfiguredAirbyteCatalog.get_stream_by_name(name: str)? This would greatly improve the readability/maintenance of our code

Serhii Lazebnyi (lazebnyi)

nice!

…rrentSource

Catherine Noll (clnoll) marked this pull request as draft November 15, 2023 02:15

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues connectors/source/salesforce labels Nov 15, 2023

Octavia Squidington IV (octavia-squidington-iv) requested a review from a team November 15, 2023 02:16

vercel Bot deployed to Preview November 15, 2023 08:04 View deployment

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch 4 times, most recently from 6a608e3 to 9979861 Compare November 16, 2023 14:31

Catherine Noll (clnoll) marked this pull request as ready for review November 16, 2023 14:44

Catherine Noll (clnoll) requested review from Alexandre Girard (girarda) and Maxime Carbonneau-Leclerc (maxi297) November 16, 2023 16:16

Alexandre Girard (girarda) approved these changes Nov 16, 2023

View reviewed changes

Maxime Carbonneau-Leclerc (maxi297) approved these changes Nov 17, 2023

View reviewed changes

Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Nov 28, 2023

vercel Bot deployed to Preview November 28, 2023 19:31 View deployment

vercel Bot deployed to Preview November 28, 2023 19:44 View deployment

Catherine Noll (clnoll) requested a review from Serhii Lazebnyi (lazebnyi) November 28, 2023 20:33

Serhii Lazebnyi (lazebnyi) approved these changes Nov 29, 2023

View reviewed changes

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch from 4c8a7e4 to 6ef3f0e Compare November 29, 2023 23:36

vercel Bot deployed to Preview November 29, 2023 23:38 View deployment

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/google-search-console label Nov 29, 2023

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch from 99026bf to 968ba5c Compare November 30, 2023 00:45

vercel Bot deployed to Preview November 30, 2023 00:46 View deployment

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch 3 times, most recently from 1ea0549 to 968ba5c Compare November 30, 2023 01:52

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch from 4e0715d to 5b05106 Compare December 1, 2023 20:07

vercel Bot deployed to Preview December 1, 2023 20:10 View deployment

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch from 5b05106 to d24d535 Compare December 3, 2023 17:09

vercel Bot deployed to Preview December 3, 2023 17:11 View deployment

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch from d24d535 to de6bc9f Compare December 3, 2023 17:18

vercel Bot deployed to Preview December 3, 2023 17:20 View deployment

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch from de6bc9f to 295d4d2 Compare December 3, 2023 17:57

vercel Bot deployed to Preview December 3, 2023 17:59 View deployment

Source Salesforce: use concurrent CDK for full refresh syncs

db6d007

Catherine Noll (clnoll) force-pushed the source-salesforce-concurrent-full-refresh branch from 295d4d2 to db6d007 Compare December 4, 2023 16:49

vercel Bot deployed to Preview December 4, 2023 16:50 View deployment

Source Salesforce: pin airbyte-cdk until its updated to use the Concu…

e403794

…rrentSource

Catherine Noll (clnoll) merged commit c11041a into master Dec 4, 2023

Catherine Noll (clnoll) deleted the source-salesforce-concurrent-full-refresh branch December 4, 2023 19:46

Navigation Menu

Repository navigation

Source Salesforce: use concurrent CDK for full refresh syncs - #32535

Source Salesforce: use concurrent CDK for full refresh syncs#32535
Catherine Noll (clnoll) merged 2 commits into
masterfrom
source-salesforce-concurrent-full-refresh

Catherine Noll (clnoll) commented Nov 15, 2023 •

edited

Loading

Uh oh!

vercel Bot commented Nov 15, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 15, 2023 •

edited by clnoll

Loading

Uh oh!

Augustin (alafanechere) commented Nov 15, 2023

Uh oh!

Alexandre Girard (girarda) left a comment

Uh oh!

Alexandre Girard (girarda) Nov 16, 2023

Uh oh!

Catherine Noll (clnoll) Nov 17, 2023 •

edited

Loading

Uh oh!

Maxime Carbonneau-Leclerc (maxi297) left a comment

Uh oh!

Maxime Carbonneau-Leclerc (maxi297) Nov 17, 2023

Uh oh!

Serhii Lazebnyi (lazebnyi) left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

Catherine Noll (clnoll) commented Nov 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Nov 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 15, 2023 • edited by clnoll Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

Augustin (alafanechere) commented Nov 15, 2023

Uh oh!

Alexandre Girard (girarda) left a comment

Choose a reason for hiding this comment

Uh oh!

Alexandre Girard (girarda) Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

Catherine Noll (clnoll) Nov 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Maxime Carbonneau-Leclerc (maxi297) left a comment

Choose a reason for hiding this comment

Uh oh!

Maxime Carbonneau-Leclerc (maxi297) Nov 17, 2023

Choose a reason for hiding this comment

Uh oh!

Serhii Lazebnyi (lazebnyi) left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Catherine Noll (clnoll) commented Nov 15, 2023 •

edited

Loading

vercel Bot commented Nov 15, 2023 •

edited

Loading

github-actions Bot commented Nov 15, 2023 •

edited by clnoll

Loading

Catherine Noll (clnoll) Nov 17, 2023 •

edited

Loading