Source SalesForce: Remove pagination and query limits #25700

artem1205 · 2023-04-30T08:58:45Z

What

Resolving https://github.com/airbytehq/oncall/issues/1787

How

Remove pagination and query limits; use only stream slices instead.
After these changes, connector successfully downloaded about 440k records.

🚨 User Impact 🚨

no breaking changes

Pre-merge Actions

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

github-actions · 2023-04-30T08:59:09Z

artem1205 · 2023-04-30T09:00:34Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/4843479008
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/4843479008
Python tests coverage:

Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
source_salesforce/utils.py                       8      0   100%
source_salesforce/__init__.py                    2      0   100%
source_salesforce/source.py                    102      6    94%
source_salesforce/streams.py                   409     32    92%
source_salesforce/api.py                       155     14    91%
source_salesforce/exceptions.py                  8      1    88%
source_salesforce/rate_limiting.py              22      3    86%
source_salesforce/availability_strategy.py      17      8    53%
----------------------------------------------------------------
TOTAL                                          723     64    91%
Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
source_salesforce/__init__.py                    2      0   100%
source_salesforce/exceptions.py                  8      1    88%
source_salesforce/api.py                       155     21    86%
source_salesforce/availability_strategy.py      17      3    82%
source_salesforce/streams.py                   409     91    78%
source_salesforce/rate_limiting.py              22      6    73%
source_salesforce/source.py                    102     34    67%
source_salesforce/utils.py                       8      7    12%
----------------------------------------------------------------
TOTAL                                          723    163    77%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 39 passed, 2 skipped in 1300.40s (0:21:40) ==================

artem1205 · 2023-04-30T10:07:24Z

I'd like to make a publish with dev tag, and pin this version for Doxo.com customer only.

artem1205 · 2023-04-30T12:22:10Z

/publish connector=connectors/source-salesforce pre-release=true

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/4844283865

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-salesforce	2.0.12-dev.e5a5ac90	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

maxi297 · 2023-05-04T15:41:37Z

Is a review still required on this @artem1205 ?

artem1205 · 2023-05-05T12:15:53Z

@maxi297 , yes, i've Reduced slice interval to 30 days, we can try this once more in test Doxo connection

airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py

Cause OOM for large responses

# Conflicts: # docs/integrations/sources/salesforce.md

artem1205 · 2023-05-18T09:12:57Z

/publish connector=connectors/source-salesforce pre-release=true

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/5012429260

| Connector | Version | Did it publish? |
| --- | --- | --- | --- |
| connectors/source-salesforce | 2.0.13-dev.b206a930 | ✅ |

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

maxi297

Tested in production as AB testing and it worked so I'll approve

maxi297

Tested in production as AB testing and it worked so I'll approve

artem1205 · 2023-05-18T18:29:26Z

/publish connector=connectors/source-salesforce

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/5017075910

| Connector | Version | Did it publish? |
| --- | --- | --- | --- |
| connectors/source-salesforce | 2.0.13 | ✅ |

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

sherifnada · 2023-05-18T21:04:13Z

airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py

@@ -432,7 +432,7 @@ def download_data(self, url: str, chunk_size: int = 1024) -> tuple[str, str]:
        # set filepath for binary data from response
        tmp_file = os.path.realpath(os.path.basename(url))
        with closing(self._send_http_request("GET", f"{url}/results", stream=True)) as response, open(tmp_file, "wb") as data_file:
-            response_encoding = response.apparent_encoding or response.encoding or self.encoding
+            response_encoding = response.encoding or self.encoding


can you clarify why you made this change?

response.apparent_encoding method tries to guess encoding by reading the whole response content. this caused OOM for large responses.

makes sense!

sherifnada · 2023-05-18T21:04:29Z

airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py

@@ -579,7 +579,7 @@ def transform_empty_string_to_none(instance: Any, schema: Any):

 class IncrementalRestSalesforceStream(RestSalesforceStream, ABC):
    state_checkpoint_interval = 500
-    STREAM_SLICE_STEP = 120
+    STREAM_SLICE_STEP = 30


what is the impact of changing this? Do other salesforce syncs now take longer?

By decreasing the window we make responses smaller and thus iterate over them quicker.
According to screen it takes roughly the same time to sync (or even slightly less):
First sync was performed using 2.0.13-dev.b206a930, next 2 using 2.0.9

# Conflicts: # airbyte-config-oss/init-oss/src/main/resources/seed/oss_registry.json

sherifnada · 2023-05-18T22:56:29Z

I was just asking questions, don't block on my review!

# Conflicts: # airbyte-config-oss/init-oss/src/main/resources/seed/oss_registry.json

* Source SalesForce: remove pagination and query limits * Source SalesForce: remove pagination and query limits * Reduce slice interval to 30 days * Source SalesForce : remove apparent_encoding guess for response; Cause OOM for large responses * Source SalesForce : bump versions --------- Co-authored-by: artem1205 <artem1205@users.noreply.github.com>

Source SalesForce: remove pagination and query limits

15395fc

octavia-squidington-iii added area/connectors Connector related issues connectors/source/salesforce labels Apr 30, 2023

Source SalesForce: remove pagination and query limits

e5a5ac9

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Apr 30, 2023

artem1205 requested review from bnchrch, maxi297, lazebnyi and roman-yermilov-gl April 30, 2023 10:06

Reduce slice interval to 30 days

330cd2e

lazebnyi requested a review from erohmensing May 9, 2023 10:32

maxi297 reviewed May 9, 2023

View reviewed changes

airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py Show resolved Hide resolved

artem1205 added 3 commits May 18, 2023 11:09

Source SalesFoorce : remove apparent_encoding guess for response;

beea80c

Cause OOM for large responses

Merge branch 'master' into artem1205/source-salesforce-OC-1787

068ba28

# Conflicts: # docs/integrations/sources/salesforce.md

Source SalesFoorce : bump versions

b206a93

Automated Change

a3079cf

maxi297 approved these changes May 18, 2023

View reviewed changes

sherifnada reviewed May 18, 2023

View reviewed changes

artem1205 and others added 2 commits May 18, 2023 23:59

Merge branch 'master' into artem1205/source-salesforce-OC-1787

d25f4b6

# Conflicts: # airbyte-config-oss/init-oss/src/main/resources/seed/oss_registry.json

Automated Change

fb7cb5d

Merge branch 'master' into artem1205/source-salesforce-OC-1787

cdffa16

# Conflicts: # airbyte-config-oss/init-oss/src/main/resources/seed/oss_registry.json

artem1205 enabled auto-merge (squash) May 18, 2023 23:04

artem1205 and others added 4 commits May 18, 2023 23:22

Automated Change

b9cdfa6

Merge branch 'master' into artem1205/source-salesforce-OC-1787

3a8073a

# Conflicts: # airbyte-config-oss/init-oss/src/main/resources/seed/oss_registry.json

Automated Change

d91bb15

Empty-Commit

39f3efa

artem1205 merged commit 1c029bc into master May 19, 2023
22 of 26 checks passed

artem1205 deleted the artem1205/source-salesforce-OC-1787 branch May 19, 2023 10:13

marcosmarxm mentioned this pull request Jun 30, 2023

Source Salesforce: synch is missing a lot of records #27048

Closed

1 task

PierreKerschgens mentioned this pull request Jul 14, 2023

Source Salesforce: wrong encoding of sync data #27305

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source SalesForce: Remove pagination and query limits #25700

Source SalesForce: Remove pagination and query limits #25700

artem1205 commented Apr 30, 2023 •

edited

github-actions bot commented Apr 30, 2023 •

edited by artem1205

artem1205 commented Apr 30, 2023 •

edited by github-actions bot

artem1205 commented Apr 30, 2023

artem1205 commented Apr 30, 2023 •

edited by github-actions bot

maxi297 commented May 4, 2023

artem1205 commented May 5, 2023

artem1205 commented May 18, 2023 •

edited by github-actions bot

maxi297 left a comment

maxi297 left a comment

artem1205 commented May 18, 2023 •

edited by github-actions bot

sherifnada May 18, 2023

artem1205 May 18, 2023

sherifnada May 18, 2023

sherifnada May 18, 2023

artem1205 May 18, 2023

sherifnada May 18, 2023

sherifnada commented May 18, 2023

Source SalesForce: Remove pagination and query limits #25700

Source SalesForce: Remove pagination and query limits #25700

Conversation

artem1205 commented Apr 30, 2023 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

github-actions bot commented Apr 30, 2023 • edited by artem1205

Before Merging a Connector Pull Request

artem1205 commented Apr 30, 2023 • edited by github-actions bot

Build Passed

artem1205 commented Apr 30, 2023

artem1205 commented Apr 30, 2023 • edited by github-actions bot

maxi297 commented May 4, 2023

artem1205 commented May 5, 2023

artem1205 commented May 18, 2023 • edited by github-actions bot

maxi297 left a comment

Choose a reason for hiding this comment

maxi297 left a comment

Choose a reason for hiding this comment

artem1205 commented May 18, 2023 • edited by github-actions bot

sherifnada May 18, 2023

Choose a reason for hiding this comment

artem1205 May 18, 2023

Choose a reason for hiding this comment

sherifnada May 18, 2023

Choose a reason for hiding this comment

sherifnada May 18, 2023

Choose a reason for hiding this comment

artem1205 May 18, 2023

Choose a reason for hiding this comment

sherifnada May 18, 2023

Choose a reason for hiding this comment

sherifnada commented May 18, 2023

artem1205 commented Apr 30, 2023 •

edited

github-actions bot commented Apr 30, 2023 •

edited by artem1205

artem1205 commented Apr 30, 2023 •

edited by github-actions bot

artem1205 commented Apr 30, 2023 •

edited by github-actions bot

artem1205 commented May 18, 2023 •

edited by github-actions bot

artem1205 commented May 18, 2023 •

edited by github-actions bot