Source Salesforce: add more unsupported bulk entities, fix fallback to rest by davydov-d · Pull Request #19286 · airbytehq/airbyte

Denys Davydov (davydov-d) · 2022-11-10T14:02:00Z

What

https://github.com/airbytehq/oncall/issues/985

How

Extend list of unsupported entities by BULK API
Bugfix: fallback to REST API if BULK is not supported

…ack to rest

Denys Davydov (davydov-d) · 2022-11-10T14:03:28Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3437450092
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3437450092
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           311     75    76%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             94     31    67%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  596    139    77%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             94      4    96%
source_salesforce/api.py               151     14    91%
source_salesforce/streams.py           311     38    88%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
--------------------------------------------------------
TOTAL                                  596     60    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              353    110    69%   53, 64-72, 77-84, 88-89, 93-94, 192, 230-247, 256-264, 268-273, 279, 312-317, 355-362, 405-407, 410, 475-483, 495-498, 503, 559-560, 566, 569, 605-615, 628-653
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1516    348    77%

Build Passed

Test summary info:

All Passed

Augustin (alafanechere)

Thank you for the addition list.
For the rest fallback fix I think I'd need a bit more context to understand how it works and got fixed.

…api-fix

Denys Davydov (davydov-d) · 2022-11-15T07:58:00Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3468506784
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3468506784
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           311     69    78%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             94     31    67%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  596    133    78%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             94      4    96%
source_salesforce/api.py               151     14    91%
source_salesforce/streams.py           311     38    88%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
--------------------------------------------------------
TOTAL                                  596     60    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              394    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 293, 331-348, 357-365, 369-374, 380, 413-418, 456-463, 506-508, 511, 576-584, 596-599, 604, 660-661, 667, 670, 706-716, 729-754
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1557    349    78%

Build Passed

Test summary info:

All Passed

Denys Davydov (davydov-d) · 2022-11-15T10:27:08Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3469581244
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3469581244
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           311     69    78%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             94     31    67%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  596    133    78%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             94      4    96%
source_salesforce/api.py               151     14    91%
source_salesforce/streams.py           311     38    88%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
--------------------------------------------------------
TOTAL                                  596     60    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              394    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 293, 331-348, 357-365, 369-374, 380, 413-418, 456-463, 506-508, 511, 576-584, 596-599, 604, 660-661, 667, 670, 706-716, 729-754
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1557    349    78%

Build Passed

Test summary info:

All Passed

Denys Davydov (davydov-d) · 2022-11-15T10:49:05Z

/publish connector=connectors/source-salesforce

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/3469743433

Connector	Did it publish?	Were definitions generated?
connectors/source-salesforce	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Pierre Kerschgens (PierreKerschgens) · 2022-11-23T16:23:26Z

+    "CaseTeamMember",
+    "AttachedContentDocument",
+    "AggregateResult",
+    "AccountHistory",


Hi!
Since this connector version I'm having trouble getting data from AccountHistory table. The table has been added to 'UNSUPPORTED_BULK_API_SALESFORCE_OBJECTS' list.

Airbyte doesn't fail the sync job even though I'm not receiving data. I only found this in all of my logs:

2022-11-15 17:45:06 �[44msource�[0m > Syncing stream: AccountHistory 2022-11-15 17:47:07 �[44msource�[0m > [{"message":"Your query request was running for too long.","errorCode":"QUERY_TIMEOUT"}] 2022-11-15 17:47:07 �[44msource�[0m > Cannot receive data for stream 'AccountHistory', error message: 'Your query request was running for too long.' 2022-11-15 17:47:07 �[44msource�[0m > Read 0 records from AccountHistory stream 2022-11-15 17:47:07 �[44msource�[0m > Finished syncing AccountHistory

After downgrading the Salesforce connector from 1.0.26 to the prior version 1.0.25 I'm able to get data from AccountHistory again.

Some context to that error: #17503

hi Pierre Kerschgens (@PierreKerschgens) thanks for the feedback! I prepared a PR to fix this: #19869

…o rest (#19286) * #985 source salesforce: add more unsupported bulk entities, fix fallback to rest * #985 source salesforce: upd changelog * #985 source salesforce - add docstring for a test * #985 source Salesforce: bump version * #985 source salesforce: move SATs to high level strictness * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

#985 source salesforce: add more unsupported bulk entities, fix fallb…

83b30a6

…ack to rest

github-actions Bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 10, 2022

Octavia Squidington IV (octavia-squidington-iv) added the connectors/source/salesforce label Nov 10, 2022

#985 source salesforce: upd changelog

e2ef400

Denys Davydov (davydov-d) requested review from Augustin (alafanechere), Daryna Ishchenko (darynaishchenko), Alexandre Girard (girarda), Serhii Chvaliuk (grubberr) and Pedro S. Lopez (pedroslopez) and removed request for Augustin (alafanechere) November 10, 2022 14:03

Augustin (alafanechere) suggested changes Nov 10, 2022

View reviewed changes

Comment thread airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py

Comment thread airbyte-integrations/connectors/source-salesforce/unit_tests/api_test.py

Comment thread airbyte-integrations/connectors/source-salesforce/source_salesforce/api.py

#985 source salesforce - add docstring for a test

4395a9c

Denys Davydov (davydov-d) requested a review from Augustin (alafanechere) November 10, 2022 15:29

Alexandre Girard (girarda) approved these changes Nov 10, 2022

View reviewed changes

Merge branch 'master' into ddavydov/#985-salesforce-fallback-to-rest-…

cf8f06e

…api-fix

Augustin (alafanechere) approved these changes Nov 14, 2022

View reviewed changes

Denys Davydov (davydov-d) added 2 commits November 15, 2022 09:56

Merge branch 'master' into ddavydov/#985-salesforce-fallback-to-rest-…

870a901

…api-fix

#985 source Salesforce: bump version

ce26778

#985 source salesforce: move SATs to high level strictness

3daa7ef

auto-bump connector version

197a521

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets November 15, 2022 11:22 Inactive

Denys Davydov (davydov-d) merged commit 217a651 into master Nov 15, 2022

Denys Davydov (davydov-d) deleted the ddavydov/#985-salesforce-fallback-to-rest-api-fix branch November 15, 2022 11:40

github-actions Bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

Octavia Squidington III (octavia-squidington-iii) mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Pierre Kerschgens (PierreKerschgens) reviewed Nov 23, 2022

View reviewed changes

Augustin (alafanechere) mentioned this pull request Nov 24, 2022

Source Salesforce: "QUERY_TIMEOUT" #17503

Closed

This was referenced Jan 16, 2023

Source Salesforce: enable high test strictness level in SAT #19063

Closed

Source Salesforce: enable high test strictness level in SAT #19176

Closed

Navigation Menu

Repository navigation

Source Salesforce: add more unsupported bulk entities, fix fallback to rest - #19286

Source Salesforce: add more unsupported bulk entities, fix fallback to rest#19286
Denys Davydov (davydov-d) merged 8 commits into
masterfrom
ddavydov/#985-salesforce-fallback-to-rest-api-fix

Denys Davydov (davydov-d) commented Nov 10, 2022

Uh oh!

Denys Davydov (davydov-d) commented Nov 10, 2022 •

edited by github-actions Bot

Loading

Uh oh!

Augustin (alafanechere) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Denys Davydov (davydov-d) commented Nov 15, 2022 •

edited by github-actions Bot

Loading

Uh oh!

Denys Davydov (davydov-d) commented Nov 15, 2022 •

edited by github-actions Bot

Loading

Uh oh!

Denys Davydov (davydov-d) commented Nov 15, 2022 •

edited by github-actions Bot

Loading

Uh oh!

Pierre Kerschgens (PierreKerschgens) Nov 23, 2022 •

edited

Loading

Uh oh!

Denys Davydov (davydov-d) Nov 29, 2022

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

Denys Davydov (davydov-d) commented Nov 10, 2022

What

How

Uh oh!

Denys Davydov (davydov-d) commented Nov 10, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Augustin (alafanechere) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Denys Davydov (davydov-d) commented Nov 15, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Denys Davydov (davydov-d) commented Nov 15, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Denys Davydov (davydov-d) commented Nov 15, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pierre Kerschgens (PierreKerschgens) Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Denys Davydov (davydov-d) Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Denys Davydov (davydov-d) commented Nov 10, 2022 •

edited by github-actions Bot

Loading

Denys Davydov (davydov-d) commented Nov 15, 2022 •

edited by github-actions Bot

Loading

Denys Davydov (davydov-d) commented Nov 15, 2022 •

edited by github-actions Bot

Loading

Denys Davydov (davydov-d) commented Nov 15, 2022 •

edited by github-actions Bot

Loading

Pierre Kerschgens (PierreKerschgens) Nov 23, 2022 •

edited

Loading