Source Salesforce: add more unsupported bulk entities, fix fallback to rest #19286

davydov-d · 2022-11-10T14:02:00Z

What

https://github.com/airbytehq/oncall/issues/985

How

Extend list of unsupported entities by BULK API
Bugfix: fallback to REST API if BULK is not supported

…ack to rest

davydov-d · 2022-11-10T14:03:28Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3437450092
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3437450092
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           311     75    76%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             94     31    67%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  596    139    77%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             94      4    96%
source_salesforce/api.py               151     14    91%
source_salesforce/streams.py           311     38    88%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
--------------------------------------------------------
TOTAL                                  596     60    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              353    110    69%   53, 64-72, 77-84, 88-89, 93-94, 192, 230-247, 256-264, 268-273, 279, 312-317, 355-362, 405-407, 410, 475-483, 495-498, 503, 559-560, 566, 569, 605-615, 628-653
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1516    348    77%

Build Passed

Test summary info:

All Passed

alafanechere

Thank you for the addition list.
For the rest fallback fix I think I'd need a bit more context to understand how it works and got fixed.

airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py

airbyte-integrations/connectors/source-salesforce/unit_tests/api_test.py

airbyte-integrations/connectors/source-salesforce/source_salesforce/api.py

…api-fix

davydov-d · 2022-11-15T07:58:00Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3468506784
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3468506784
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           311     69    78%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             94     31    67%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  596    133    78%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             94      4    96%
source_salesforce/api.py               151     14    91%
source_salesforce/streams.py           311     38    88%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
--------------------------------------------------------
TOTAL                                  596     60    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              394    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 293, 331-348, 357-365, 369-374, 380, 413-418, 456-463, 506-508, 511, 576-584, 596-599, 604, 660-661, 667, 670, 706-716, 729-754
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1557    349    78%

Build Passed

Test summary info:

All Passed

davydov-d · 2022-11-15T10:27:08Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3469581244
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3469581244
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           311     69    78%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             94     31    67%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  596    133    78%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             94      4    96%
source_salesforce/api.py               151     14    91%
source_salesforce/streams.py           311     38    88%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
--------------------------------------------------------
TOTAL                                  596     60    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              394    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 293, 331-348, 357-365, 369-374, 380, 413-418, 456-463, 506-508, 511, 576-584, 596-599, 604, 660-661, 667, 670, 706-716, 729-754
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1557    349    78%

Build Passed

Test summary info:

All Passed

davydov-d · 2022-11-15T10:49:05Z

/publish connector=connectors/source-salesforce

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/3469743433

Connector	Did it publish?	Were definitions generated?
connectors/source-salesforce	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

PierreKerschgens · 2022-11-23T16:23:26Z

airbyte-integrations/connectors/source-salesforce/source_salesforce/api.py

+    "CaseTeamMember",
+    "AttachedContentDocument",
+    "AggregateResult",
+    "AccountHistory",


Hi!
Since this connector version I'm having trouble getting data from AccountHistory table. The table has been added to 'UNSUPPORTED_BULK_API_SALESFORCE_OBJECTS' list.

Airbyte doesn't fail the sync job even though I'm not receiving data. I only found this in all of my logs:

2022-11-15 17:45:06 �[44msource�[0m > Syncing stream: AccountHistory 2022-11-15 17:47:07 �[44msource�[0m > [{"message":"Your query request was running for too long.","errorCode":"QUERY_TIMEOUT"}] 2022-11-15 17:47:07 �[44msource�[0m > Cannot receive data for stream 'AccountHistory', error message: 'Your query request was running for too long.' 2022-11-15 17:47:07 �[44msource�[0m > Read 0 records from AccountHistory stream 2022-11-15 17:47:07 �[44msource�[0m > Finished syncing AccountHistory

After downgrading the Salesforce connector from 1.0.26 to the prior version 1.0.25 I'm able to get data from AccountHistory again.

Some context to that error: #17503

hi @PierreKerschgens thanks for the feedback! I prepared a PR to fix this: #19869

…o rest (#19286) * #985 source salesforce: add more unsupported bulk entities, fix fallback to rest * #985 source salesforce: upd changelog * #985 source salesforce - add docstring for a test * #985 source Salesforce: bump version * #985 source salesforce: move SATs to high level strictness * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

#985 source salesforce: add more unsupported bulk entities, fix fallb…

83b30a6

…ack to rest

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 10, 2022

octavia-squidington-iv added the connectors/source/salesforce label Nov 10, 2022

#985 source salesforce: upd changelog

e2ef400

davydov-d requested review from alafanechere, darynaishchenko, girarda, grubberr and pedroslopez and removed request for alafanechere November 10, 2022 14:03

alafanechere requested changes Nov 10, 2022

View reviewed changes

#985 source salesforce - add docstring for a test

4395a9c

davydov-d requested a review from alafanechere November 10, 2022 15:29

girarda approved these changes Nov 10, 2022

View reviewed changes

Merge branch 'master' into ddavydov/#985-salesforce-fallback-to-rest-…

cf8f06e

…api-fix

alafanechere approved these changes Nov 14, 2022

View reviewed changes

davydov-d added 2 commits November 15, 2022 09:56

Merge branch 'master' into ddavydov/#985-salesforce-fallback-to-rest-…

870a901

…api-fix

#985 source Salesforce: bump version

ce26778

#985 source salesforce: move SATs to high level strictness

3daa7ef

auto-bump connector version

197a521

octavia-squidington-iii temporarily deployed to more-secrets November 15, 2022 11:22 Inactive

davydov-d merged commit 217a651 into master Nov 15, 2022

davydov-d deleted the ddavydov/#985-salesforce-fallback-to-rest-api-fix branch November 15, 2022 11:40

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

PierreKerschgens reviewed Nov 23, 2022

View reviewed changes

alafanechere mentioned this pull request Nov 24, 2022

Source Salesforce: "QUERY_TIMEOUT" #17503

Closed

This was referenced Jan 16, 2023

Source Salesforce: enable high test strictness level in SAT #19063

Closed

Source Salesforce: enable high test strictness level in SAT #19176

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Salesforce: add more unsupported bulk entities, fix fallback to rest #19286

Source Salesforce: add more unsupported bulk entities, fix fallback to rest #19286

davydov-d commented Nov 10, 2022

davydov-d commented Nov 10, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

davydov-d commented Nov 15, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 15, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 15, 2022 •

edited by github-actions bot

Loading

PierreKerschgens Nov 23, 2022 •

edited

Loading

davydov-d Nov 29, 2022

Source Salesforce: add more unsupported bulk entities, fix fallback to rest #19286

Source Salesforce: add more unsupported bulk entities, fix fallback to rest #19286

Conversation

davydov-d commented Nov 10, 2022

What

How

davydov-d commented Nov 10, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere left a comment

Choose a reason for hiding this comment

davydov-d commented Nov 15, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Nov 15, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Nov 15, 2022 • edited by github-actions bot Loading

PierreKerschgens Nov 23, 2022 • edited Loading

Choose a reason for hiding this comment

davydov-d Nov 29, 2022

Choose a reason for hiding this comment

davydov-d commented Nov 10, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 15, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 15, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 15, 2022 •

edited by github-actions bot

Loading

PierreKerschgens Nov 23, 2022 •

edited

Loading