Source Zendesk Support: refactor and debug stuck syncs #20900

davydov-d · 2022-12-28T16:07:13Z

What

https://github.com/airbytehq/oncall/issues/1231

How

Reduce duplicate sleeps in a main thread
Reduce backoff time (for unknown reason it was doubled)
Added warning if a ProcessPoolExecutor is used instead of a ThreadPoolExecutor
Added more logs to understand what makes the sync stuck
Introduced ignore_pagination config variable to speed up SATs

…reduce backoff time

davydov-d · 2022-12-28T16:11:35Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3795199632
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3795199632
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_zendesk_support/__init__.py       2      0   100%
source_zendesk_support/streams.py      360     31    91%
source_zendesk_support/source.py        56      6    89%
--------------------------------------------------------
TOTAL                                  418     37    91%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
================== 38 passed, 3 skipped in 661.48s (0:11:01) ===================

…rt-sync-stucks

davydov-d · 2023-01-10T12:11:54Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3883200185

…rt-sync-stucks

davydov-d · 2023-01-16T18:54:16Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3933201233
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3933201233
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_zendesk_support/__init__.py       2      0   100%
source_zendesk_support/streams.py      380     37    90%
source_zendesk_support/source.py        56      6    89%
--------------------------------------------------------
TOTAL                                  438     43    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
================== 39 passed, 2 skipped in 390.89s (0:06:30) ===================

davydov-d · 2023-01-16T18:55:30Z

airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py

+        assert len(records) == records_count
+        end = datetime.datetime.now()
+        sleep_time = int(60 / x_rate_limit)
+        assert sleep_time - 1 <= (end - start).seconds <= sleep_time + 1


this test shows 6 seconds sleeping against 44 on a previous version

bazarnov

Please move the common method from the stream class to their base class.

davydov-d · 2023-01-17T15:47:06Z

Please move the common method from the stream class to their base class.

@bazarnov unfortunately I can not do it because the base class has it's own implementation and some of the descendants inherit it

davydov-d · 2023-01-17T15:47:43Z

/publish connector=connectors/source-zendesk-support

🕑 Publishing the following connectors:
connectors/source-zendesk-support
https://github.com/airbytehq/airbyte/actions/runs/3940910725

Connector	Did it publish?	Were definitions generated?
connectors/source-zendesk-support	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2023-01-17T16:26:32Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	26.59%	🍏

#1231 source zendesk support: remove synchronous sleep, add logging, …

33954d2

…reduce backoff time

octavia-squidington-iv added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/zendesk-support labels Dec 28, 2022

#1231 source zendesk support: upd changelog

3ca5bd6

davydov-d requested review from artem1205, bazarnov, darynaishchenko, grubberr, lazebnyi, roman-yermilov-gl and arsenlosenko December 28, 2022 16:11

bazarnov approved these changes Dec 29, 2022

View reviewed changes

lazebnyi approved these changes Jan 3, 2023

View reviewed changes

Merge branch 'master' into ddavydov/#1231-oncall-source-zendesk-suppo…

4020b73

…rt-sync-stucks

davydov-d added 3 commits January 13, 2023 14:10

Merge branch 'master' into ddavydov/#1231-oncall-source-zendesk-suppo…

9ea138f

…rt-sync-stucks

#1231 source zendesk: reduce sleeps

d2f9dc8

Merge branch 'master' into ddavydov/#1231-oncall-source-zendesk-suppo…

8293630

…rt-sync-stucks

davydov-d commented Jan 16, 2023

View reviewed changes

davydov-d requested review from lazebnyi, bazarnov and evantahler January 16, 2023 18:55

bazarnov approved these changes Jan 17, 2023

View reviewed changes

auto-bump connector version

93ab95b

octavia-squidington-iii temporarily deployed to more-secrets January 17, 2023 16:15 — with GitHub Actions Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 17, 2023 16:16 — with GitHub Actions Inactive

lazebnyi approved these changes Jan 17, 2023

View reviewed changes

davydov-d merged commit f31d92c into master Jan 17, 2023

davydov-d deleted the ddavydov/#1231-oncall-source-zendesk-support-sync-stucks branch January 17, 2023 16:54

marcosmarxm mentioned this pull request Jan 23, 2023

Source Zendesk Support: organizations sync hangs #20750

Closed

octavia-squidington-iii mentioned this pull request Jan 24, 2023

Bump Airbyte version from 0.40.28 to 0.40.29 #21767

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Zendesk Support: refactor and debug stuck syncs #20900

Source Zendesk Support: refactor and debug stuck syncs #20900

davydov-d commented Dec 28, 2022 •

edited

Loading

davydov-d commented Dec 28, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jan 10, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 16, 2023 •

edited by github-actions bot

Loading

davydov-d Jan 16, 2023

bazarnov left a comment

davydov-d commented Jan 17, 2023

davydov-d commented Jan 17, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 17, 2023

Source Zendesk Support: refactor and debug stuck syncs #20900

Source Zendesk Support: refactor and debug stuck syncs #20900

Conversation

davydov-d commented Dec 28, 2022 • edited Loading

What

How

davydov-d commented Dec 28, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Jan 10, 2023 • edited by github-actions bot Loading

davydov-d commented Jan 16, 2023 • edited by github-actions bot Loading

Build Passed

davydov-d Jan 16, 2023

Choose a reason for hiding this comment

bazarnov left a comment

Choose a reason for hiding this comment

davydov-d commented Jan 17, 2023

davydov-d commented Jan 17, 2023 • edited by github-actions bot Loading

github-actions bot commented Jan 17, 2023

Airbyte Code Coverage

davydov-d commented Dec 28, 2022 •

edited

Loading

davydov-d commented Dec 28, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jan 10, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 16, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 17, 2023 •

edited by github-actions bot

Loading