Source Zendesk Support: do not use nonpublic API #19432

davydov-d · 2022-11-15T15:09:20Z

What

https://github.com/airbytehq/oncall/issues/1010

How

Remove call to undocumented (nonpublic) API, use a test read instead

davydov-d · 2022-11-15T15:13:37Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3471695888
❌ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3471695888
🐛 https://gradle.com/s/6fjtj6auqpxr2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_secret_is_properly_marked[inputs0] - Fail...
=================== 1 failed, 40 passed in 624.52s (0:10:24) ===================

davydov-d · 2022-11-16T09:05:44Z

blocked by #19465

…nonpublic-api

davydov-d · 2022-11-16T12:19:32Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3479274787
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3479274787
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_zendesk_support/__init__.py       2      0   100%
source_zendesk_support/streams.py      333     34    90%
source_zendesk_support/source.py        54      6    89%
--------------------------------------------------------
TOTAL                                  389     40    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

All Passed

bazarnov · 2022-11-16T15:29:05Z

@davydov-d
Please note, with this change the connector will not be able to check the permissions for certain streams, based on the user's subscription plan. Thus, we should be ready to face with even more issues, rather than 422 errors.

airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/source.py

…nonpublic-api

davydov-d · 2022-11-21T09:45:52Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3513249044
❌ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3513249044
🐛 https://gradle.com/s/l4ncwdx2pyeu4

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/unit_test.py::test_full_access_streams[forms_inaccessible]
	 �[31m================= �[31m�[1m1 failed�[0m, �[32m155 passed�[0m, �[33m968 warnings�[0m�[31m in 3.57s�[0m�[31m ==================�[0m

davydov-d · 2022-11-21T09:46:55Z

@davydov-d Please note, with this change the connector will not be able to check the permissions for certain streams, based on the user's subscription plan. Thus, we should be ready to face with even more issues, rather than 422 errors.

yep, but we can not rely on the non public API cause it is not documented and provides no guarantees as well :(

davydov-d · 2022-11-21T10:32:24Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3513590549
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3513590549
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_zendesk_support/__init__.py       2      0   100%
source_zendesk_support/streams.py      333     34    90%
source_zendesk_support/source.py        56      6    89%
--------------------------------------------------------
TOTAL                                  391     40    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

All Passed

davydov-d · 2022-11-23T08:17:11Z

/test connector=connectors/source-zendesk-support

davydov-d · 2022-11-28T08:58:21Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3563207198
❌ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3563207198
🐛 https://gradle.com/s/soznbit5hedse

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream ticke...
=================== 1 failed, 40 passed in 612.59s (0:10:12) ===================

…nonpublic-api

davydov-d · 2022-11-29T09:36:36Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3572804946
❌ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3572804946
🐛 https://gradle.com/s/tvbf7zzlh622y

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream ticke...
=================== 1 failed, 40 passed in 601.78s (0:10:01) ===================

davydov-d · 2022-11-29T10:30:19Z

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3573214063
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3573214063
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_zendesk_support/__init__.py       2      0   100%
source_zendesk_support/streams.py      338     35    90%
source_zendesk_support/source.py        56      6    89%
--------------------------------------------------------
TOTAL                                  396     41    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

All Passed

alafanechere · 2022-11-29T14:46:27Z

airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/source.py

+                    streams.append(ticket_forms_stream)
+                    break
+        except Exception as e:
+            logger.warning(f"An exception occurred while trying to access TicketForms stream: {str(e)}. Skipping this stream.")


cc @erohmensing FYI, another connector implementing custom stream skipping logic.

alafanechere

Thanks for your effort in setting hightest strictness level without any bypass_reason 👏 .

alafanechere · 2022-11-29T14:46:52Z

airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/source.py

+        # TicketForms stream is only available for Enterprise Plan users but Zendesk API does not provide
+        # a public API to get user's subscription plan. That's why we try to read at least one record and expose this stream
+        # in case of success or skip it otherwise


alafanechere · 2022-11-29T14:48:48Z

airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/source.py

+            for stream_slice in ticket_forms_stream.stream_slices(sync_mode=SyncMode.full_refresh):
+                for _ in ticket_forms_stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice):
+                    streams.append(ticket_forms_stream)
+                    break


Would it be useful to store the inferred plan (Enterprise Plan) in the configuration?

Do you mean exposing a new field in the config.json?

I don't know exactly. Just a way for the connector to be aware of the inferred plan; wondering if this could be useful for other logic.

Well we could possibly store it but as for now we do not have a way to get a plan and be 100% sure it's correct

…nonpublic-api

davydov-d · 2022-11-29T17:27:40Z

/publish connector=connectors/source-zendesk-support

🕑 Publishing the following connectors:
connectors/source-zendesk-support
https://github.com/airbytehq/airbyte/actions/runs/3576516306

Connector	Did it publish?	Were definitions generated?
connectors/source-zendesk-support	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

#1010 oncall Source Zendesk Support: do not use nonpublic API

1954209

octavia-squidington-iv added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/zendesk-support labels Nov 15, 2022

#1010 source zendesk-support: upd changelog

868d373

davydov-d requested review from darynaishchenko, girarda, grubberr and pedroslopez November 15, 2022 15:13

davydov-d requested a review from bazarnov November 16, 2022 11:55

Merge branch 'master' into ddavydov/#1010-zendesk-support-do-not-use-…

ca5e0fb

…nonpublic-api

girarda approved these changes Nov 18, 2022

View reviewed changes

airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/source.py Outdated Show resolved Hide resolved

davydov-d added 2 commits November 21, 2022 11:23

Merge branch 'master' into ddavydov/#1010-zendesk-support-do-not-use-…

bbaccfe

…nonpublic-api

#1010 source zendesk-support: log an error during discover

bad769d

#1010 source zendesk support: fix unit test

7686eb1

#1010 source zendesk support: fix expected records

0bf558a

bazarnov approved these changes Nov 28, 2022

View reviewed changes

davydov-d added 2 commits November 29, 2022 11:35

Merge branch 'master' into ddavydov/#1010-zendesk-support-do-not-use-…

2bac3a7

…nonpublic-api

#1010 source zendesk support: upd expected records

321b00a

#1010 source zendesk support: another records update

5c35d0d

davydov-d requested a review from a team November 29, 2022 10:57

alafanechere reviewed Nov 29, 2022

View reviewed changes

alafanechere approved these changes Nov 29, 2022

View reviewed changes

Merge branch 'master' into ddavydov/#1010-zendesk-support-do-not-use-…

f24172e

…nonpublic-api

auto-bump connector version

3daf536

octavia-squidington-iii temporarily deployed to more-secrets November 29, 2022 17:57 Inactive

davydov-d merged commit 1ab02de into master Nov 29, 2022

davydov-d deleted the ddavydov/#1010-zendesk-support-do-not-use-nonpublic-api branch November 29, 2022 23:25

This was referenced Nov 29, 2022

Source Zendesk Support: enable high test strictness level in SAT #19071

Closed

Source Zendesk Support: enable high test strictness level in SAT #19184

Closed

erohmensing mentioned this pull request Nov 30, 2022

Remove custom code for checking 403s* to skip unavailable streams for a connector, identify other affected connectors #17853

Closed

octavia-squidington-iii mentioned this pull request Nov 30, 2022

Bump Airbyte version from 0.40.22 to 0.40.23 #19941

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Zendesk Support: do not use nonpublic API #19432

Source Zendesk Support: do not use nonpublic API #19432

davydov-d commented Nov 15, 2022

davydov-d commented Nov 15, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 16, 2022

davydov-d commented Nov 16, 2022 •

edited by github-actions bot

Loading

bazarnov commented Nov 16, 2022

davydov-d commented Nov 21, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 21, 2022

davydov-d commented Nov 21, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 23, 2022

davydov-d commented Nov 28, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

alafanechere Nov 29, 2022

alafanechere left a comment

alafanechere Nov 29, 2022

alafanechere Nov 29, 2022

davydov-d Nov 29, 2022

alafanechere Nov 29, 2022

davydov-d Nov 29, 2022

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

Source Zendesk Support: do not use nonpublic API #19432

Source Zendesk Support: do not use nonpublic API #19432

Conversation

davydov-d commented Nov 15, 2022

What

How

davydov-d commented Nov 15, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 16, 2022

davydov-d commented Nov 16, 2022 • edited by github-actions bot Loading

Build Passed

bazarnov commented Nov 16, 2022

davydov-d commented Nov 21, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 21, 2022

davydov-d commented Nov 21, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Nov 23, 2022

davydov-d commented Nov 28, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 29, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 29, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere Nov 29, 2022

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere Nov 29, 2022

Choose a reason for hiding this comment

alafanechere Nov 29, 2022

Choose a reason for hiding this comment

davydov-d Nov 29, 2022

Choose a reason for hiding this comment

alafanechere Nov 29, 2022

Choose a reason for hiding this comment

davydov-d Nov 29, 2022

Choose a reason for hiding this comment

davydov-d commented Nov 29, 2022 • edited by github-actions bot Loading

davydov-d commented Nov 15, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 16, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 21, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 21, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 28, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading