Source Zendesk Support: organizations sync hangs

[validumitru]

Environment

• Airbyte version: 0.40.18
• Source Connector and version: 0.2.19
• Step where error happened: Sync job

Current Behavior

After the changes in #19967 the sync hangs. The issue is being discussed in the comments of #14253

[marcosmarxm]

I added the issue to oncall issues: https://github.com/airbytehq/oncall/issues/1231

[emilmar]

Thanks for following up on this. Here are logs from me of a multi-hour hanging for just 114 records.

b68c83a1_ed6f_4997_8794_b499009297de_logs_278_txt.txt

[marcosmarxm]

@validumitru and @emilmar can you check the latest version (0.2.20) of the connector? The PR #20900 try to solve the issue

[emilmar]

@marcosmarxm It appears to be working! A bit spammy in regards to all the threads (I'm at over 10k log lines just for retries) but currently 1 hour in and at 160k records. Not so sure about how it will handle the incremental part though, I'll update you when it's done.

2023-01-24 09:12:56 source > Sleeping 14 seconds before next request
2023-01-24 09:12:56 source > Will retry the request for other reason
2023-01-24 09:12:56 source > Adding a request to be retried in 4 seconds
2023-01-24 09:12:56 source > Starting another while loop iteration
2023-01-24 09:12:56 source > Sleeping 14 seconds before next request
2023-01-24 09:12:56 source > Will retry the request for other reason
2023-01-24 09:12:56 source > Adding a request to be retried in 4 seconds
2023-01-24 09:12:56 source > Starting another while loop iteration
2023-01-24 09:12:56 source > Will retry the request for other reason
2023-01-24 09:12:56 source > Adding a request to be retried in 16 seconds
2023-01-24 09:12:56 source > Starting another while loop iteration
2023-01-24 09:12:56 source > Will retry the request for other reason
2023-01-24 09:12:56 source > Adding a request to be retried in 16 seconds
2023-01-24 09:12:56 source > Starting another while loop iteration

[emilmar]

@davydov-d Your PR almost seemed to solve it! But I ended up with an error again

airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://tripletexhelp.zendesk.com/api/v2/organizations?start_time=1577836800&page=1119, Response Code: 429, Response Text: Pagination limit reached for offset model

Adding the full logs here:
b68c83a1_ed6f_4997_8794_b499009297de_logs_1507_txt.txt

Second attempt succeeded, but it only read 100k records. Personally I feel like Im going into the sunk cost fallacy here, for us it's not a killer feature to make this work. Thanks for trying out new things 🤗

[davydov-d]

hi @emilmar thanks for the feedback! it looks like the connector uses the previous pagination model which is offset-based, meanwhile Zendesk introduced a new pagination model, the cursor-based pagination. To encourage migration to the newest pagination model Zendesk restricts the pagination to 1000 pages with the offset-based model.
Since this is a different problem from the one originally reported, I'd like to ask you to close this issue and open a new one, thanks!

[emilmar]

Alright, thanks for the input! Not my issue but OP or admin can probably go ahead and close it. @marcosmarxm @validumitru

[validumitru]

I can confirm it's not hanging anymore but the sync failed now with a new exception not sure if this is a coincidence ? I'm attaching the logs
ee85716a_cad6_45bb_954b_34c8500e95b5_logs_3466_txt.txt

Happy to close this if it's a separate issue?

[marcosmarxm]

@validumitru can you open a new github issue?