-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛Source Zendesk Support: fix pagination for incremetal export streams #33680
Merged
roman-yermilov-gl
merged 3 commits into
master
from
ryermilov/source-zendesk-support-fix-pagination
Jan 8, 2024
Merged
🐛Source Zendesk Support: fix pagination for incremetal export streams #33680
roman-yermilov-gl
merged 3 commits into
master
from
ryermilov/source-zendesk-support-fix-pagination
Jan 8, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
octavia-squidington-iii
added
area/connectors
Connector related issues
connectors/source/zendesk-support
labels
Dec 20, 2023
Contributor
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
octavia-squidington-iii
added
the
area/documentation
Improvements or additions to documentation
label
Dec 20, 2023
roman-yermilov-gl
changed the title
Source Zendesk Support: fix pagination for incremetal export streams
🐛Source Zendesk Support: fix pagination for incremetal export streams
Dec 20, 2023
artem1205
approved these changes
Jan 8, 2024
roman-yermilov-gl
deleted the
ryermilov/source-zendesk-support-fix-pagination
branch
January 8, 2024 14:22
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 26, 2024
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 26, 2024
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area/connectors
Connector related issues
area/documentation
Improvements or additions to documentation
connectors/source/zendesk-support
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Oncall issue: https://github.com/airbytehq/oncall/issues/3706
There are multiple problems with pagination which affect descendants of the SourceZendeskIncrementalExportStream especially Organizations and Articles.
Organizations`s pagination mechanizm is based on
end_of_stream
flag andstart_date/end_date
params. For this streamstart_date
were never replaced byend_date
(becausenext_page_token
updates onlycursor
) so we infinitely requesting 1st pageArticles use
start_date/end_date
params but there is noend_of_stream
indiactor in response. Instead of this we can understand when pagination comes to the end by emptynext_page
field. We did not take into account absence of theend_of_stream
indicator for this particular stream and always stopped reading at 1st pageOther descendants (Users, Tickets, TicketComments) use
start_date
as an initial param and then rely on next pagecursor
value provided by API. These streams were not affected because of design features which is a random luckHow
Upgraded the machanism of how we prepare request for getting next page. The best way to do this is to use url from
next_page
field which comes in response. The url contains all required properties provided in request as well as updated next page token. Using this we can support all the types of pagination in one place and avoid problems listed before at the same time