Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When we encounter a truncated CSV get a new stateful query session #54

Merged
merged 1 commit into from
Mar 31, 2021

Conversation

cosimon
Copy link
Contributor

@cosimon cosimon commented Mar 31, 2021

Description of change

When we encounter a truncated CSV get a new stateful query session.

The Zuora API can sometimes send us a non-rectangular CSV file (ie the number if items in a row is less than the number of columns). Testing has shown that when this happens in a stateful AQuA session then that session will from then on continuously return non-rectangular CSV.

This tap uses the version bookmark as part of the key for the AQuA query session. By generating a new version bookmark the tap ensures that on the next sync after encountering the non-rectangular CSV it will get a new query session.

Manual QA steps

  • Ran this on a connection that was receiving this non-rectangular exception. Running on this branch the tap encountered the exception once and fixed its state. On the subsequent run it was able to complete the sync.

Risks

Rollback steps

  • revert this branch

@cosimon cosimon merged commit b94e4f3 into master Mar 31, 2021
@cosimon cosimon deleted the fix/drop-query-session-on-truncated-csv branch March 31, 2021 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants