Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix iterator page time overlap #2

Open
judgej opened this issue Aug 7, 2019 · 1 comment
Open

Fix iterator page time overlap #2

judgej opened this issue Aug 7, 2019 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@judgej
Copy link
Member

judgej commented Aug 7, 2019

The iterator fetches pages with overlapping time by one second, with the assumption that only a couple of transactions can be located within that second and so there will only be a few duplicates to discard.

The problem is that creating batch payments will result in all payments in that batch being created pretty much at the same time, with many dozens (at least) being created in the same millisecond. This can cause the same page of payments to be fetched over and over, and discarded as duplicates in its entirety each time.

To fix this, fetching a second and subsequent page much start at exactly the same time as the last record of the previous page, not at a slightly rewound timestamp. If the next page is discarded in its entirety as duplicates, then the time should be wound forward by 1mS and tried again. There is a risk that some payments in the previous page may be missed. However, a payments page seems to be quite large - well over 1000 payments - so that risk is small for now.

Ideally the "wind on" time will be 1uS, but I don't believe the search conditions goes down below mS resolution at this time.

@judgej judgej added the bug Something isn't working label Aug 7, 2019
@judgej judgej self-assigned this Aug 7, 2019
@judgej
Copy link
Member Author

judgej commented Aug 7, 2019

The "last updated since" resolution when fetching payments seems to be whole seconds. Checking the Xero API docs, it says "accurate to a second", so we need to extend the timestamp by a second to get the next page if we get a page of duplicates.

That's not good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant