Skip to content

Improve synchronisation docs#451

Merged
leplatrem merged 8 commits intomasterfrom
improve-synchronization-docs
Feb 29, 2016
Merged

Improve synchronisation docs#451
leplatrem merged 8 commits intomasterfrom
improve-synchronization-docs

Conversation

@leplatrem
Copy link
Contributor

We will use ``sort=last_modified`` and ``_since=<timestamp>``:

#. First sync: ``timestamp := 0``
#. Next sync: ``timestamp := MAX(local_records['last_modified'])``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timestamp := MAX(local_records['last_modified'])-1 because it may be that two or more changes happened during the same timestamp, and you did not get all of them during the first sync.

This is especially relevant if you combine this with _limit. For instance, in FxSync you sometimes see 1000 records with the exact same timestamp, so you shouldn't discard 950 of them after fetching only the first 50.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because it may be that two or more changes happened during the same timestamp

This is not possible on server side. Even if two records happens at the same timestamp, they will have different timestamps.

See http://kinto.readthedocs.org/en/latest/api/1.x/cliquet/timestamps.html

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in FxSync you sometimes see 1000 records with the exact same timestamp, so you shouldn't discard 950 of them after fetching only the first 50.

This is a real problem for syncto because Kinto guarantees that it won't happend. If Sync doesn't it means that we have a problem in syncto.

@n1k0
Copy link
Contributor

n1k0 commented Feb 26, 2016

Very clear and insightful, answers a bunch of questions I had; r+.

@leplatrem
Copy link
Contributor Author

why would you do it this way instead of just keep going in reverse chronological order?

Yeah, I changed it. In the third approach, they both go backwards. One fetches the old records with _before=MIN(...) and the other one fetches the new records with _since=MAX()


#. Obtain a few pages of recent records using the *newest first* strategy from above
#. In background, fetch old records using ``_sort=-lastmodified`` and ``_before=MIN(local_records[last_modified])``
#. Recent changes can be obtained using ``_sort=-lastmodified`` and ``_since=MAX(local_records[last_modified])``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think you'll skip records with this strategy unless you can be sure that each record has a unique lastmodified timestamp. As soon as you have 1 local record with a given timestamp, all other remote records with that same timestamp will be ignored, right?

leplatrem added a commit that referenced this pull request Feb 29, 2016
@leplatrem leplatrem merged commit ebbf01e into master Feb 29, 2016
@leplatrem leplatrem deleted the improve-synchronization-docs branch February 29, 2016 16:30
@leplatrem leplatrem modified the milestone: 2.0 Mar 10, 2016
lavish205 pushed a commit to lavish205/kinto that referenced this pull request Jun 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants