New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persist temporary revision body backups on import #3740

Closed
adamcfraser opened this Issue Sep 12, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@adamcfraser
Copy link
Contributor

adamcfraser commented Sep 12, 2018

On a normal Sync Gateway write, SG makes a temporary backup of the previous revision body, as a convenience for in-flight replications. The temporary backup is particularly useful for documents being updated in quick succession, as connected clients have a smaller window to replicate the previous revision before it's updated. The temporary backup is primarily a benefit to SG nodes other than the one that performed the write, as they are less likely to have the revision resident in their rev cache.

In the case of shared bucket access (import), Sync Gateway doesn't currently make a backup of the previous revision, as the body has been overwritten by the non-SG write. However, it's possible that the revision may still be resident in the rev cache of the node performing the import. This is particularly likely for docs recently updated, as the previous import will have loaded the revision into the rev cache.

Import processing could be enhanced to persist a temporary copy of the previous revision body when they are available in the rev cache. To avoid performance overhead during initial import we could consider disabling these backups during import backfill.

@adamcfraser adamcfraser added this to the Iridium milestone Sep 12, 2018

@adamcfraser

This comment has been minimized.

Copy link
Contributor

adamcfraser commented Sep 12, 2018

This is particularly relevant for documents that are continuously being frequently updated via the SDK. Without some type of backup, the window in which each revision is available to be synchronized via Sync Gateway shrinks as the update frequency increases. This could be avoided by making the same writes through SG's REST API, but it would be useful to have the same temporary backup capability for shared bucket access/import.

@adamcfraser adamcfraser changed the title Temporary revision body backups on import Persist temporary revision body backups on import Sep 12, 2018

@adamcfraser adamcfraser self-assigned this Sep 17, 2018

@adamcfraser adamcfraser modified the milestones: Iridium, 2.1.1 Sep 17, 2018

@adamcfraser adamcfraser added ready and removed backlog labels Sep 24, 2018

@adamcfraser

This comment has been minimized.

Copy link
Contributor

adamcfraser commented Sep 29, 2018

This change will have a non-negligible impact on import/DCP throughput in scenarios with high volumes of imported SDK writes - Sync Gateway will be performing two kv writes instead of one per import.

To avoid performance degradation in use cases that don't need this functionality (i.e. aren't rapidly updating the same doc via the SDK), this should be enabled via an opt-in flag.

@bbrks

This comment has been minimized.

Copy link
Member

bbrks commented Oct 11, 2018

Implemented in #3775

@adamcfraser

This comment has been minimized.

Copy link
Contributor

adamcfraser commented Oct 25, 2018

Merged to master w/ #3806

@adamcfraser adamcfraser removed the ready label Oct 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment