Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filtered replication doesn't record progress unless change is returned #5145

Closed
garethbowen opened this issue May 10, 2016 · 1 comment
Closed

Comments

@garethbowen
Copy link
Member

I use filtered replication to reduce the number of records replicated to Pouch. When a change passes the filter function the doc is replicated to Pouch and the local seq is updated which future replications use as the since parameter so already filtered changes can be skipped over.

However when no changes are returned the seq is not updated so subsequent replications use the old value for since. This means each doc can be run through the filter function multiple times until a change is made that passes the filter.

For example if the couchdb seq is 2000 and the local seq is 1000 the replication request will look something like...

.../_changes?style=all_docs&heartbeat=10000&filter=medic%2Fdoc_by_place&id=7E7633CB-C6CD-D85D-9ABD-1C6B7570F000&since=1000&limit=100

Then the server runs all the changes from 1000 to 2000 through the filter and responds with...

{
  "results":[],
  "last_seq":2000
}

pouchdb then ignores the response as there are no results to update.

Modify pouchdb to store the last_seq from the changes response in the replication checkpoint so replication starts from the new seq next time.

garethbowen added a commit to garethbowen/pouchdb that referenced this issue May 10, 2016
garethbowen added a commit to garethbowen/pouchdb that referenced this issue May 10, 2016
garethbowen added a commit to garethbowen/pouchdb that referenced this issue May 12, 2016
daleharvey pushed a commit that referenced this issue May 13, 2016
* (#5145) - Update checkpoint even when no changes
* (#5145) - Fix eslint
* (#5146) - Test requested parameter, couchdb 2 compatibility
* (#5145) - Tidy up test to be more promisey
@nolanlawson
Copy link
Member

fixed by f1989ce

willholley added a commit that referenced this issue Jun 3, 2016
An attempt to provide a better fix for #5145. It seems like
it's safe to eagerly write a checkpoint in the event there are no
changes returned (e.g. because a filter did not generate results).

This at least allows the tests to pass - possibly this means we
need more tests.
willholley added a commit that referenced this issue Jun 3, 2016
An attempt to provide a better fix for #5145. It seems like
it's safe to eagerly write a checkpoint in the event there are no
changes returned (e.g. because a filter did not generate results).

This at least allows the tests to pass - possibly this means we
need more tests.
willholley added a commit that referenced this issue Jun 3, 2016
An attempt to provide a better fix for #5145. It seems like
it's safe to eagerly write a checkpoint in the event there are no
changes returned (e.g. because a filter did not generate results).

This also removes some assertions around last_seq / since values
in the tests, since they don't hold in CouchDB 2.x / Cloudant.

This at least allows the tests to pass - possibly this means we
need more tests to cover situations where a small number of docs
are returned.
willholley added a commit that referenced this issue Jun 3, 2016
An attempt to provide a better fix for #5145. It seems like
it's safe to eagerly write a checkpoint in the event there are no
changes returned (e.g. because a filter did not generate results).

This also removes some assertions around last_seq / since values
in the tests, since they don't hold in CouchDB 2.x / Cloudant.

This at least allows the tests to pass - possibly this means we
need more tests to cover situations where a small number of docs
are returned.
daleharvey pushed a commit that referenced this issue Jun 3, 2016
An attempt to provide a better fix for #5145. It seems like
it's safe to eagerly write a checkpoint in the event there are no
changes returned (e.g. because a filter did not generate results).

This also removes some assertions around last_seq / since values
in the tests, since they don't hold in CouchDB 2.x / Cloudant.

This at least allows the tests to pass - possibly this means we
need more tests to cover situations where a small number of docs
are returned.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants