Replication - when doc_ids is defined the changes feed is dependent on batch_size

[jdelibas]

If a couchDB database has more than 100 documents and the specified ids in doc_ids exists beyond those first 100 then the documents specified will not replicate since the changes feed with only check up until the amount specified by the batch_size which defaults to 100.

So assuming couchDB database with
500 documents
and IDs as
001,002,003,....498.499,500

var db = new PouchDB('objects');
db.sync('http://couchdb:5984', {doc_ids : ['101','201','301']})
      .on('change', function (info) {
        console.log(info);
      }).on('uptodate', function (info) {
        console.log(info);
      }).on('complete', function (info) {  
        console.log(info);
      }).on('error', function (err) {
        console.log(err);
      });

This will not replicate any documents.

Unless batch_size is defined as the total size of the database it will only check the first 100.

db.sync('http://couchdb:5984', {batch_size : 500, doc_ids : ['101','201','301']})

This will replicate all three documents with the IDs '101','201','301'.

I think this is a fairly specific use case and this might be intentional because of that.

[calvinmetcalf]

It's a bug, we look for a batch with nothing in it to know when replication
is done, we likely need a special optimization for doc_ids
On Jul 1, 2014 4:53 AM, "jdelibas" notifications@github.com wrote:

If a couchDB database has more than 100 documents and the specified ids in
doc_ids exists beyond those first 100 then the documents specified will not
replicate since the changes feed with only check up until the amount
specified by the batch_size which defaults to 100.

So assuming couchDB database with
500 documents
and IDs as
001,002,003,....498.499,500

var db = new PouchDB('objects');db.sync('http://couchdb:5984', {doc_ids : ['101','201','301']})
.on('change', function (info) {
console.log(info);
}).on('uptodate', function (info) {
console.log(info);
}).on('complete', function (info) {
console.log(info);
}).on('error', function (err) {
console.log(err);
});

This will not replicate any documents.

Unless batch_size is defined as the total size of the database it will
only check the first 100.

db.sync('http://couchdb:5984', {batch_size : 500, doc_ids : ['101','201','301']})

This will replicate all three documents with the IDs '101','201','301'.

I think this is a fairly specific use case and this might be intentional
because of that.

—
Reply to this email directly or view it on GitHub
#2426.

[wendall911]

doc_ids are not passed to the changes feed used in the replicator. This generates unnecessary overhead, since every change in the db is passed through the changes feed used by the replicator.

[nolanlawson]

@wendall911 Are you sure?

pouchdb/lib/replicate.js

Line 472 in a8c8624

doc_ids: doc_ids,

[wendall911]

@nolanlawson Yes, I'm sure. The underlying _changes does not make use of doc_ids, so this is only a client-side filter at this point.

[daleharvey]

So I figured this behaviour has likely changed over 2 years, it looks to be working now and added a test to verify @ #5254

[daleharvey]

07a7bca