_replicate calls are extremely slow

[tudordumitriu]

We have noticed that even if there is almost no load on CouchDB the _replicate call takes from 1 second to 5 seconds on the development machine and on the server it reaches even 15 seconds, which compared to standard execution time for a standard call (GET, PUT doc which is around 0.01 seconds) seems to be extremely slow.

We are using _replicate calls with doc_ids (varying from one to few items up to hundreds/thousands) to replicate various changes to user db's (we are using per user database).

This behavior is reproducible on development machine (CouchDB 2.3.1) and as well on staging server (CouchDB 2.1.1)

Steps to Reproduce (for bugs)

1. Standard replication call
   URL: "http://sa:replacedPassword@localhost:5984/_replicate"
   BODY: "{"source": "http://sa:replacedPassword@localhost:5984/userdb-6465763830","target": "http://sa:replacedPassword@localhost:5984/userdb-78" , "doc_ids" : ["l_11b7d2bd75d444d4947fa4cecf985767_org.couchdb.user:x"] }"
2. Average execution time is around 3s (on a warmed up instance)
3. I have attached for the above call the couchdb.log (log level = debug)
   couch.log

Context

Our db architecture:

• Client PouchDB
• Server CouchDB - Per user db
• Server CouchDB - Central DB (only a subset of each PerUser db gets replicated to the CentralDB using https://github.com/redgeoff/spiegel)
• Shared Documents to multiple PerUser Databases (that have to go through advanced sharing validation) go to our API (https://github.com/redgeoff/spiegel listens to global_changes and calls our API) which calls the problematic _replicate
• API is .Net web.api using https://github.com/danielwertheim/mycouch as a CouchDB client
• At some point the server becomes unusable.Maybe not very useful but when we are replicating multiple documents the duration increases dramatically.serverReplicateCallsDuration.txt
• This is perfectly reproducible using PostMan or any other tool

Your Environment

• Version used: 2.1.1 to 2.3.1
• Operating System and version (desktop or mobile): Windows 10 / Windows 2016 Server

[rnewson]

_replicate has to read all changes from the source and apply them to the target, which can take some time. As you're using a doc id filter, this almost certainly means the entire source database is being scanned, which is quite inefficient. It's far easier to apply a single doc update change to both servers. Replication is appropriate when replicating all or the majority of the source documents.

[nickva]

What @rnewson said. Also another reason could be a random amount of sleep used by replication jobs when they start up. By default it is between 0 and 5000 milliseconds. You can adjust it via the [replicator] startup_jitter = 5000 config setting. Try setting it to 0 and see if you still see the issue.

The reason for the jitter is to avoid an initial load spike a node with a larger number of replication jobs is restarted.

[tudordumitriu]

Thanks for the quick replies.

Indeed setting the startup_jitter = 0 fixed a big part of the problem, but there are still moments when it takes 7-10 seconds. So as @rnewson said it might be because replicate is scanning the source db (even thought I would have thought that by applying the doc_ids filter we would avoid that, because it has already the ids, but anyways).
So, @rnewson, are you suggesting that instead of using Replicate we should be applying a single doc change to both dbs, but I'm not exactly sure what you mean.
Because, as far as I know, the update doc mechanism works by retrieving the original document form the db, apply the updates, and then put it back to the db (which generates a new _rev, different from the one from the source db).
So I'm afraid that by doing this we'll end up with 2 branches (different _rev numbers) for the same document (which we do not want of course).
The _replicate call is responsible of moving (re-conciliating?) the document (tree) entirely, and also (opposite to interacting with _replicator db, is not storing any documents for every single replication) so it was the perfect choice for us because it is at the core of our document sharing mechanism between different user dbs.
If you're suggesting that there is a better performing option (which accomplishes the above as well) I'll be forever grateful if you'd point me in the right direction.

[rnewson]

replication is reading the _changes feed, which is ordered (roughly) by last update time, the filter is then applied to the changes as they come through. There might be some internal optimization for the doc_ids filter but given the numbers you're showing, it might not always be possible (or it just might not be that big an optimization).

If you apply the same changes to a document in two separate couchdb servers, you will get the same _rev value and therefore no conflict when you replicate. This is by design. You obviously have to apply the exact same changes in the exact same order to benefit from this, which you might struggle with in practice.

[tudordumitriu]

Thanks
We have tried it and if initially it seemed to be working (applying same updates in fauxton) in practice we are getting different rev numbers, but indeed, even if it would work ,in practice, in our architecture, is quite hard to benefit from it.
I am still interested about this potential improvement, and maybe at some point we'll give it another try.
Eventually please tell me why wouldn't it generate same rev even if we are applying same json (does formatting count?). Because I have compared the docs in various file comparers and the only difference is the updated field, but is still having a different rev no.

[rnewson]

meaningless whitespace shouldn't count (that is, outside of JSON string values), but it seems you're saying the 'updated' field is different, and that means you didn't make the same change to the document, which is why the revs are different.

[rnewson]

property ordering would count, though. (within objects and arrays)

[tudordumitriu]

We are doing something like this:
When the update is notified in global changes, we take it, the new version of the document, change with the _rev no from the target database, and put the update.
But I think there must be something wrong we are doing, and even though I appreciate very much your help, continuing this discussion is not in the spirit of the bug.
So, I'll just close it for now.
Cheers!