One doc replication issue in 3.1.0

[greenais]

Description

after upgrade to 3.1.0 from some point Couchdb has stopped replicate one document from local DB to another local DB.

In 2.2 following code worked just fine, with no issues:

const commonOptions = {
      protocol: 'http:',
      hostname: CONFIG.CDB.HOSTNAME,
      port: CONFIG.CDB.PORT,
      auth: CONFIG.CDB.ADMIN + ':' + CONFIG.CDB.ADMINPASS,
      headers: {
        'Content-Type': 'application/json',
        }
    };

    const postOptions = Object.assign({}, commonOptions);
    postOptions.method = 'POST';
    postOptions.timeout = CONFIG.CDB.PUTTIMEOUT;

    CDB.replicateDoc = (sourceDB, targetDB, docID)=> {
      return new Promise((resolve, reject)=> {
        let payload = postOptions;
        payload.path = '/_replicate';
        let doc = {};
        doc.source = sourceDB;
        doc.target = targetDB;
        doc.doc_ids = [docID];
        let req = http.request(payload, (res) => {
          let data = '';
          res.setEncoding('utf8');
          res.on('data', (chunk)=> {data += chunk})
          res.on('end', ()=> {
            if (res.statusCode === 200) {
              return resolve('replicateDoc_done')
            } else {
              return reject('CDB.replicateDoc status' + res.statusCode + ': ' + data)
            }
          })
        });
        req.on('error', (err)=> {return reject('CDB.replicateDoc ERR: ' + err)})
        req.write(JSON.stringify(doc));
        req.end();
      })
    };

Now I've got no errors, status 200 (in CDB logs as well), but in fact replicated doc doesn't appear in target DB.
Looks like it has started to happen since doc with same _id was previously deleted form target DB.
I'm out of ideas what to check additionally, it's production DB already(
Any help would be greatly appreciated.

Expected Behaviour

Doc is really replicated

Your Environment

{"couchdb":"Welcome","version":"3.1.0","git_sha":"ff0feea20","uuid":"6fd6fcaf95854eb6db4a7afa44b0454a","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
OS; Debian 10 (64)/Windows 8.1 (64)
Node v12.16.3

[wohali]

Is your document bigger than the new maximum document size allowed in 3.1.0?

[greenais]

@wohali no, kind of ~30KB only.

[janl]

does the route db have newer updates for the doc than the target? If not, then this is expected behaviour

[greenais]

So you say it's expected that if doc was deleted in target - the same doc couldn't be replicated from source to target anymore even if CDB is instructed to do it? Such DB logic sounds quite strange and unexpected to me

[janl]

CouchDB documents have a linear history. You can have branches for conflicts, but we can ignore that for now. Replication in CouchDB is defined as move the latest revision of a document from source to target. CouchDB uses the document revision history to decide whether it has to replicate a document at all. If the target has the same or a new or revision, as defined by the revision history, the goal of replication has already been achieved, and CouchDB should be does not replicate the doctor again. This has been the behaviour since the very first versions of CouchDB and it is well documented, so I’m surprised this is unexpected.

[greenais]

But it is, since as I wrote such replication worked just fine in my app until upgrade from 2.2 to 3.1. And to my point of view it's quite strange to ignore deleted flag in such cases - from rational side this logic makes no sense.
Replication means that I want doc copy in new place, but taking into account just revision change (which really occurs when deleted flag is added) and neglecting the fact that doc was deleted (and literally isn't available for consuming) makes this impossible.

[janl]

this behaviour didn’t change between 2 and 3, so if you see this, something else must be going on. Maybe you can post a full example with just a few docs, ideally using curl commands that demonstrates the issue, then we can find out better what the problem is.