Missing headers in replication documents

[pnagle]

Sometimes headers go missing from replication document causing replication failure because of the unauthorized request. We are using clustered environment (replica=2 & shards=1). Also the user id and password goes missing from the target object.

Expected Behavior

The headers should be present

{
  "doc": {
    "_id": "aG91bTovL3RkMTExMTExNjQwOTkyNjUzNDcxMjYwaG91bWluMTUyMjc2NDIwODA0My1jNDYxY2Y3OS5ob3VtLm1l_houm4_pull60faf6ed-4cf4-4411-bc23-9cb13f4ea118",
    "_rev": "1-59092a13de9ec7029f445251cad90aa6",
    "houmId": "****://td1111116409926534712*******.****.**",
    "use_checkpoints": true,
    "checkpoint_interval": "10",
    "filter": "full_replication/level_replication",
    "source": {
      "url": "http://*******.***:80/DB_NAME"
      "headers": {
       "Authorization": "Bearer <token>"
      }
    },
   
    "target": "http://user_id:db_password@irep-couchdb:5984/userdb-7464313131313131363430393932363533343731323630686f756d696e",
    "createdAt": "2018-04-04T04:35:34.786Z",
    "lastUpdated": "2018-04-04T04:35:34.786Z",
    "replicationType": "_db_name_pull",
    "owner": "admin"
  }
},

Current Behavior

Headers goes missing, possible after an update of CouchDB itself.
Sample:-

{
  "doc": {
    "_id": "aG91bTovL3RkMTExMTExNjQwOTkyNjUzNDcxMjYwaG91bWluMTUyMjc2NDIwODA0My1jNDYxY2Y3OS5ob3VtLm1l_houm4_pull60faf6ed-4cf4-4411-bc23-9cb13f4ea118",
    "_rev": "1-59092a13de9ec7029f445251cad90aa6",
    "houmId": "****://td1111116409926534712*******.****.**",
    "use_checkpoints": true,
    "checkpoint_interval": "10",
    "filter": "full_replication/level_replication",
    "source": {
      "url": "http://*******.***:80/DB_NAME"
    },
    "target": "http://irep-couchdb:5984/userdb-7464313131313131363430393932363533343731323630686f756d696e",
    "createdAt": "2018-04-04T04:35:34.786Z",
    "lastUpdated": "2018-04-04T04:35:34.786Z",
    "replicationType": "_db_name_pull",
    "owner": "admin"
  }
},

Context

Our replication is failing in between, we are trying to accomplish unhindered heavy replicaiton.

Your Environment

• Version used: 2.2.0

[wohali]

When you say "version 2.2.0" do you mean you are running from master? Can you provide a git reference (commit) against which you are running? 2.2.0 isn't released yet.

/cc @nickva

[pnagle]

Yes, we are running from the master. Git reference 101f29b.

[jiangphcn]

Just want to exclude one possible scenario: which user you are using to check replication document? Is it owner or other users? We intentionally hide credential information for readers using #1126. Of course, this PR will not have impact on replication functionality, just for the purpose of security.

[pnagle]

We are using owner/admin. This happens on a lot of the documents and replication is failing but those documents whose headers are still available is working properly. Does replication jobs update the replication documents? Or is it happening because of replication document update conflict? Or an error in replication of replication database?

[jiangphcn]

For the case I mentioned above, it is just for display. So the actual content of replication content is not changed when non-owner is trying to view replication document.

For the case you mentioned, you mean that you are using owner/admin. If the header in replication document is not visible to owner, I don't believe that it belongs to the case I mentioned. There must be other concern case which causes the failure of replication.

[pnagle]

Thanks for the prompt reply guys. Have you found anything? If you want any more clarifications let me know.

[wohali]

If you can provide a script that allows for regular reproduction of the problem, that'd be very helpful.

[pnagle]

Sure, we are working on the script. Will share as soon as we get it done.

[pnagle]

We tried creating a replication script but we are unable to reproduce the issue on the script level. Meanwhile, did you find anything as to why this is occurring?

[jiangphcn]

So far I didn't get more progress. It is really helpful if we can get reproduce steps, even if it is not written in script.

[janl]

Closing until we have a reproducer.