Implement _bulk_get aka _bulk_revs

[nolanlawson]

Counterpart to pouchdb/express-pouchdb#196.

[nolanlawson]

It's debatable, though, whether we actually want to implement a db.bulkGet() or db.bulkRevs(). For local-local replication it doesn't offer any benefit, and may actually confuse users.

As a first pass, though, we can simply update replicate.js to use a server-side bulk endpoint if it exists.

[dholth]

See also https://github.com/dholth/pouchdb-bulk-get

[robertkeizer]

+1

[nolanlawson]

@daleharvey On the CouchDB side (COUCHDB-2310), there definitely seems to be an "intent to ship" in both 1.x and 2.x.

So how do you feel about adding db.bulkRevs()? I'm leaning towards implementing it at the PouchDB API level, i.e. not simply an endpoint on PouchDB Server that wraps Promise.all and db.get. Although most endusers will never use it themselves, it's nice for consistency's sake.

[daleharvey]

I am not so sure about this, if couchdb do add it then we probably have to for compat, but I think we would be able to spec and implement stream before this bulkrevs is ever released and replication-stream is a much more desirable end goal, going from what we have now to adding bulkDocs / bulkRevs (+detection) to adding Streaming + detection doesnt seem ideal

[nolanlawson]

So you think we should just jump directly to the replication-stream stuff?

[robertkeizer]

I'm finding that a middleware layer that performs the replication-stream stuff works fairly well. Have some hacky code working for express-pouchdb but need to clean it up before committing.

Personally I'd like to see Pouch implement this without replication-stream functionality as soon as couch has it.

[dholth]

_bulk_revs (a more efficient API that can replace the 1-at-a-time
GET API) is useful for replication, but it's not necessarily just
about replication. You might just want to get a lot of things with a
single call.

On Mon, Feb 9, 2015, at 08:50 AM, Robert Keizer wrote:

I'm finding that a middleware layer that performs the
replication-stream stuff works fairly well. Have some hacky code
working for express-pouchdb but need to clean it up before committing.

Personally I'd like to see Pouch implement this without
replication-stream functionality as soon as couch has it.

— Reply to this email directly or view it on GitHub[1].

Links:

1. Implement _bulk_get aka _bulk_revs #3424 (comment)

[daleharvey]

You might just want to get a lot of things with a single call.

Hrm, I thought allDocs supported fetching multiple documents, doesnt look like so.

If couch implements it then we should, but in terms of focus for replication improvements I think -stream is the best thing to focus on, it is going to be very hard to implement any of these new changes well since they will be implemented at different times across different platforms, I think its worth focussing on what gets us the best bang for buck

[dholth]

Maybe it does. The thing I found irritating was that none of the proposed or existing bulk APIs supported ALL of the parameters allowed in a GET request, they are arbitrarily different making them harder to learn.

[willholley]

@nolanlawson when testing against the CouchDB implementation, our attachment tests are failing. Looking specifically at suite2 test.attachments.js- local:http / Attachments replicate back and forth, I see the following requests around the failure:

without _bulk_get:

GET http://localhost:2020/test_attach_remote/doc?revs=true&open_revs=[%221-67421d156cc0ba65d13b9c601c558381%22]&_nonce=1441282816675

returns

[{ "ok": {
  "_id": "doc",
  "_rev": "1-67421d156cc0ba65d13b9c601c558381",
  "_revisions": {
     "start": 1,
     "ids": [
        "67421d156cc0ba65d13b9c601c558381"
     ]
  },
  "_attachments": {
     "foo.txt": {
        "content_type": "text/plain",
        "revpos": 1,
        "digest": "md5-UVHkefsf5MqAl/2ALjx5Zg==",
        "length": 3,
        "stub": true
     }
  }
}}]

with _bulk_get:

 POST http://localhost:2020/test_attach_remote/_bulk_get?_nonce=1441284778447
 {"revs":true,"attachments":true,"binary":true,"docs":[{"id":"doc","rev":"1-67421d156cc0ba65d13b9c601c558381"}]}

returns

 {"results": [{
  "id": "doc", 
   "docs": [{
    "ok": {
     "_id":"doc",
     "_rev":"1-67421d156cc0ba65d13b9c601c558381",
     "_attachments":{
      "foo.txt":{
       "content_type":"text/plain",
       "revpos":1,
       "digest":"md5-UVHkefsf5MqAl/2ALjx5Zg==",
       "length":3,
       "stub":true
      }
     }
    }
   }]
 }]}

so the core information around the attachment looks the same.

However, when this is processed at

pouchdb/lib/deps/docs/preprocessAttachments.js

Line 87 in 39da601

for (var key in docInfo.data._attachments) {

, the docInfo object is very different.

without _bulk_get:

    {
  "metadata": {
    "id": "doc2",
    "rev": "1-67421d156cc0ba65d13b9c601c558381",
    "revisions": {
      "start": 1,
      "ids": [
        "67421d156cc0ba65d13b9c601c558381"
      ]
    },
    "rev_tree": [
      {
        "pos": 1,
        "ids": [
          "67421d156cc0ba65d13b9c601c558381",
          {
            "status": "available"
          },
          []
        ]
      }
    ]
  },
  "data": {
    "_attachments": {
      "foo.txt": {
        "data": {},
        "digest": "md5-rL0Y20zC+Fzt72VPzMSk2A==",
        "content_type": "text/plain",
        "revpos": 1,
        "length": 3
      }
    }
  }
}

with bulk_get:

    {
  "metadata": {
    "id": "doc2",
    "rev": "1-67421d156cc0ba65d13b9c601c558381",
    "rev_tree": [
      {
        "pos": 1,
        "ids": [
          "67421d156cc0ba65d13b9c601c558381",
          {
            "status": "available"
          },
          []
        ]
      }
    ]
  },
  "data": {
    "_attachments": {
      "foo.txt": {
        "digest": "md5-UVHkefsf5MqAl/2ALjx5Zg==",
        "content_type": "text/plain",
        "length": 3,
        "revpos": 1,
        "stub": true
      }
    }
  }
}

Note that when using _bulk_get, the attachment remains as a stub and the digest is different (but matches what was returned from the server in both cases).

[willholley]

I think the problem is simply that we don't call fetchAttachments for attachment stubs returned by _bulk_get

[nolanlawson]

_bulk_get will be released in pouchdb 5.0.0 and available in the next release of pouchdb server

[nolanlawson]

this is merged now

[dholth]

:-)

[dholth]

So this means PouchDB will attempt to use _bulk_get for replication now?

[willholley]

yep - if the endpoint is there, it will use it.

[ronycohen]

which current couchDB version will work with _bulk_get ?

[nolanlawson]

CouchDB 2.0+ (unreleased) and PouchDB Server 1.0.0. Cloudant soon.

[ronycohen]

Thank you @nolanlawson

[yaronyg]

@nolanlawson @daleharvey At the risk of sounding like a hopeless fan boy this is super cool! Thank you!!!!

BTW, as a side note today we did a demo using PouchDB with memdown on top of node.ch on top of Windows IoT on top of a Raspberry PI 2. It worked super well! So my thanks only increases!

[nolanlawson]

@yaronyg woop woop! :)