How about a fields parameter for the changes feed?

[nono]

Hello,

the fields parameter for the mango requests allow to fetch only some fields inside the documents, and it can help a lot to avoid some heavy fields and decrease the transfer size. I would like to have the same things for the changes feed. Sometimes, I use the changes feed not for replication, but just as a way to process some documents that have changed since the last time, and I don't need all the fields of the documents. Am I the only one in this case? What do you think of having a fields parameter for other things that mango?

[nickva]

@nono I had the same idea many years ago when implementing the _changes feed selector feature and added in the fields part. With the idea that the replicator might use it one day.

So it should already work!

Sadly, I don't think we ever documented it...

[nickva]

http put $DB/db
http put $DB/db/d1 a=b d=e
http put $DB/db/d2 a=b1 d=e1
http put $DB/db/d3 a=b42 d=e99
http put $DB/db/d4 a=b15 d=e17

http post $DB/db/_changes'?include_docs=true&filter=_selector' selector:='{"a" : {"$regex": "^b1"}}' fields:='["a"]'
{
    "last_seq": "4-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE5lygQLsKSYmaYYmlpjKcRqRxwIkGRqA1H8Uk4yMkkwsjM0wdWUBAEGvI-k",
    "pending": 0,
    "results": [
        {
            "changes": [
                {
                    "rev": "1-a90ad30f15b1be9fe5922cb55161ff1b"
                }
            ],
            "doc": {
                "a": "b1"
            },
            "id": "d2",
            "seq": "2-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYExlzgQLsKSYmaYYmlpjKcRqRxwIkGRqA1H8Uk4yMkkwsjM0wdWUBAEElI-c"
        },
        {
            "changes": [
                {
                    "rev": "1-3d231eb23b5e64ea45e0913b3912982a"
                }
            ],
            "doc": {
                "a": "b15"
            },
            "id": "d4",
            "seq": "4-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE5lygQLsKSYmaYYmlpjKcRqRxwIkGRqA1H8Uk4yMkkwsjM0wdWUBAEGvI-k"
        }
    ]
}

It still works it seems

http here is just the httpie command line client, and DB=http://adm:pass@127.0.0.1:15984, if you want to verify yourself

[nono]

Oh, it's very powerful. We can also use it to skip _deleted documents (which can be a nice optimization the first time we do a replication):

$ curl "http://localhost:5984/db/_changes?include_docs=true&filter=_selector"  -H "Content-Type: application/json" -d '{"selector": {"_deleted": {"$exists": false}}, "fields": ["a"]}'

What do you think about documenting it now? Is it something that will be still available with CouchDB 4?

[nickva]

We should document it since it already works, yeah.

The idea was to apply it to replications. It would work at the code level easily, but it would introduce tricky semantic side-effects. For example, it would replicate docs with same revision, say 1-a123... but having doc bodies which are different . In Apache CouchDB (unlike other CouchDB-like databases), revisions are generated from a hash of the doc body contents and the previous revision. Perhaps this change is easy to document and explain but I haven't thought of possible negative ramifications of it.

For skipping _deleted docs, it's possible to already do with:

"selector": {
  "_deleted": {
    "$exists": false
  }
}

Borrowed from https://blog.cloudant.com/2019/12/13/Filtered-Replication.html#ignoring-deletions

[nono]

IIRC, Pouch doesn't use the same algorithm for revisions, and I don't think it's an issue for replications between CouchDB and PouchDB.