Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
CouchDB
Erlang Python C C++ Shell Ruby Other

MB-13948 Queue items periodically

When a certain size (`QUEUE_ACC_BATCH_SIZE`) of data is accumulated
during the map phase, put all items into the writer queue. Without
this change, the memory consumption can grow almost unbound.

The reason for this is that the map queue is bound to the size of the
documents and not to the size of the emits. This means that a lot of
emits can lead to not anticipated sizes.

It can be simulated with the following view map function:

    function (doc, meta) {
        for(var i=0; i<100; i++) {
            emit([meta.id, i], "abcdefghij");
        }
    }

It emits 100 key-values per document. For small document sizes, for
example with a value size of 10 bytes (like above), this view function
lead to a memory usage of about 2.1GB on my machine. With this commit
applied it only took around 200MB.

Change-Id: Ib5102acbf4575794a5b6f5948c1dac3df5ded5e2
Reviewed-on: http://review.couchbase.org/48307
Tested-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Nimish Gupta <nimish@couchbase.com>
latest commit 8ee8f2b699
@vmx vmx authored vmx committed
Something went wrong with that request. Please try again.