Skip to content

add batch view method for faster pagination #156

Open
wants to merge 3 commits into from

2 participants

@jhecking
jhecking commented Oct 6, 2012

'Fast Paging', a.k.a. Batch Loading

Pagination using just skip and limit can be very slow as CouchDB
still needs to read all the rows of the view that it skips. That's
why the CouchDB team recommends to use the skip parameter only with
"single digit values." The batch method follows the 'Fast Paging'
recipe [1] for using startkey(_docid) for faster pagination.

Note that 'Fast Paging' does not allow jumping directly to a specific
page within the results. If that is required the page method should
be used instead.

Use of the batch method is recommended esp. for use cases where it is
necessary to load all the documents in a view for processing but
loading them all at once is undesirable, e.g. due to memory
constraints.

[1] http://guide.couchdb.org/draft/recipes.html#pagination

jhecking added some commits Oct 7, 2012
@jhecking jhecking add batch view method for faster pagination
'Fast Paging', a.k.a. Batch Loading

Pagination using just skip and limit can be very slow as CouchDB
still needs to read all the rows of the view that it skips. That's
why the CouchDB team recommends to use the skip parameter only with
"single digit values." The batch method follows the 'Fast Paging'
recipe [1] for using startkey(_docid) for faster pagination.

Note that 'Fast Paging' does not allow jumping directly to a specific
page within the results. If that is required the page method should
be used instead.

Use of the batch method is recommended esp. for use cases where it is
necessary to load all the documents in a view for processing but
loading them all at once is undesirable, e.g. due to memory
constraints.

[1] http://guide.couchdb.org/draft/recipes.html#pagination
abf92c4
@jhecking jhecking support Enumerators for batch loading
When called without a block argument the batch() method should return an
Enumerator object instead.
b168082
@jhecking

Is anyone still actively working on couchrest_model at all?

@samlown
CouchRest member
samlown commented Jan 11, 2013

Hi. It is still being worked on, just very slowly, and by very few people. I indeed use it in a production environment and endeavour to fix things when they go wrong.

Regarding the pull request. It looks good, but I'm not entirely convinced about including it directly in the View module as it is not strictly part of how views in CouchDB work; it works on top-of views, which suggests it should be contained in a separate helper module. Let me think about it and see what can be done.

Additionally, it would be great if there could be some additional tests to catch the exceptions.

@jhecking

I've added more specs for the exceptions as you recommended.

I implemented the batch functionality in the view class directly since I thought it was similar to the kaminari-compatible pagination functionality the class already has. Granted the pagination methods are a bit simpler and there is a stronger case for supporting that interface directly on the view.

An alternative interface for the batch functionality could look something like this:

view = Model.by_name()
Model.batch_load(view, batch_size) do |docs|
  # do something interesting with docs
end

Or with a possible convenience shortcut notation:

Model.batch_load(:by_name, batch_size) do |docs|
  # do something interesting with docs
end

Just passing in the view name would instantiate the view automatically with default parameters.

I haven't thought about the implementation of this alternative interface but I don't foresee any mayor complications. If you feel very strongly that the batch load support should not be in the view I could take a stab at this.

Or let me know if you have some other idea about where this functionality belongs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.