Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

165 lines (118 sloc) 4.989 kb

Mongoid::Scroll

Gem Version Build Status Dependency Status Code Climate

Mongoid extension that enables infinite scrolling for Mongoid::Criteria and Moped::Query.

Demo

Check out shows on artsy.net. Keep scrolling down.

There're also two code samples for Mongoid and Moped in examples. Run bundle exec ruby examples/mongoid_scroll_feed.rb.

The Problem

Traditional pagination does not work when data changes between paginated requests, which makes it unsuitable for infinite scroll behaviors.

  • If a record is inserted before the current page limit, items will shift right, and the next page will include a duplicate.
  • If a record is removed before the current page limit, items will shift left, and the next page will be missing a record.

The solution implemented by the scroll extension paginates data using a cursor, giving you the ability to restart pagination where you left it off. This is a non-trivial problem when combined with sorting over non-unique record fields, such as timestamps.

Installation

Add the gem to your Gemfile and run bundle install.

gem 'mongoid-scroll'

Usage

Mongoid

A sample model.

module Feed
  class Item
    include Mongoid::Document
    field :title, type: String
    field :position, type: Integer
    index({ position: 1, _id: 1 })
  end
end

Scroll by :position and save a cursor to the last item.

saved_cursor = nil
Feed::Item.desc(:position).limit(5).scroll do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Resume iterating using the previously saved cursor.

Feed::Item.desc(:position).limit(5).scroll(saved_cursor) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

The iteration finishes when no more records are available. You can also finish iterating over the remaining records by omitting the query limit.

Feed::Item.desc(:position).scroll(saved_cursor) do |record, next_cursor|
  # each record, one-by-one
end

Moped

Scroll and save a cursor to the last item. You must also supply a field_type of the sort criteria.

saved_cursor = nil
session[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Resume iterating using the previously saved cursor.

session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_cursor, { field_type: DateTime }) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Indexes and Performance

A query without a cursor is identical to a query without a scroll.

# db.feed_items.find().sort({ position: 1 }).limit(7)
Feed::Item.desc(:position).limit(7).scroll

Subsequent queries use an $or to avoid skipping items with the same value as the one at the current cursor position.

# db.feed_items.find({ "$or" : [
#   { "position" : { "$gt" : 13 }},
#   { "position" : 13, "_id": { "$gt" : ObjectId("511d7c7c3b5552c92400000e") }}
# ]}).sort({ position: 1 }).limit(7)
Feed:Item.desc(:position).limit(7).scroll(cursor)

This means you need to hit an index on position and _id.

# db.feed_items.ensureIndex({ position: 1, _id: 1 })

module Feed
  class Item
    ...
    index({ position: 1, _id: 1 })
  end
end

Cursors

You can use Mongoid::Scroll::Cursor.from_record to generate a cursor. A cursor points at the last record of the previous iteration and unlike MongoDB cursors will not expire.

record = Feed::Item.desc(:position).limit(3).last
cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["position"] })
# cursor or cursor.to_s can be returned to a client and passed into .scroll(cursor)

You can also a field_name and field_type instead of a Mongoid field.

cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: DateTime, field_name: "position" })

Contributing

Fork the project. Make your feature addition or bug fix with tests. Send a pull request. Bonus points for topic branches.

Copyright and License

MIT License, see LICENSE for details.

(c) 2013 Daniel Doubrovkine, based on code by Frank Macreery, Artsy Inc.

Jump to Line
Something went wrong with that request. Please try again.