Mongoid extension that enables infinite scrolling with MongoDB.
Ruby
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
examples Upgraded Rubocop to 0.43.0. Sep 27, 2016
lib
spec Mongoid 7 (#22) May 2, 2018
.gitignore Initial commit, base infrastructure. Feb 14, 2013
.rspec
.rubocop.yml Fix mongoid scroll without block returning wrong criteria (#18) Mar 11, 2018
.rubocop_todo.yml Run rubocop -a ; rubocop --auto-gen-config Mar 13, 2018
.travis.yml Mongoid 7 (#22) May 2, 2018
CHANGELOG.md Mongoid 7 (#22) May 2, 2018
Dangerfile Added Danger, PR linter. Sep 27, 2016
Gemfile Mongoid 7 (#22) May 2, 2018
LICENSE.md Added support for mongo-ruby-driver. Oct 22, 2015
README.md Update README to reflect support for mongoid 7 May 4, 2018
RELEASING.md Correct example versions in RELEASING.md May 1, 2018
Rakefile Upgraded Rubocop to 0.49.1. Sep 13, 2017
mongoid-scroll.gemspec Repository moved from dblock to the mongoid organization. Aug 2, 2016

README.md

Mongoid::Scroll

Gem Version Build Status Dependency Status Code Climate

Mongoid extension that enables infinite scrolling for Mongoid::Criteria, Moped::Query and Mongo::Collection::View.

Compatibility

This gem supports Mongoid 3, 4, 5, 6, 7, Moped and Mongo-Ruby-Driver.

Demo

Check out shows on artsy.net. Keep scrolling down.

There're also two code samples for Mongoid and Moped in examples. Run bundle exec ruby examples/mongoid_scroll_feed.rb.

The Problem

Traditional pagination does not work when data changes between paginated requests, which makes it unsuitable for infinite scroll behaviors.

  • If a record is inserted before the current page limit, items will shift right, and the next page will include a duplicate.
  • If a record is removed before the current page limit, items will shift left, and the next page will be missing a record.

The solution implemented by the scroll extension paginates data using a cursor, giving you the ability to restart pagination where you left it off. This is a non-trivial problem when combined with sorting over non-unique record fields, such as timestamps.

Installation

Add the gem to your Gemfile and run bundle install.

gem 'mongoid-scroll'

Usage

Mongoid

A sample model.

module Feed
  class Item
    include Mongoid::Document
    field :title, type: String
    field :position, type: Integer
    index({ position: 1, _id: 1 })
  end
end

Scroll by :position and save a cursor to the last item.

saved_cursor = nil
Feed::Item.desc(:position).limit(5).scroll do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Resume iterating using the previously saved cursor.

Feed::Item.desc(:position).limit(5).scroll(saved_cursor) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

The iteration finishes when no more records are available. You can also finish iterating over the remaining records by omitting the query limit.

Feed::Item.desc(:position).scroll(saved_cursor) do |record, next_cursor|
  # each record, one-by-one
end

Moped (Mongoid 3 and 4)

Scroll a Moped::Query and save a cursor to the last item. You must also supply a field_type of the sort criteria.

saved_cursor = nil
session[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Resume iterating using the previously saved cursor.

session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_cursor, { field_type: DateTime }) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Mongo-Ruby-Driver (Mongoid 5)

Scroll a Mongo::Collection::View and save a cursor to the last item. You must also supply a field_type of the sort criteria.

saved_cursor = nil
client[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Resume iterating using the previously saved cursor.

session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_cursor, { field_type: DateTime }) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Indexes and Performance

A query without a cursor is identical to a query without a scroll.

# db.feed_items.find().sort({ position: 1 }).limit(7)
Feed::Item.desc(:position).limit(7).scroll

Subsequent queries use an $or to avoid skipping items with the same value as the one at the current cursor position.

# db.feed_items.find({ "$or" : [
#   { "position" : { "$gt" : 13 }},
#   { "position" : 13, "_id": { "$gt" : ObjectId("511d7c7c3b5552c92400000e") }}
# ]}).sort({ position: 1 }).limit(7)
Feed:Item.desc(:position).limit(7).scroll(cursor)

This means you need to hit an index on position and _id.

# db.feed_items.ensureIndex({ position: 1, _id: 1 })

module Feed
  class Item
    ...
    index({ position: 1, _id: 1 })
  end
end

Cursors

You can use Mongoid::Scroll::Cursor.from_record to generate a cursor. A cursor points at the last record of the previous iteration and unlike MongoDB cursors will not expire.

record = Feed::Item.desc(:position).limit(3).last
cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["position"] })
# cursor or cursor.to_s can be returned to a client and passed into .scroll(cursor)

You can also a field_name and field_type instead of a Mongoid field.

cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: DateTime, field_name: "position" })

Contributing

Fork the project. Make your feature addition or bug fix with tests. Send a pull request. Bonus points for topic branches.

Copyright and License

MIT License, see LICENSE for details.

(c) 2013-2015 Daniel Doubrovkine, based on code by Frank Macreery, Artsy Inc.