Support reverse scroll

[pferrel]

Not sure how to contact you so I'll try here.

I have a bi-directional infinite scroll. I enter at an object defined by a query. It has some location in an infinite list defined by a sort order. Batches of objects are filled in downwards with skip/limit. If the order_by is simple and unique (like username), I reverse the sort and get objects above the entry point with skip/limit--since the order_by is reverse this gets object above.

When the order_by is complex, there is no way to define a query on the reverse sorted collection that gets to the same "entry point" as the forward order.

What I really want is to have negative skip values on the forward sorted collection. This would be ideal and this seems possible with a cursor but I haven't bothered learning about them since they are time limited.

For example, a sorted collection might have the following order_by:

scope :rating_watchable, order_by(avg_rating: :desc, release_date: :desc, title: :asc)

given that this has the unique "title" in it the list can be reversed. The "entry point" might be the first item with a rating of 3 defined like this

Object.rating_watchable.where(:avg_rating.gte => 3).skip(0).limit(BATCH_SIZE)

To scroll forward I simply increase the skip count. To scroll backward I have to reverse the sort order. If I were using a simple sort order like alphabetical on a unique field this isn't a problem but for the one above it doesn't work. But in the example above I can't determine a "where" that puts me in exactly the same spot since the "where" will have to be relative.

I think of the problem as a long list (defined by a sort order). I can find an entry point and save it's UUID. Then I'd like to move forward and back from that object in this list.

Will mongoid-scroll handle forward and reverse movement from some "entry point"?

[dblock]

That's an interesting problem.

Mongoid-scroll is based of the concept of a cursor, which can be literally anything that can point to an item that can be clearly compared to. So it will definitely handle the forward movement without any code changes. The backward movement is something that would need to be implemented.

I've renamed this question into "Support reverse scroll", which I think is everything that you need. From the user point of view I think it could be a .reverse_each operation. Would love some specs to start and I could probably help with an implementation.

[pferrel]

Oops, changed a typo--in case you are getting this in email I've fixed a couple things below.

Awesome. I have an implementation of the thing on a list sorted by a unique title. Maybe we can use it as an example, since it already works then replace the sort order with the one above, which breaks my implementation.

forward sort:

scope :alpha_watchable, order_by(title: :asc)

reverse:

scope :alpha_watchable_up, order_by(title: :desc)

The user decides to start scrolling/browsing at "M" so I append a start_at where clause. Thus:

alpha_watchable.where(:title.gte => start_at).skip(index).limit(count)

Then each time the client asks for an index and count from that point. I'm using ng-scroll and angular. The directive supports negative indexes so then the user scrolls up I get a request for index = -1, and some batch size. This I turn into the following where:

alpha_watchable_up.where(:title.lt => start_at).skip(-skip).limit(limit)

When I get a request that straddles my "entry point" defined by index = 0, I have to create two queries. So this is the case where index = -10, count = 50 or something like that. I have this working scrolling up and down infinitely, with bookmarks and back working. To do bookmark/back I replace the location bar with the query params that will retrieve the first item in the viewport (actually UUID). Then this item becomes the "entry point" and subsequent scrolls are relative to it. I have to calculate the appropriate query from the object at the UUID and this is another place the problem shows up since I can't just tell the skip/limit to start at a UUID in the sorted list.

This is all to get around not having a .skip(-1). And if you replace the simple sort with the complex multi-field one, you can no longer use :title.lt on the reverse sort to get the right starting point. Imagine this sort order:

scope :release_watchable, order_by(release_date: :desc, avg_rating: :desc, title: :asc)

While I can reverse it I can't define a "where" from the entry point object that starts at the right point in the reversed list.

Ng-scroll has a pretty simple api. It asks for index=[-100000..100000], count=n, which fits skip/limit in the forward direction. If your mongoid-scroll allowed the same type of negative skip/index and allowed a starting point defined by a UUID or some item returning query, this would be so slick and easy.

BTW ng-scroll also does deletion of objects too far off the viewport and so is pretty robust and realistic to use generally. You can see it here: https://github.com/Hill30/NGScroller It is part of the Angular-UI project.

I'm pretty motivated to solve this so I'll help any way I can if you think mongoid-scroll is a good fit for the problem.

There is a stackoverflow question about this here: http://stackoverflow.com/questions/20557674/mongodb-using-limit-and-skip-on-complex-sort-order-in-either-direction

[pferrel]

From your examples it looks like they use a unique sort key, like :position? Not sure if this is a requirement but it is at the crux of my problem. I could, in a background task, insert a sort key for each sort order. Then index the key. This would solve my problem but is rather crude. To handle delete and insert the key would have to be a float and you'd have to calculate the right sort key on every insert--ugh!

  Feed::Item.asc(:position).limit(scroll_by).scroll(current_cursor) do |item, cursor|

Seems to move the cursor in a forward manner like an iterator. Is there a way to iterate in reverse? Does it work to change .asc to .desc on any scope?

Any thoughts?

[dblock]

The short answer is that I am not sure. You should start by trying to implement a reverse iterator in this library, and start with tests.

[pferrel]

Unfortunately I don't know how. In my current implementation the reverse iterator requires a reverse sort order from the same starting point as the forward sort order. Reversing the sort order makes it impossible to define a starting point that will be the same as in the forward order.

I looked at how you define the starting cursor, with a criteria that will go into .where clause with some .gt or .lt and my example above breaks this. Also the cursor seems to take only one field value for the sort so it wouldn't work for me without major mods anyway. I was hoping that you were actually using a Mongo cursor in some way that keeps it alive indefinitely. Something that is beyond me. But from reading the MongoDB cursor def I'm not sure it would avoid the forward and reverse from the same spot either.

      def initialize(value = nil, options = {})
        @field_type, @field_name = Mongoid::Scroll::Cursor.extract_field_options(options)
        @direction = options[:direction] || 1
        parse(value)
      end

      def criteria
        mongo_value = value.class.mongoize(value) if value
        compare_direction = direction == 1 ? "$gt" : "$lt"
        cursor_criteria = { field_name => { compare_direction => mongo_value } } if mongo_value
        tiebreak_criteria = { field_name => mongo_value, :_id => { compare_direction => tiebreak_id } } if mongo_value && tiebreak_id
        cursor_criteria || tiebreak_criteria ? { '$or' => [cursor_criteria, tiebreak_criteria].compact } : {}
      end

[dblock]

See above. Write failing tests that describe the functionality you want first. I'll help with the implementation.