Arel Extra (#189)

README.md

@@ -108,6 +108,8 @@ Use the official extras, or write your own in just a few lines. Extras add speci
 
 ### Backend Extras
 
+- [arel](http://ddnexus.github.io/pagy/extras/arel): Better performance of grouped ActiveRecord collections
+Paginate arrays efficiently, avoiding expensive array-wrapping and without overriding
 - [array](http://ddnexus.github.io/pagy/extras/array): Paginate arrays efficiently, avoiding expensive array-wrapping and without overriding
 - [countless](http://ddnexus.github.io/pagy/extras/countless): Paginate without the need of any count, saving one query per rendering
 - [elasticsearch_rails](http://ddnexus.github.io/pagy/extras/elasticsearch_rails): Paginate `ElasticsearchRails` response objects

docs/_layouts/default.html

@@ -102,6 +102,7 @@ <h1 id="site-title">{{ site.title | default: site.github.repository_name }}
       <a href="{{ site.baseurl }}/api/frontend"><p class="indent1" {% if page.title == 'Pagy::Frontend' %}id="active"{% endif %} >Pagy::Frontend</p></a>
       <a href="{{ site.baseurl }}/api/countless"><p class="indent1" {% if page.title == 'Pagy::Countless' %}id="active"{% endif %} >Pagy::Countless</p></a>
       <a href="{{ site.baseurl }}/extras"><p {% if page.title == 'Extras' %}id="active"{% endif %} >Extras</p></a>
+      <a href="{{ site.baseurl }}/extras/arel"><p class="indent1" {% if page.title == 'Arel' %}id="active"{% endif %} >Arel</p></a>
       <a href="{{ site.baseurl }}/extras/array"><p class="indent1" {% if page.title == 'Array' %}id="active"{% endif %} >Array</p></a>
       <a href="{{ site.baseurl }}/extras/bootstrap"><p class="indent1" {% if page.title == 'Bootstrap' %}id="active"{% endif %} >Bootstrap</p></a>
       <a href="{{ site.baseurl }}/extras/bulma"><p class="indent1" {% if page.title == 'Bulma' %}id="active"{% endif %} >Bulma</p></a>

docs/extras.md

@@ -7,6 +7,7 @@ Pagy comes with a few optional extensions/extras:
 
 | Extra                 | Description                                                                                                                                | Links                                                                                                                                                                                   |
 |:----------------------|:-------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `arel`                | Better performance of grouped ActiveRecord collections                                                                                     | [arel.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/arel.rb), [documentation](http://ddnexus.github.io/pagy/extras/arel)                                              |
 | `array`               | Paginate arrays efficiently avoiding expensive array-wrapping and without overriding                                                       | [array.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/array.rb), [documentation](http://ddnexus.github.io/pagy/extras/array)                                           |
 | `bootstrap`           | Add nav, nav_js and combo_nav_js helpers for the Bootstrap [pagination component](https://getbootstrap.com/docs/4.1/components/pagination) | [bootstrap.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/bootstrap.rb), [documentation](http://ddnexus.github.io/pagy/extras/bootstrap)                               |
 | `bulma`               | Add nav, nav_js and combo_nav_js helpers for the Bulma [pagination component](https://bulma.io/documentation/components/pagination)        | [bulma.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/bulma.rb), [documentation](http://ddnexus.github.io/pagy/extras/bulma)                                           |

docs/extras/arel.md

@@ -0,0 +1,47 @@
+---
+title: Arel
+---
+# Arel Extra
+
+This extra adds a specialized pagination for collections from sql databases with `GROUP BY` clauses, by computing the total number of results with `COUNT(*) OVER ()`. It was tested against MySQL (8.0.17) and Postgres (11.5). Before using in a different database, make sure the sql `COUNT(*) OVER ()` performs a count of all the lines after the `GROUP BY` clause is applied.
+
+## Synopsis
+
+See [extras](../extras.md) for general usage info.
+
+In the `pagy.rb` initializer:
+
+```ruby
+require 'pagy/extras/arel'
+```
+
+In a controller:
+
+```ruby
+@pagy_a, @items   = pagy_arel(a_collection, ...)
+
+# independently paginate some other collections as usual
+@pagy_b, @records = pagy(some_scope, ...)
+```
+
+## Files
+
+- [arel.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/arel.rb)
+
+## Methods
+
+This extra adds the `pagy_arel` method to the `Pagy::Backend` to be used in place (or in parallel) of the `pagy` method when you want to paginate a collection with sql `GROUP BY` clause, where the number of groups/rows is big (for instance when group by day of the last 3 years). It also adds a `pagy_arel_get_variables` sub-method, used for easy customization of variables by overriding.
+
+**Notice**: there is no `pagy_arel_get_items` method to override, since the items are fetched directly by the specialized `pagy_arel` method.
+
+### pagy_arel(collection, vars=nil)
+
+This method is the same as the generic `pagy` method, but with improved speed for SQL `GROUP BY` collections. (see the [pagy doc](../api/backend.md#pagycollection-varsnil))
+
+### pagy_arel_get_vars(collection)
+
+This sub-method is the same as the `pagy_get_vars` sub-method, but it is called only by the `pagy_arel` method. (see the [pagy_get_vars doc](../api/backend.md#pagy_get_varscollection-vars)).
+
+### pagy_arel_count(collection)
+
+This sub-method it is called only by the `pagy_get_vars` method. It will detect which query to perform based on the active record groups (sql `GROUP BY`s). In case there aren't group values performs a normal `.count(:all)`, otherwise it will perform a `COUNT(*) OVER ()`. The last tells database to perform a count of all the lines after the `GROUP BY` clause is applied.

docs/how-to.md

@@ -307,7 +307,7 @@ Please, use the [array](extras/array.md) extra.
 
 ## Paginate ActiveRecord collections
 
-Pagy works out of the box with `ActiveRecord` collections.
+Pagy works out of the box with `ActiveRecord` collections. See also the [arel](http://ddnexus.github.io/pagy/extras/arel) for better performance of grouped ActiveRecord collections.
 
 ## Paginate Ransack results
 
@@ -390,6 +390,8 @@ custom_count = ...
 
 **Notice**: pagy will efficiently skip its internal count query and will just use the passed `:count` variable
 
+See also the [arel](http://ddnexus.github.io/pagy/extras/arel) for better performance of grouped ActiveRecord collections.
+
 ## Using the pagy_nav* helpers
 
 These helpers take the Pagy object and return the HTML string with the pagination links, which are wrapped in a `nav` tag and are ready to use in your view. For example:
@@ -501,6 +503,10 @@ after_destroy { Rails.cache.delete_matched /^pagy-#{self.class.name}:/}
 
 That may work very well with static (or almost static) DBs, where there is not much writing and mostly reading. Less so with more DB writing, and probably not particularly useful with a DB in constant change.
 
+### Using the arel extra
+
+For better performance of grouped ActiveRecord collection counts, you may want to take a look at the [arel](http://ddnexus.github.io/pagy/extras/arel).
+
 ### Avoiding the count
 
 When the count caching is not an option, you may want to use the [countless extra](extras/countless.md), which totally avoid the need for a count query, still providing an acceptable subset of the full pagination features.

lib/pagy/extras/arel.rb

@@ -0,0 +1,31 @@
+# See the Pagy documentation: https://ddnexus.github.io/pagy/extras/arel
+# encoding: utf-8
+# frozen_string_literal: true
+
+class Pagy
+  module Backend ; private
+
+    def pagy_arel(collection, vars={})
+      pagy = Pagy.new(pagy_arel_get_vars(collection, vars))
+      return pagy, pagy_get_items(collection, pagy)
+    end
+
+    def pagy_arel_get_vars(collection, vars)
+      vars[:count] ||= pagy_arel_count(collection)
+      vars[:page]  ||= params[ vars[:page_param] || VARS[:page_param] ]
+      vars
+    end
+
+    def pagy_arel_count(collection)
+      if collection.group_values.empty?
+        # COUNT(*)
+        collection.count(:all)
+      else
+        # COUNT(*) OVER ()
+        sql = Arel.star.count.over(Arel::Nodes::Grouping.new([]))
+        collection.unscope(:order).limit(1).pluck(sql).first
+      end
+    end
+
+  end
+end

test/mock_helpers/arel.rb

@@ -0,0 +1,20 @@
+module Arel
+  def self.star(*)
+    self
+  end
+
+  def self.count(*)
+    self
+  end
+
+  def self.over(*)
+    self
+  end
+
+  module Nodes
+    class Grouping
+      def initialize(array)
+      end
+    end
+  end
+end

test/mock_helpers/collection.rb

@@ -11,18 +11,38 @@ def offset(value)
   end
 
   def limit(value)
-    @collection[0, value]
+    if value == 1
+      self # used in pluck
+    else
+      @collection[0, value]
+    end
   end
 
   def count(*)
     size
   end
 
+  def pluck(*)
+    [size]
+  end
+
+  def group_values
+    []
+  end
+
   class Grouped < MockCollection
 
     def count(*)
       Hash[@collection.map { |value| [value, value + 1] }]
     end
 
+    def unscope(*)
+      self
+    end
+
+    def group_values
+      [:other_table_id]
+    end
+
   end
 end

test/pagy/extras/arel_test.rb

@@ -0,0 +1,94 @@
+# encoding: utf-8
+# frozen_string_literal: true
+
+require_relative '../../test_helper'
+require_relative '../../mock_helpers/arel'
+require 'pagy/extras/arel'
+
+describe Pagy::Backend do
+
+  let(:controller) { MockController.new }
+
+  describe "#pagy_arel" do
+
+    before do
+      @collection = MockCollection.new
+    end
+
+    it 'paginates with defaults' do
+      pagy, items = controller.send(:pagy_arel, @collection)
+      pagy.must_be_instance_of Pagy
+      pagy.count.must_equal 1000
+      pagy.items.must_equal Pagy::VARS[:items]
+      pagy.page.must_equal controller.params[:page]
+      items.size.must_equal Pagy::VARS[:items]
+      items.must_equal [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60]
+    end
+
+    it 'paginates with vars' do
+      pagy, items = controller.send(:pagy_arel, @collection, page: 2, items: 10, link_extra: 'X')
+      pagy.must_be_instance_of Pagy
+      pagy.count.must_equal 1000
+      pagy.items.must_equal 10
+      pagy.page.must_equal 2
+      pagy.vars[:link_extra].must_equal 'X'
+      items.size.must_equal 10
+      items.must_equal [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
+    end
+
+  end
+
+  describe "#pagy_arel_get_vars" do
+
+    before do
+      @collection = MockCollection.new
+    end
+
+    it 'gets defaults' do
+      vars   = {}
+      merged = controller.send :pagy_arel_get_vars, @collection, vars
+      merged.keys.must_include :count
+      merged.keys.must_include :page
+      merged[:count].must_equal 1000
+      merged[:page].must_equal 3
+    end
+
+    it 'gets vars' do
+      vars   = {page: 2, items: 10, link_extra: 'X'}
+      merged = controller.send :pagy_arel_get_vars, @collection, vars
+      merged.keys.must_include :count
+      merged.keys.must_include :page
+      merged.keys.must_include :items
+      merged.keys.must_include :link_extra
+      merged[:count].must_equal 1000
+      merged[:page].must_equal 2
+      merged[:items].must_equal 10
+      merged[:link_extra].must_equal 'X'
+    end
+
+    it 'works with grouped collections' do
+      @collection = MockCollection::Grouped.new((1..1000).to_a)
+      vars   = {page: 2, items: 10, link_extra: 'X'}
+      merged = controller.send :pagy_arel_get_vars, @collection, vars
+      merged.keys.must_include :count
+      merged.keys.must_include :page
+      merged.keys.must_include :items
+      merged.keys.must_include :link_extra
+      merged[:count].must_equal 1000
+      merged[:page].must_equal 2
+      merged[:items].must_equal 10
+      merged[:link_extra].must_equal 'X'
+    end
+
+    it 'overrides count and page' do
+      vars   = {count: 10, page: 32}
+      merged = controller.send :pagy_arel_get_vars, @collection, vars
+      merged.keys.must_include :count
+      merged[:count].must_equal 10
+      merged.keys.must_include :page
+      merged[:page].must_equal 32
+    end
+
+  end
+
+end