diff --git a/README.md b/README.md index 5b15f3cd4..cef1b6873 100644 --- a/README.md +++ b/README.md @@ -108,6 +108,8 @@ Use the official extras, or write your own in just a few lines. Extras add speci ### Backend Extras +- [arel](http://ddnexus.github.io/pagy/extras/arel): Better performance of grouped ActiveRecord collections +Paginate arrays efficiently, avoiding expensive array-wrapping and without overriding - [array](http://ddnexus.github.io/pagy/extras/array): Paginate arrays efficiently, avoiding expensive array-wrapping and without overriding - [countless](http://ddnexus.github.io/pagy/extras/countless): Paginate without the need of any count, saving one query per rendering - [elasticsearch_rails](http://ddnexus.github.io/pagy/extras/elasticsearch_rails): Paginate `ElasticsearchRails` response objects diff --git a/docs/_layouts/default.html b/docs/_layouts/default.html index 6d6bc21ba..71d8c9e21 100644 --- a/docs/_layouts/default.html +++ b/docs/_layouts/default.html @@ -102,6 +102,7 @@

{{ site.title | default: site.github.repository_name }}

Pagy::Frontend

Pagy::Countless

Extras

 +

Arel

Array

Bootstrap

Bulma

 diff --git a/docs/extras.md b/docs/extras.md index 07aecedc5..fb05fc619 100644 --- a/docs/extras.md +++ b/docs/extras.md @@ -7,6 +7,7 @@ Pagy comes with a few optional extensions/extras: | Extra | Description | Links | |:----------------------|:-------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `arel` | Better performance of grouped ActiveRecord collections | [arel.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/arel.rb), [documentation](http://ddnexus.github.io/pagy/extras/arel) | | `array` | Paginate arrays efficiently avoiding expensive array-wrapping and without overriding | [array.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/array.rb), [documentation](http://ddnexus.github.io/pagy/extras/array) | | `bootstrap` | Add nav, nav_js and combo_nav_js helpers for the Bootstrap [pagination component](https://getbootstrap.com/docs/4.1/components/pagination) | [bootstrap.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/bootstrap.rb), [documentation](http://ddnexus.github.io/pagy/extras/bootstrap) | | `bulma` | Add nav, nav_js and combo_nav_js helpers for the Bulma [pagination component](https://bulma.io/documentation/components/pagination) | [bulma.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/bulma.rb), [documentation](http://ddnexus.github.io/pagy/extras/bulma) | diff --git a/docs/extras/arel.md b/docs/extras/arel.md new file mode 100644 index 000000000..0cf822d42 --- /dev/null +++ b/docs/extras/arel.md @@ -0,0 +1,47 @@ +--- +title: Arel +--- +# Arel Extra + +This extra adds a specialized pagination for collections from sql databases with `GROUP BY` clauses, by computing the total number of results with `COUNT(*) OVER ()`. It was tested against MySQL (8.0.17) and Postgres (11.5). Before using in a different database, make sure the sql `COUNT(*) OVER ()` performs a count of all the lines after the `GROUP BY` clause is applied. + +## Synopsis + +See [extras](../extras.md) for general usage info. + +In the `pagy.rb` initializer: + +```ruby +require 'pagy/extras/arel' +``` + +In a controller: + +```ruby +@pagy_a, @items = pagy_arel(a_collection, ...) + +# independently paginate some other collections as usual +@pagy_b, @records = pagy(some_scope, ...) +``` + +## Files + +- [arel.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/arel.rb) + +## Methods + +This extra adds the `pagy_arel` method to the `Pagy::Backend` to be used in place (or in parallel) of the `pagy` method when you want to paginate a collection with sql `GROUP BY` clause, where the number of groups/rows is big (for instance when group by day of the last 3 years). It also adds a `pagy_arel_get_variables` sub-method, used for easy customization of variables by overriding. + +**Notice**: there is no `pagy_arel_get_items` method to override, since the items are fetched directly by the specialized `pagy_arel` method. + +### pagy_arel(collection, vars=nil) + +This method is the same as the generic `pagy` method, but with improved speed for SQL `GROUP BY` collections. (see the [pagy doc](../api/backend.md#pagycollection-varsnil)) + +### pagy_arel_get_vars(collection) + +This sub-method is the same as the `pagy_get_vars` sub-method, but it is called only by the `pagy_arel` method. (see the [pagy_get_vars doc](../api/backend.md#pagy_get_varscollection-vars)). + +### pagy_arel_count(collection) + +This sub-method it is called only by the `pagy_get_vars` method. It will detect which query to perform based on the active record groups (sql `GROUP BY`s). In case there aren't group values performs a normal `.count(:all)`, otherwise it will perform a `COUNT(*) OVER ()`. The last tells database to perform a count of all the lines after the `GROUP BY` clause is applied. diff --git a/docs/how-to.md b/docs/how-to.md index 496b7bc4d..5f1df3063 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -307,7 +307,7 @@ Please, use the [array](extras/array.md) extra. ## Paginate ActiveRecord collections -Pagy works out of the box with `ActiveRecord` collections. +Pagy works out of the box with `ActiveRecord` collections. See also the [arel](http://ddnexus.github.io/pagy/extras/arel) for better performance of grouped ActiveRecord collections. ## Paginate Ransack results @@ -390,6 +390,8 @@ custom_count = ... **Notice**: pagy will efficiently skip its internal count query and will just use the passed `:count` variable +See also the [arel](http://ddnexus.github.io/pagy/extras/arel) for better performance of grouped ActiveRecord collections. + ## Using the pagy_nav* helpers These helpers take the Pagy object and return the HTML string with the pagination links, which are wrapped in a `nav` tag and are ready to use in your view. For example: @@ -501,6 +503,10 @@ after_destroy { Rails.cache.delete_matched /^pagy-#{self.class.name}:/} That may work very well with static (or almost static) DBs, where there is not much writing and mostly reading. Less so with more DB writing, and probably not particularly useful with a DB in constant change. +### Using the arel extra + +For better performance of grouped ActiveRecord collection counts, you may want to take a look at the [arel](http://ddnexus.github.io/pagy/extras/arel). + ### Avoiding the count When the count caching is not an option, you may want to use the [countless extra](extras/countless.md), which totally avoid the need for a count query, still providing an acceptable subset of the full pagination features. diff --git a/lib/pagy/extras/arel.rb b/lib/pagy/extras/arel.rb new file mode 100644 index 000000000..494274ec8 --- /dev/null +++ b/lib/pagy/extras/arel.rb @@ -0,0 +1,31 @@ +# See the Pagy documentation: https://ddnexus.github.io/pagy/extras/arel +# encoding: utf-8 +# frozen_string_literal: true + +class Pagy + module Backend ; private + + def pagy_arel(collection, vars={}) + pagy = Pagy.new(pagy_arel_get_vars(collection, vars)) + return pagy, pagy_get_items(collection, pagy) + end + + def pagy_arel_get_vars(collection, vars) + vars[:count] ||= pagy_arel_count(collection) + vars[:page] ||= params[ vars[:page_param] || VARS[:page_param] ] + vars + end + + def pagy_arel_count(collection) + if collection.group_values.empty? + # COUNT(*) + collection.count(:all) + else + # COUNT(*) OVER () + sql = Arel.star.count.over(Arel::Nodes::Grouping.new([])) + collection.unscope(:order).limit(1).pluck(sql).first + end + end + + end +end diff --git a/test/mock_helpers/arel.rb b/test/mock_helpers/arel.rb new file mode 100644 index 000000000..d38c3f796 --- /dev/null +++ b/test/mock_helpers/arel.rb @@ -0,0 +1,20 @@ +module Arel + def self.star(*) + self + end + + def self.count(*) + self + end + + def self.over(*) + self + end + + module Nodes + class Grouping + def initialize(array) + end + end + end +end diff --git a/test/mock_helpers/collection.rb b/test/mock_helpers/collection.rb index f9b3f4b34..ae9798595 100644 --- a/test/mock_helpers/collection.rb +++ b/test/mock_helpers/collection.rb @@ -11,18 +11,38 @@ def offset(value) end def limit(value) - @collection[0, value] + if value == 1 + self # used in pluck + else + @collection[0, value] + end end def count(*) size end + def pluck(*) + [size] + end + + def group_values + [] + end + class Grouped < MockCollection def count(*) Hash[@collection.map { |value| [value, value + 1] }] end + def unscope(*) + self + end + + def group_values + [:other_table_id] + end + end end diff --git a/test/pagy/extras/arel_test.rb b/test/pagy/extras/arel_test.rb new file mode 100644 index 000000000..ff75ff3af --- /dev/null +++ b/test/pagy/extras/arel_test.rb @@ -0,0 +1,94 @@ +# encoding: utf-8 +# frozen_string_literal: true + +require_relative '../../test_helper' +require_relative '../../mock_helpers/arel' +require 'pagy/extras/arel' + +describe Pagy::Backend do + + let(:controller) { MockController.new } + + describe "#pagy_arel" do + + before do + @collection = MockCollection.new + end + + it 'paginates with defaults' do + pagy, items = controller.send(:pagy_arel, @collection) + pagy.must_be_instance_of Pagy + pagy.count.must_equal 1000 + pagy.items.must_equal Pagy::VARS[:items] + pagy.page.must_equal controller.params[:page] + items.size.must_equal Pagy::VARS[:items] + items.must_equal [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60] + end + + it 'paginates with vars' do + pagy, items = controller.send(:pagy_arel, @collection, page: 2, items: 10, link_extra: 'X') + pagy.must_be_instance_of Pagy + pagy.count.must_equal 1000 + pagy.items.must_equal 10 + pagy.page.must_equal 2 + pagy.vars[:link_extra].must_equal 'X' + items.size.must_equal 10 + items.must_equal [11, 12, 13, 14, 15, 16, 17, 18, 19, 20] + end + + end + + describe "#pagy_arel_get_vars" do + + before do + @collection = MockCollection.new + end + + it 'gets defaults' do + vars = {} + merged = controller.send :pagy_arel_get_vars, @collection, vars + merged.keys.must_include :count + merged.keys.must_include :page + merged[:count].must_equal 1000 + merged[:page].must_equal 3 + end + + it 'gets vars' do + vars = {page: 2, items: 10, link_extra: 'X'} + merged = controller.send :pagy_arel_get_vars, @collection, vars + merged.keys.must_include :count + merged.keys.must_include :page + merged.keys.must_include :items + merged.keys.must_include :link_extra + merged[:count].must_equal 1000 + merged[:page].must_equal 2 + merged[:items].must_equal 10 + merged[:link_extra].must_equal 'X' + end + + it 'works with grouped collections' do + @collection = MockCollection::Grouped.new((1..1000).to_a) + vars = {page: 2, items: 10, link_extra: 'X'} + merged = controller.send :pagy_arel_get_vars, @collection, vars + merged.keys.must_include :count + merged.keys.must_include :page + merged.keys.must_include :items + merged.keys.must_include :link_extra + merged[:count].must_equal 1000 + merged[:page].must_equal 2 + merged[:items].must_equal 10 + merged[:link_extra].must_equal 'X' + end + + it 'overrides count and page' do + vars = {count: 10, page: 32} + merged = controller.send :pagy_arel_get_vars, @collection, vars + merged.keys.must_include :count + merged[:count].must_equal 10 + merged.keys.must_include :page + merged[:page].must_equal 32 + end + + end + +end