Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use count(:all) in HasManyAssociation#count_records #27561

Merged
merged 1 commit into from Jan 2, 2018
Merged

Use count(:all) in HasManyAssociation#count_records #27561

merged 1 commit into from Jan 2, 2018

Conversation

klaseskilson
Copy link
Contributor

Problem: Calling count on an association can cause invalid SQL queries to be created where the SELECT COUNT(a, b, c) function receives multiple columns. This will cause a StatementInvalid exception later on.

Solution: Use count(:all), which generates a SELECT COUNT(*)... query independently of the association.

Background: This appears to be previously an known problem, see #15138 and #13648.

Hit me with your feedback! 👍

Copy link
Member

@rafaelfranca rafaelfranca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good to me and it make sense to be fixed but could you add a test case to make sure it doesn't break again?

@klaseskilson
Copy link
Contributor Author

Sure thing, I'll give it a whirl. 👍

@klaseskilson
Copy link
Contributor Author

Other things are craving for my attention right now. For some reason, I am unable to replicate the problem in a test case. From what I've understood, the issue appears when you call size on a ActiveRecord::Associations::CollectionProxy, for a has_many association.

The stack looks like this:

activerecord (5.0.1) lib/active_record/connection_adapters/abstract_adapter.rb:578:in `translate_exception_class'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract_adapter.rb:591:in `rescue in log'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract_adapter.rb:589:in `log'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract_mysql_adapter.rb:218:in `execute'
activerecord (5.0.1) lib/active_record/connection_adapters/mysql/database_statements.rb:31:in `execute'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract_mysql_adapter.rb:225:in `execute_and_free'
activerecord (5.0.1) lib/active_record/connection_adapters/mysql/database_statements.rb:36:in `exec_query'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract/query_cache.rb:93:in `block in select_all'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract/query_cache.rb:108:in `cache_sql'
activerecord (5.0.1) lib/active_record/connection_adapters/abstract/query_cache.rb:93:in `select_all'
activerecord (5.0.1) lib/active_record/connection_adapters/mysql/database_statements.rb:10:in `select_all'
activerecord (5.0.1) lib/active_record/relation/calculations.rb:248:in `execute_simple_calculation'
activerecord (5.0.1) lib/active_record/relation/calculations.rb:207:in `perform_calculation'
activerecord (5.0.1) lib/active_record/relation/calculations.rb:121:in `calculate'
activerecord (5.0.1) lib/active_record/relation/calculations.rb:40:in `count'
activerecord (5.0.1) lib/active_record/associations/has_many_association.rb:79:in `count_records'
activerecord (5.0.1) lib/active_record/associations/collection_association.rb:323:in `size'
activerecord (5.0.1) lib/active_record/associations/collection_proxy.rb:767:in `size'
...

@klaseskilson
Copy link
Contributor Author

What I have so far in terms of test is

author = Author.create!(name: "A")
5.times { author.posts << Post.create!(title: "a", body: "b") }
same_authors_posts = Author.find(author.id).posts
assert_equal 5, same_authors_posts.size

The issue is that this works, and I'm not really sure what to do in order to actually end up in the same place as I do when the issue appears. Do you have any suggestions on how to proceed?

@rafaelfranca rafaelfranca self-assigned this Jan 6, 2017
@rafaelfranca
Copy link
Member

I'm afraid I don't have any good suggestion how to proceed other than try to isolate the code in your application that trigger the bug and replicate in a Rails test case. Maybe there is a select being used in the application for the association?

@klaseskilson
Copy link
Contributor Author

klaseskilson commented Jan 10, 2017

@rafaelfranca Yes, there was a select in there, I missed it since it was hidden in a default_scope.

There is now a test case here too. Thoughts? (Sorry for squashing the commits before pushing.)

@klaseskilson
Copy link
Contributor Author

@rafaelfranca Sorry to bother you again. Any chance you'll get to take a look at this anytime soon?

Problem: Calling `count` on an association can cause invalid SQL queries
to be created where the `SELECT COUNT(a, b, c)` function receives
multiple  columns. This will cause a `StatementInvalid` exception later
on.

Solution: Use `count(:all)`, which generates a `SELECT COUNT(*)...`
query independently of the association.

This also includes a test case that, before the fix, broke.
@klaseskilson
Copy link
Contributor Author

Hey @rafaelfranca, I rebased on top of master. Let me know if you still have any concerns regarding this.

@kamipo kamipo merged commit 4a7b4f8 into rails:master Jan 2, 2018
kamipo added a commit that referenced this pull request Jan 2, 2018
…ation

Use `count(:all)` in HasManyAssociation#count_records
kamipo added a commit that referenced this pull request Jan 3, 2018
…ation

Use `count(:all)` in HasManyAssociation#count_records
bogdanvlviv added a commit to bogdanvlviv/rails that referenced this pull request Mar 18, 2018
- Remove
  ```
  *   Namespace error pages' CSS selectors to stop the styles from bleeding
      into other pages when using Turbolinks.
      ([Pull Request](rails#28814))
  ```
  since it was backported to
  `5-1-stable` by 50d5baf
  and
  `5-0-stable` by d1c4a39.

- Remove
  ```
  *   Allow irb options to be passed from `rails console` command.
      ([Pull Request](rails#29010))
  ```
  since it was backported to
  `5-1-stable` by e91b483.

- Remove
  ```
  *   Load environment file in `dbconsole` command.
      ([Pull Request](rails#29725))
  ```
  since it was backported to
  `5-1-stable` by 7f93428.

- Remove
  ```
  *   Gemfile for new apps: upgrade redis-rb from ~> 3.0 to 4.0.
      ([Pull Request](rails#30748))
  ```
  since it was backported to
  `5-1-stable` by 3789531.

- Remove
  ```
  *   Fix minitest rails plugin.
      The custom reporters are added only if needed.
      This will fix conflicts with others plugins.
      ([Commit](rails@ac99916))
  ```
  since it was backported to
  `5-1-stable` by caa7695.

- Remove
  ```
  *   Add support for compatibility with redis-rb gem for 4.0 version.
      ([Pull Request](rails#30748))
  ```
  since it was backported to
  `5-1-stable` by 3789531.

- Remove
  ```
  *   Add `action_controller_api` and `action_controller_base` load hooks to be
      called in `ActiveSupport.on_load`.
      ([Pull Request](rails#28402))
  ```
  since it was backported to
  `5-1-stable` by b9a5fd7.

- Remove
  ```
  *   `driven_by` now registers poltergeist and capybara-webkit.
      ([Pull Request](rails#29315))
  ```
  since it was backported to
  `5-1-stable` by c5dd451.

- Remove
  ```
  *   Fallback `ActionController::Parameters#to_s` to `Hash#to_s`.
      ([Pull Request](rails#29630))
  ```
  since it was backported to
  `5-1-stable` by c1014e4
  and
  `5-0-stable` by 0e71fc3.

- Remove
  ```
  *   Make `take_failed_screenshot` work within engine.
      ([Pull Request](rails#30421))
  ```
  since it was backported to
  `5-1-stable` by 595a231.

- Remove
  ```
  *   Fix optimized url helpers when using relative url root.
      ([Pull Request](rails#31261))
  ```
  since it was backported to
  `5-1-stable` by e9b7795.

- Remove
  ```
  *   Update `distance_of_time_in_words` helper to display better error messages
      for bad input.
      ([Pull Request](rails#20701))
  ```
  since it was backported to
  `5-1-stable` by 2c97fbf.

- Remove
  ```
  *   Generate field ids in `collection_check_boxes` and
      `collection_radio_buttons`.
      ([Pull Request](rails#29412))
  ```
  since it was backported to
  `5-1-stable` by 2d8c10a.

- Remove
  ```
  *   Fix issues with scopes and engine on `current_page?` method.
      ([Pull Request](rails#29503))
  ```
  since it was backported to
  `5-1-stable` by 2135daf.

- Remove
  ```
  *   Bring back proc with arity of 1 in `ActionMailer::Base.default` proc
      since it was supported in Rails 5.0 but not deprecated.
      ([Pull Request](rails#30391))
  ```
  since it was backported to
  `5-1-stable` by b2bedb1.

- Remove
  ```
  *   Add type caster to `RuntimeReflection#alias_name`.
      ([Pull Request](rails#28961))
  ```
  since it was backported to
  `5-1-stable` by f644e7a.

- Remove
  ```
  *   Loading model schema from database is now thread-safe.
      ([Pull Request](rails#29216))
  ```
  since it was backported to
  `5-1-stable` by 02926cf.
  and
  `5-0-stable` by 84bcfe5

- Remove
  ```
  *   Fix destroying existing object does not work well when optimistic locking
      enabled and `locking_column` is null in the database.
      ([Pull Request](rails#28926))
  ```
  since it was backported to
  `5-1-stable` by e498052.

- Remove
  ```
  *   `ActiveRecord::Persistence#touch` does not work well
      when optimistic locking enabled and `locking_column`,
      without default value, is null in the database.
      ([Pull Request](rails#28914))
  ```
  since it was backported to
  `5-1-stable` by 1e2f63d.

- Remove
  ```
  *   Previously, when building records using a `has_many :through` association,
      if the child records were deleted before the parent was saved,
      they would still be persisted. Now, if child records are deleted
      before the parent is saved on a `has_many :through` association,
      the child records will not be persisted.
      ([Pull Request](rails#29593))
  ```
  since it was backported to
  `5-1-stable` by a22c39e.

- Remove
  ```
  *   Query cache was unavailable when entering the `ActiveRecord::Base.cache`
      block without being connected.
      ([Pull Request](rails#29609))
  ```
  since it was backported to
  `5-1-stable` by fd6c8cd
  and
  `5-0-stable` by 9f2532b.

- Remove
  ```
  *   `Relation#joins` is no longer affected by the target model's
      `current_scope`, with the exception of `unscoped`.
      ([Commit](rails@5c71000))
  ```
  since it was backported to
  `5-1-stable` by 3630d63.

- Remove
  ```
  *   Fix `unscoped(where: [columns])` removing the wrong bind values.
      ([Pull Request](rails#29780))
  ```
  since it was backported to
  `5-1-stable` by d378fcb.

- Remove
  ```
  *   When a `has_one` association is destroyed by `dependent: destroy`,
      `destroyed_by_association` will now be set to the reflection, matching the
      behaviour of `has_many` associations.
      ([Pull Request](rails#29855))
  ```
  since it was backported to
  `5-1-stable` by 8254a8b.

- Remove
  ```
  *   Fix `COUNT(DISTINCT ...)` with `ORDER BY` and `LIMIT`
      to keep the existing select list.
      ([Pull Request](rails#29848))
  ```
  since it was backported to
  `5-1-stable` by 0e8d4ed.

- Remove
  ```
  *   Ensure `sum` honors `distinct` on `has_many :through` associations.
      ([Commit](rails@566f1fd))
  ```
  since it was backported to
  `5-1-stable` by c0a1dc2.

- Remove
  ```
  *   Fix `COUNT(DISTINCT ...)` for `GROUP BY` with `ORDER BY` and `LIMIT`.
      ([Commit](rails@5668dc6))
  ```
  since it was backported to
  `5-1-stable` by 87ca68e.

- Remove
  ```
  *   MySQL: Don't lose `auto_increment: true` in the `db/schema.rb`.
      ([Commit](rails@9493d45))
  ```
  since it was backported to
  `5-1-stable` by 8b6e694.

- Remove
  ```
  *   Fix longer sequence name detection for serial columns.
      ([Pull Request](rails#28339))
  ```
  since it was backported to
  `5-1-stable` by af9c170
  and
  `5-0-stable` by 7025b1d.

- Remove
  ```
  *   Fix `bin/rails db:setup` and `bin/rails db:test:prepare` create wrong
      ar_internal_metadata's data for a test database.
      ([Pull Request](rails#30579))
  ```
  since it was backported to
  `5-1-stable` by bb67b5f
  and
  `5-0-stable` by 60437e6.

- Remove
  ```
  *   Fix conflicts `counter_cache` with `touch: true` by optimistic locking.
      ([Pull Request](rails#31405))
  ```
  since it was backported to
  `5-1-stable` by 5236dda.

- Remove
  ```
  *   Fix `count(:all)` to correctly work `distinct` with custom SELECT list.
      ([Commit](rails@c6cd9a5))
  ```
  since it was backported to
  `5-1-stable` by 6beb4de.

- Remove
  ```
  *   Fix to invoke callbacks when using `update_attribute`.
      ([Commit](rails@732aa34))
  ```
  since it was backported to
  `5-1-stable` by 6346683.

- Remove
  ```
  *   Use `count(:all)` in `HasManyAssociation#count_records` to prevent invalid
      SQL queries for association counting.
      ([Pull Request](rails#27561))
  ```
  since it was backported to
  `5-1-stable` by eef3c89.

- Remove
  ```
  *   Fix `count(:all)` with eager loading and having an order other than
      the driving table.
      ([Commit](rails@ebc09ed))
  ```
  since it was backported to
  `5-1-stable` by 6df9b69.

- Remove
  ```
  *   PostgreSQL: Allow pg-1.0 gem to be used with Active Record.
      ([Pull Request](rails#31671))
  ```
  since it was backported to
  `5-1-stable` by a9c06f6.

- Remove
  ```
  *   Fix that after commit callbacks on update does not triggered
      when optimistic locking is enabled.
      ([Commit](rails@7f9bd03))
  ```
  since it was backported to
  `5-1-stable` by aaee10e.

- Remove
  ```
  *   Fix regression in numericality validator when comparing Decimal and Float
      input values with more scale than the schema.
      ([Pull Request](rails#28584))
  ```
  since it was backported to
  `5-1-stable` by 5b1c3e5.
  Note that there was incorrect link to PR,
  original PR is rails#29249.

- Remove
  ```
  *   Fix to working before/after validation callbacks on multiple contexts.
      ([Pull Request](rails#31483))
  ```
  since it was backported to
  `5-1-stable` by 0f7046a.

- Remove
  ```
  *   Fix implicit coercion calculations with scalars and durations.
      ([Pull Request](rails#29163),
      [Pull Request](rails#29971))
  ```
  since it was backported to
  `5-1-stable` by 51ea27c,
                  4d82e2a.

- Remove
  ```
  *   Fix modulo operations involving durations.
      ([Commit](rails@a54e13b))
  ```
  since it was backported to
  `5-1-stable` by 233fa7e.

- Remove
  ```
  *   Return all mappings for a timezone identifier in `country_zones`.
      ([Commit](rails@cdce6a7))
  ```
  since it was backported to
  `5-1-stable` by 0222ebb.

- Remove
  ```
  *   Add support for compatibility with redis-rb gem for 4.0 version.
      ([Pull Request](rails#30748))
  ```
  since it was backported to
  `5-1-stable` by 3789531.
  Related to rails#32252.

Related to rails#32222, rails#32222 (comment).
Follow up a489cc8.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants