Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add `config.active_record.warn_on_records_fetched_greater_than` option #18846

Merged
merged 1 commit into from Mar 26, 2015

Conversation

@hundredwatt
Copy link

@hundredwatt hundredwatt commented Feb 7, 2015

When set to an integer, a warning will be logged whenever a result set
larger than the specified size is returned by a query. Fixes #16463

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch Feb 7, 2015
@kaspth
kaspth reviewed Feb 7, 2015
View changes
activerecord/lib/active_record/core.rb Outdated
mattr_accessor :warn_on_result_set_size, instance_writer: false
self.warn_on_result_set_size = nil


This comment has been minimized.

@kaspth

kaspth Feb 7, 2015
Member

✂️ this line

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch Feb 7, 2015
@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 7, 2015

@kaspth ✂️-ed

@kaspth kaspth added the activerecord label Feb 7, 2015
@kaspth kaspth added this to the 5.0.0 milestone Feb 7, 2015
@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 8, 2015

Wondering if we should move the exec_queries method to a module and then use inheritance to add the warning only if the config value is set. That way, this would have 0 cost when not enabled.

@kaspth
Copy link
Member

@kaspth kaspth commented Feb 8, 2015

I don't think you need to touch the existing #exec_queries method. You could try the module approach using #prepend.

Unless #prepend is frowned upon in Rails, which I need someone else to verify.

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 8, 2015

I had the same thought, however there's no other use of Module#prepend in AR. I believe the idea was to start allowing it for Rails 5, when backwards compatibility for 1.9 will be dropped.

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 8, 2015

Gave Module#prepend a try with c5751fe. This will fail CI, but I'm curious to get some feedback on the approach.

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 8, 2015

Nevermind comment about CI, travis-ci matrix is only Ruby 2.2 now.

@kaspth
kaspth reviewed Feb 8, 2015
View changes
activerecord/test/cases/relation/warn_on_result_set_size_test.rb Outdated
require 'models/post'
require 'models/comment'
require 'models/author'
require 'models/rating'

This comment has been minimized.

@kaspth

kaspth Feb 8, 2015
Member

Can we remove the requires for the other models?

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch Feb 8, 2015
@kaspth
Copy link
Member

@kaspth kaspth commented Feb 8, 2015

I think you need to rebase this on master before Travis will try a build.

@kaspth
Copy link
Member

@kaspth kaspth commented Feb 8, 2015

Otherwise the code looks good to me 👍. It depends on whether others are okay with the initializer approach here.

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch Feb 8, 2015
@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 8, 2015

Thanks for feedback @kaspth 😄. Will await additional feedback and then can squash before merging.

@kaspth
Copy link
Member

@kaspth kaspth commented Feb 8, 2015

Sure thing 😄

Your test leaks state. The module you prepend is still in Relation after the test, so all the other tests fail because you're comparing x >= nil where x is an integer.

I don't know if unprepending a module is considered a dirty practice, so let's wait for others to chime in 😁

@kaspth
Copy link
Member

@kaspth kaspth commented Feb 8, 2015

What you did just now is probably fine too :)

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 8, 2015

@kaspth 👍

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 8, 2015

Just patched Delayed Job in a production Rails app with this warning and set up a log alert. Already found an instance of Model.each that should be Model.find_each 😄:

Feb 08 15:00:44 GaggleAMP: Result set size exceeded 1001 (model: Authentication::GoogleAuthentication, records: 1507)
@kaspth
Copy link
Member

@kaspth kaspth commented Feb 8, 2015

Cool! I think the error message needs a little tweaking to read a bit more naturally.

Would it be of use to broadcast this as an Active Support notification too?

@rafaelfranca
rafaelfranca reviewed Feb 10, 2015
View changes
activerecord/lib/active_record/railtie.rb Outdated
@@ -102,6 +102,17 @@ class Railtie < Rails::Railtie # :nodoc:
end
end

initializer "active_record.warn_on_result_set_size" do
if config.active_record.warn_on_result_set_size
config.after_initialize do |app|

This comment has been minimized.

@rafaelfranca

rafaelfranca Feb 10, 2015
Member

Why we need the after_initialize block?

This comment has been minimized.

@hundredwatt

hundredwatt Feb 10, 2015
Author

It's not needed, its only there because I copied and pasted from the previous initializer, doh! Will remove

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch Feb 10, 2015
@kaspth
Copy link
Member

@kaspth kaspth commented Feb 10, 2015

I've been thinking about the config option name and result_set doesn't fit with Active Record's terminology.
What about something like warn_on_fetches_above or warn_on_record_fetches_greater_than?

@kaspth
kaspth reviewed Feb 10, 2015
View changes
activerecord/lib/active_record/relation/warn_on_result_set_size.rb Outdated
super.tap do
if logger && warn_on_result_set_size
if @records.length >= warn_on_result_set_size
logger.warn "Result set size exceeded #{warn_on_result_set_size} (model: #{@klass}, records: #{@records.size})"

This comment has been minimized.

@kaspth

kaspth Feb 10, 2015
Member

What about changing the message to something like:

"Large query detected. Exceeded #{warn_on_result_set_size} threshold by loading #{@records.size} #{@klass} records."
@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 11, 2015

@kaspth I like where you're going. How about:

For the config setting: config.active_record.warn_on_records_fetched_greater_than

For the message:

"Large query detected. Exceeded #{warn_on_result_set_size} threshold by fetching #{@records.size} #{@klass} records."

For the module, maybe we can call it LargeQueryDetection instead of WarnOnResultSetSize?

@kaspth
Copy link
Member

@kaspth kaspth commented Feb 12, 2015

I like your suggestions way better than warn_on_result_set_size 😄

I have some more thoughts about the warning message.

There's a part of me that feels fetching is to lightweight a word. Is there a word that better fits we're dealing with a large query?

I think we should explain the threshold just a bit more. Something like this reads a bit broken:
"Large query detected. Exceeded 1001 threshold by fetching 1200 Kitten records."

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 13, 2015

Agreed, I'm struggling with how to phrase this 😄. Here's a number of ideas:

"Query returned large result set. Warning threshold of #{warn_on_result_set_size} exceeded by fetching #{@records.size} #{@klass} records."
"Query fetched #{@records.size} #{@klass} records, exceeding threshold of {warn_on_result_set_size}."
"Query fetched large number of records:  #{@records.size} #{@klass} records (exceeded threshold of #{warn_on_result_set_size})."

Alternatively, do we need to re-state the configured value by including #{warn_on_result_set_size}? Since its explicitly set I'm not sure we need to repeat in the warning message it:

"Query fetched #{@records.size} #{@klass} records"

or maybe have a recommendation?:

"Query fetched #{@records.size} #{@klass} records. Please use batch finder methods to limit memory consumption."

Do any of those options resonate with you?

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 13, 2015

Additionally, we could probably use AS::Notifications to capture the SQL query itself and include it in the warning. I'll play with that.

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 14, 2015

Ok, I made the name and phrase changes.

Also, @kaspth nice job on this weeks Rails Weekly 😄 ❤️

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch Feb 14, 2015
@kaspth
Copy link
Member

@kaspth kaspth commented Feb 14, 2015

Well done and thanks for the kind words 👍

I'm going to need someone else to look through the notification stuff, so feel free to jump in.

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch Feb 26, 2015
@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Feb 26, 2015

I squashed my previous commits and fixed the documentation as per @fxn's recommendation

# When this module is prepended to ActiveRecord::Relation and
# `config.active_record.warn_on_records_fetched_greater_than` is
# set to an integer, if the number of records a query returns is
# greater than the value of `warn_on_records_fetched_greater_than`,

This comment has been minimized.

@kaspth

kaspth Feb 26, 2015
Member

Your documentation says greater than but the check is >= 😉

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch to 3c6675c Mar 16, 2015
@kaspth
Copy link
Member

@kaspth kaspth commented Mar 25, 2015

@hundredwatt hey, completely forgot about this until today 😄

Can you rebase and squash your commits? Then this gets my 👍

Great work, ❤️

@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch from 3c6675c to 613abe2 Mar 25, 2015
When set to an integer, a warning will be logged whenever a result set
larger than the specified size is returned by a query. Fixes #16463

The warning is outputed a module which is prepended in an initializer,
so there will be no performance impact if
`config.active_record.warn_on_records_fetched_greater_than` is not set.
@hundredwatt hundredwatt force-pushed the hundredwatt:feat/warn-on-result-set-size branch from 613abe2 to 4d6fbe2 Mar 25, 2015
@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Mar 25, 2015

@kaspth No worries, I know there's a lot of other PRs out there 😉

I rebased/squashed, but the Travis build didn't kick off. Any ideas?

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Mar 25, 2015

@kaspth Nevermind, its building now

payload = args.last

QueryRegistry.queries << payload[:sql]
end

This comment has been minimized.

@kaspth

kaspth Mar 25, 2015
Member

Could accomplish the same thing with:

ActiveSupport::Notifications.subscribe("sql.active_record") do |*, payload|
  QueryRegistry.queries << payload[:sql]
end
@kaspth
Copy link
Member

@kaspth kaspth commented Mar 25, 2015

The build passed, but I don't know why GitHub didn't signal anything here.

@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Mar 25, 2015

The build passed, but I don't know why GitHub didn't signal anything here.

¯_(ツ)_/¯

rafaelfranca added a commit that referenced this pull request Mar 26, 2015
Add `config.active_record.warn_on_result_set_size` option
@rafaelfranca rafaelfranca merged commit 8b451e3 into rails:master Mar 26, 2015
@kaspth
Copy link
Member

@kaspth kaspth commented Mar 26, 2015

@kaspth kaspth changed the title Add `config.active_record.warn_on_result_set_size` option Add `config.active_record.warn_on_records_fetched_greater_than` option Mar 26, 2015
jonatack added a commit to jonatack/rails that referenced this pull request Mar 27, 2015
Add `config.active_record.warn_on_records_fetched_greater_than` to the
Configuring Rails Guide.
schneems added a commit that referenced this pull request Mar 27, 2015
@hundredwatt hundredwatt deleted the hundredwatt:feat/warn-on-result-set-size branch Mar 27, 2015
@hundredwatt
Copy link
Author

@hundredwatt hundredwatt commented Mar 27, 2015

@kaspth @rafaelfranca Thanks so much for your feedback! 😄 💙

# `config.active_record.warn_on_records_fetched_greater_than` is
# set to an integer, if the number of records a query returns is
# greater than the value of `warn_on_records_fetched_greater_than`,
# a warning is logged. This allows for the dection of queries that

This comment has been minimized.

@aripollak

aripollak Apr 6, 2015
Contributor

s/dection/detection/

This comment has been minimized.

@kaspth

kaspth Apr 6, 2015
Member

This change was merged a while ago and after looking at the master branch I see it's already fixed: fe6de0f 😁

@rafaelfranca rafaelfranca modified the milestones: 5.0.0 [temp], 5.0.0 Dec 30, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

4 participants
You can’t perform that action at this time.