New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add `config.active_record.warn_on_records_fetched_greater_than` option #18846

Merged
merged 1 commit into from Mar 26, 2015

Conversation

Projects
None yet
4 participants
@hundredwatt

hundredwatt commented Feb 7, 2015

When set to an integer, a warning will be logged whenever a result set
larger than the specified size is returned by a query. Fixes #16463

@kaspth

View changes

Show outdated Hide outdated activerecord/lib/active_record/core.rb
@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt commented Feb 7, 2015

@kaspth ✂️-ed

@kaspth kaspth added the activerecord label Feb 7, 2015

@kaspth kaspth added this to the 5.0.0 milestone Feb 7, 2015

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 8, 2015

Wondering if we should move the exec_queries method to a module and then use inheritance to add the warning only if the config value is set. That way, this would have 0 cost when not enabled.

hundredwatt commented Feb 8, 2015

Wondering if we should move the exec_queries method to a module and then use inheritance to add the warning only if the config value is set. That way, this would have 0 cost when not enabled.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 8, 2015

Member

I don't think you need to touch the existing #exec_queries method. You could try the module approach using #prepend.

Unless #prepend is frowned upon in Rails, which I need someone else to verify.

Member

kaspth commented Feb 8, 2015

I don't think you need to touch the existing #exec_queries method. You could try the module approach using #prepend.

Unless #prepend is frowned upon in Rails, which I need someone else to verify.

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 8, 2015

I had the same thought, however there's no other use of Module#prepend in AR. I believe the idea was to start allowing it for Rails 5, when backwards compatibility for 1.9 will be dropped.

hundredwatt commented Feb 8, 2015

I had the same thought, however there's no other use of Module#prepend in AR. I believe the idea was to start allowing it for Rails 5, when backwards compatibility for 1.9 will be dropped.

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 8, 2015

Gave Module#prepend a try with c5751fe. This will fail CI, but I'm curious to get some feedback on the approach.

hundredwatt commented Feb 8, 2015

Gave Module#prepend a try with c5751fe. This will fail CI, but I'm curious to get some feedback on the approach.

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 8, 2015

Nevermind comment about CI, travis-ci matrix is only Ruby 2.2 now.

hundredwatt commented Feb 8, 2015

Nevermind comment about CI, travis-ci matrix is only Ruby 2.2 now.

@kaspth

View changes

Show outdated Hide outdated activerecord/test/cases/relation/warn_on_result_set_size_test.rb
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 8, 2015

Member

I think you need to rebase this on master before Travis will try a build.

Member

kaspth commented Feb 8, 2015

I think you need to rebase this on master before Travis will try a build.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 8, 2015

Member

Otherwise the code looks good to me 👍. It depends on whether others are okay with the initializer approach here.

Member

kaspth commented Feb 8, 2015

Otherwise the code looks good to me 👍. It depends on whether others are okay with the initializer approach here.

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 8, 2015

Thanks for feedback @kaspth 😄. Will await additional feedback and then can squash before merging.

hundredwatt commented Feb 8, 2015

Thanks for feedback @kaspth 😄. Will await additional feedback and then can squash before merging.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 8, 2015

Member

Sure thing 😄

Your test leaks state. The module you prepend is still in Relation after the test, so all the other tests fail because you're comparing x >= nil where x is an integer.

I don't know if unprepending a module is considered a dirty practice, so let's wait for others to chime in 😁

Member

kaspth commented Feb 8, 2015

Sure thing 😄

Your test leaks state. The module you prepend is still in Relation after the test, so all the other tests fail because you're comparing x >= nil where x is an integer.

I don't know if unprepending a module is considered a dirty practice, so let's wait for others to chime in 😁

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 8, 2015

Member

What you did just now is probably fine too :)

Member

kaspth commented Feb 8, 2015

What you did just now is probably fine too :)

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt commented Feb 8, 2015

@kaspth 👍

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 8, 2015

Just patched Delayed Job in a production Rails app with this warning and set up a log alert. Already found an instance of Model.each that should be Model.find_each 😄:

Feb 08 15:00:44 GaggleAMP: Result set size exceeded 1001 (model: Authentication::GoogleAuthentication, records: 1507)

hundredwatt commented Feb 8, 2015

Just patched Delayed Job in a production Rails app with this warning and set up a log alert. Already found an instance of Model.each that should be Model.find_each 😄:

Feb 08 15:00:44 GaggleAMP: Result set size exceeded 1001 (model: Authentication::GoogleAuthentication, records: 1507)
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 8, 2015

Member

Cool! I think the error message needs a little tweaking to read a bit more naturally.

Would it be of use to broadcast this as an Active Support notification too?

Member

kaspth commented Feb 8, 2015

Cool! I think the error message needs a little tweaking to read a bit more naturally.

Would it be of use to broadcast this as an Active Support notification too?

@rafaelfranca

View changes

Show outdated Hide outdated activerecord/lib/active_record/railtie.rb
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 10, 2015

Member

I've been thinking about the config option name and result_set doesn't fit with Active Record's terminology.
What about something like warn_on_fetches_above or warn_on_record_fetches_greater_than?

Member

kaspth commented Feb 10, 2015

I've been thinking about the config option name and result_set doesn't fit with Active Record's terminology.
What about something like warn_on_fetches_above or warn_on_record_fetches_greater_than?

@kaspth

View changes

Show outdated Hide outdated activerecord/lib/active_record/relation/warn_on_result_set_size.rb
@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 11, 2015

@kaspth I like where you're going. How about:

For the config setting: config.active_record.warn_on_records_fetched_greater_than

For the message:

"Large query detected. Exceeded #{warn_on_result_set_size} threshold by fetching #{@records.size} #{@klass} records."

For the module, maybe we can call it LargeQueryDetection instead of WarnOnResultSetSize?

hundredwatt commented Feb 11, 2015

@kaspth I like where you're going. How about:

For the config setting: config.active_record.warn_on_records_fetched_greater_than

For the message:

"Large query detected. Exceeded #{warn_on_result_set_size} threshold by fetching #{@records.size} #{@klass} records."

For the module, maybe we can call it LargeQueryDetection instead of WarnOnResultSetSize?

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 12, 2015

Member

I like your suggestions way better than warn_on_result_set_size 😄

I have some more thoughts about the warning message.

There's a part of me that feels fetching is to lightweight a word. Is there a word that better fits we're dealing with a large query?

I think we should explain the threshold just a bit more. Something like this reads a bit broken:
"Large query detected. Exceeded 1001 threshold by fetching 1200 Kitten records."

Member

kaspth commented Feb 12, 2015

I like your suggestions way better than warn_on_result_set_size 😄

I have some more thoughts about the warning message.

There's a part of me that feels fetching is to lightweight a word. Is there a word that better fits we're dealing with a large query?

I think we should explain the threshold just a bit more. Something like this reads a bit broken:
"Large query detected. Exceeded 1001 threshold by fetching 1200 Kitten records."

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 13, 2015

Agreed, I'm struggling with how to phrase this 😄. Here's a number of ideas:

"Query returned large result set. Warning threshold of #{warn_on_result_set_size} exceeded by fetching #{@records.size} #{@klass} records."
"Query fetched #{@records.size} #{@klass} records, exceeding threshold of {warn_on_result_set_size}."
"Query fetched large number of records:  #{@records.size} #{@klass} records (exceeded threshold of #{warn_on_result_set_size})."

Alternatively, do we need to re-state the configured value by including #{warn_on_result_set_size}? Since its explicitly set I'm not sure we need to repeat in the warning message it:

"Query fetched #{@records.size} #{@klass} records"

or maybe have a recommendation?:

"Query fetched #{@records.size} #{@klass} records. Please use batch finder methods to limit memory consumption."

Do any of those options resonate with you?

hundredwatt commented Feb 13, 2015

Agreed, I'm struggling with how to phrase this 😄. Here's a number of ideas:

"Query returned large result set. Warning threshold of #{warn_on_result_set_size} exceeded by fetching #{@records.size} #{@klass} records."
"Query fetched #{@records.size} #{@klass} records, exceeding threshold of {warn_on_result_set_size}."
"Query fetched large number of records:  #{@records.size} #{@klass} records (exceeded threshold of #{warn_on_result_set_size})."

Alternatively, do we need to re-state the configured value by including #{warn_on_result_set_size}? Since its explicitly set I'm not sure we need to repeat in the warning message it:

"Query fetched #{@records.size} #{@klass} records"

or maybe have a recommendation?:

"Query fetched #{@records.size} #{@klass} records. Please use batch finder methods to limit memory consumption."

Do any of those options resonate with you?

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 13, 2015

Additionally, we could probably use AS::Notifications to capture the SQL query itself and include it in the warning. I'll play with that.

hundredwatt commented Feb 13, 2015

Additionally, we could probably use AS::Notifications to capture the SQL query itself and include it in the warning. I'll play with that.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 13, 2015

Member

I think you're right about not mentioning the threshold.

👍 on option 4. Though I miss the "Large query." from the earlier revisions.

For me option 5 depends on how people use this API. If they explicitly set the threshold, aren't they hunting for large queries and know how to handle them? Alternatively, I think we'd be better off linking to the batch APIs in when documenting the threshold and explaining its use case.

We're still renaming the config option, right?

Do play around with the notifications 👍

Member

kaspth commented Feb 13, 2015

I think you're right about not mentioning the threshold.

👍 on option 4. Though I miss the "Large query." from the earlier revisions.

For me option 5 depends on how people use this API. If they explicitly set the threshold, aren't they hunting for large queries and know how to handle them? Alternatively, I think we'd be better off linking to the batch APIs in when documenting the threshold and explaining its use case.

We're still renaming the config option, right?

Do play around with the notifications 👍

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 14, 2015

Ok, I made the name and phrase changes.

Also, @kaspth nice job on this weeks Rails Weekly 😄 ❤️

hundredwatt commented Feb 14, 2015

Ok, I made the name and phrase changes.

Also, @kaspth nice job on this weeks Rails Weekly 😄 ❤️

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Feb 14, 2015

Member

Well done and thanks for the kind words 👍

I'm going to need someone else to look through the notification stuff, so feel free to jump in.

Member

kaspth commented Feb 14, 2015

Well done and thanks for the kind words 👍

I'm going to need someone else to look through the notification stuff, so feel free to jump in.

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Feb 26, 2015

I squashed my previous commits and fixed the documentation as per @fxn's recommendation

hundredwatt commented Feb 26, 2015

I squashed my previous commits and fixed the documentation as per @fxn's recommendation

# When this module is prepended to ActiveRecord::Relation and
# `config.active_record.warn_on_records_fetched_greater_than` is
# set to an integer, if the number of records a query returns is
# greater than the value of `warn_on_records_fetched_greater_than`,

This comment has been minimized.

@kaspth

kaspth Feb 26, 2015

Member

Your documentation says greater than but the check is >= 😉

@kaspth

kaspth Feb 26, 2015

Member

Your documentation says greater than but the check is >= 😉

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Mar 25, 2015

Member

@hundredwatt hey, completely forgot about this until today 😄

Can you rebase and squash your commits? Then this gets my 👍

Great work, ❤️

Member

kaspth commented Mar 25, 2015

@hundredwatt hey, completely forgot about this until today 😄

Can you rebase and squash your commits? Then this gets my 👍

Great work, ❤️

Add `config.active_record.warn_on_records_fetched_greater_than` option
When set to an integer, a warning will be logged whenever a result set
larger than the specified size is returned by a query. Fixes #16463

The warning is outputed a module which is prepended in an initializer,
so there will be no performance impact if
`config.active_record.warn_on_records_fetched_greater_than` is not set.
@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Mar 25, 2015

@kaspth No worries, I know there's a lot of other PRs out there 😉

I rebased/squashed, but the Travis build didn't kick off. Any ideas?

hundredwatt commented Mar 25, 2015

@kaspth No worries, I know there's a lot of other PRs out there 😉

I rebased/squashed, but the Travis build didn't kick off. Any ideas?

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Mar 25, 2015

@kaspth Nevermind, its building now

hundredwatt commented Mar 25, 2015

@kaspth Nevermind, its building now

payload = args.last
QueryRegistry.queries << payload[:sql]
end

This comment has been minimized.

@kaspth

kaspth Mar 25, 2015

Member

Could accomplish the same thing with:

ActiveSupport::Notifications.subscribe("sql.active_record") do |*, payload|
  QueryRegistry.queries << payload[:sql]
end
@kaspth

kaspth Mar 25, 2015

Member

Could accomplish the same thing with:

ActiveSupport::Notifications.subscribe("sql.active_record") do |*, payload|
  QueryRegistry.queries << payload[:sql]
end
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Mar 25, 2015

Member

The build passed, but I don't know why GitHub didn't signal anything here.

Member

kaspth commented Mar 25, 2015

The build passed, but I don't know why GitHub didn't signal anything here.

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Mar 25, 2015

The build passed, but I don't know why GitHub didn't signal anything here.

¯_(ツ)_/¯

hundredwatt commented Mar 25, 2015

The build passed, but I don't know why GitHub didn't signal anything here.

¯_(ツ)_/¯

rafaelfranca added a commit that referenced this pull request Mar 26, 2015

Merge pull request #18846 from hundredwatt/feat/warn-on-result-set-size
Add `config.active_record.warn_on_result_set_size` option

@rafaelfranca rafaelfranca merged commit 8b451e3 into rails:master Mar 26, 2015

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth
Member

kaspth commented Mar 26, 2015

@kaspth kaspth changed the title from Add `config.active_record.warn_on_result_set_size` option to Add `config.active_record.warn_on_records_fetched_greater_than` option Mar 26, 2015

jonatack added a commit to jonatack/rails that referenced this pull request Mar 27, 2015

[skip ci] Update configuring.md with #18846
Add `config.active_record.warn_on_records_fetched_greater_than` to the
Configuring Rails Guide.

schneems added a commit that referenced this pull request Mar 27, 2015

@hundredwatt hundredwatt deleted the hundredwatt:feat/warn-on-result-set-size branch Mar 27, 2015

@hundredwatt

This comment has been minimized.

Show comment
Hide comment
@hundredwatt

hundredwatt Mar 27, 2015

@kaspth @rafaelfranca Thanks so much for your feedback! 😄 💙

hundredwatt commented Mar 27, 2015

@kaspth @rafaelfranca Thanks so much for your feedback! 😄 💙

# `config.active_record.warn_on_records_fetched_greater_than` is
# set to an integer, if the number of records a query returns is
# greater than the value of `warn_on_records_fetched_greater_than`,
# a warning is logged. This allows for the dection of queries that

This comment has been minimized.

@aripollak

aripollak Apr 6, 2015

Contributor

s/dection/detection/

@aripollak

aripollak Apr 6, 2015

Contributor

s/dection/detection/

This comment has been minimized.

@kaspth

kaspth Apr 6, 2015

Member

This change was merged a while ago and after looking at the master branch I see it's already fixed: fe6de0f 😁

@kaspth

kaspth Apr 6, 2015

Member

This change was merged a while ago and after looking at the master branch I see it's already fixed: fe6de0f 😁

@rafaelfranca rafaelfranca modified the milestones: 5.0.0 [temp], 5.0.0 Dec 30, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment