Add config.active_record.warn_on_records_fetched_greater_than option

[hundredwatt]

When set to an integer, a warning will be logged whenever a result set
larger than the specified size is returned by a query. Fixes #16463

[kaspth]

✂️ this line

[hundredwatt]

@kaspth ✂️-ed

[hundredwatt]

Wondering if we should move the exec_queries method to a module and then use inheritance to add the warning only if the config value is set. That way, this would have 0 cost when not enabled.

[kaspth]

I don't think you need to touch the existing #exec_queries method. You could try the module approach using #prepend.

Unless #prepend is frowned upon in Rails, which I need someone else to verify.

[hundredwatt]

I had the same thought, however there's no other use of Module#prepend in AR. I believe the idea was to start allowing it for Rails 5, when backwards compatibility for 1.9 will be dropped.

[hundredwatt]

Gave Module#prepend a try with c5751fe. This will fail CI, but I'm curious to get some feedback on the approach.

[hundredwatt]

Nevermind comment about CI, travis-ci matrix is only Ruby 2.2 now.

[kaspth]

Can we remove the requires for the other models?

[kaspth]

I think you need to rebase this on master before Travis will try a build.

[kaspth]

Otherwise the code looks good to me 👍. It depends on whether others are okay with the initializer approach here.

[hundredwatt]

Thanks for feedback @kaspth 😄. Will await additional feedback and then can squash before merging.

[kaspth]

Sure thing 😄

Your test leaks state. The module you prepend is still in Relation after the test, so all the other tests fail because you're comparing x >= nil where x is an integer.

I don't know if unprepending a module is considered a dirty practice, so let's wait for others to chime in 😁

[kaspth]

What you did just now is probably fine too :)

[hundredwatt]

@kaspth 👍

[hundredwatt]

Just patched Delayed Job in a production Rails app with this warning and set up a log alert. Already found an instance of Model.each that should be Model.find_each 😄:

Feb 08 15:00:44 GaggleAMP: Result set size exceeded 1001 (model: Authentication::GoogleAuthentication, records: 1507)

[kaspth]

Cool! I think the error message needs a little tweaking to read a bit more naturally.

Would it be of use to broadcast this as an Active Support notification too?

[rafaelfranca]

Why we need the after_initialize block?

[hundredwatt]

It's not needed, its only there because I copied and pasted from the previous initializer, doh! Will remove

[kaspth]

I've been thinking about the config option name and result_set doesn't fit with Active Record's terminology.
What about something like warn_on_fetches_above or warn_on_record_fetches_greater_than?

[kaspth]

What about changing the message to something like:

"Large query detected. Exceeded #{warn_on_result_set_size} threshold by loading #{@records.size} #{@klass} records."

[hundredwatt]

@kaspth I like where you're going. How about:

For the config setting: config.active_record.warn_on_records_fetched_greater_than

For the message:

"Large query detected. Exceeded #{warn_on_result_set_size} threshold by fetching #{@records.size} #{@klass} records."

For the module, maybe we can call it LargeQueryDetection instead of WarnOnResultSetSize?

[kaspth]

I like your suggestions way better than warn_on_result_set_size 😄

I have some more thoughts about the warning message.

There's a part of me that feels fetching is to lightweight a word. Is there a word that better fits we're dealing with a large query?

I think we should explain the threshold just a bit more. Something like this reads a bit broken:
"Large query detected. Exceeded 1001 threshold by fetching 1200 Kitten records."

[hundredwatt]

Agreed, I'm struggling with how to phrase this 😄. Here's a number of ideas:

"Query returned large result set. Warning threshold of #{warn_on_result_set_size} exceeded by fetching #{@records.size} #{@klass} records."

"Query fetched #{@records.size} #{@klass} records, exceeding threshold of {warn_on_result_set_size}."

"Query fetched large number of records:  #{@records.size} #{@klass} records (exceeded threshold of #{warn_on_result_set_size})."

Alternatively, do we need to re-state the configured value by including #{warn_on_result_set_size}? Since its explicitly set I'm not sure we need to repeat in the warning message it:

"Query fetched #{@records.size} #{@klass} records"

or maybe have a recommendation?:

"Query fetched #{@records.size} #{@klass} records. Please use batch finder methods to limit memory consumption."

Do any of those options resonate with you?

[hundredwatt]

Additionally, we could probably use AS::Notifications to capture the SQL query itself and include it in the warning. I'll play with that.

[kaspth]

Well done and thanks for the kind words 👍

I'm going to need someone else to look through the notification stuff, so feel free to jump in.

[hundredwatt]

I squashed my previous commits and fixed the documentation as per @fxn's recommendation

[kaspth]

Your documentation says greater than but the check is >= 😉

[kaspth]

@hundredwatt hey, completely forgot about this until today 😄

Can you rebase and squash your commits? Then this gets my 👍

Great work, ❤️

[hundredwatt]

@kaspth No worries, I know there's a lot of other PRs out there 😉

I rebased/squashed, but the Travis build didn't kick off. Any ideas?

[hundredwatt]

@kaspth Nevermind, its building now

[kaspth]

Could accomplish the same thing with:

ActiveSupport::Notifications.subscribe("sql.active_record") do |*, payload|
  QueryRegistry.queries << payload[:sql]
end

[kaspth]

The build passed, but I don't know why GitHub didn't signal anything here.

[hundredwatt]

The build passed, but I don't know why GitHub didn't signal anything here.

¯_(ツ)_/¯

[kaspth]

@hundredwatt @rafaelfranca ❤️

[hundredwatt]

@kaspth @rafaelfranca Thanks so much for your feedback! 😄 💙

[aripollak]

s/dection/detection/

[kaspth]

This change was merged a while ago and after looking at the master branch I see it's already fixed: fe6de0f 😁