Idempotent option for after_commit :destroy callback #27248

stefanmb · 2016-12-02T00:53:33Z

Summary

In SQL a DELETE statement is always idempotent (will always succeed even if the underlying table is only modified once). Consequently the after_commit callbacks on :destroywill be invoked multiple times for a given model, even if a single deletion occurred. This means that user handlers must also be idempotent, which is not always the case (for example, if counting resources).

The issue addressed in this PR is related to #14735 and is fixed in a similar fashion:

Introduce an optional :idempotent flag for the after_commit callback.
Default to true to preserve existing behaviour.
Cache whether the operation affected any rows and provide an actually_destroyed? getter.
Demonstrate problem and solution with new unit tests.

We (Shopify) have been running a monkey patched version of this PR since October in production with no ill-effects.

r: @byroot @sgrif
cc: @tenderlove

P.S. This is my first Rails PR, so if there are any silly issues or violated conventions please let me know and I will resolve them as soon as possible. Thank you!

rails-bot · 2016-12-02T00:53:36Z

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @sgrif (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

byroot

Looks good to me overall (for what it's worth)

byroot · 2016-12-02T07:44:41Z

activerecord/test/cases/transaction_callbacks_test.rb

+    assert_equal [:destroy, :destroy], TopicWithIdempotentCallbacksOnDestroy.history
+  end
+
+  def test_helper(klass)


I would name that differently. Something like: simulate_race_condition or something like that.

Oops, you're right. The test_ naming scheme is not good for helper (gets run as a test standalone). I've changed it as you suggested.

byroot · 2016-12-02T07:47:31Z

Looks like your tests don't pass either.

stefanmb · 2016-12-02T15:12:11Z

CI is green, sorry about the test mess. Thanks!

byroot · 2016-12-02T17:08:35Z

👏

stefanmb · 2016-12-06T18:05:19Z

@rafaelfranca Pinging you since you have some context on this issue.

rafaelfranca · 2016-12-06T18:06:30Z

ACK! I have some things to review now but I put this in the top of my list.

sgrif · 2016-12-06T19:54:10Z

I don't think we need to make this an option. after_commit :stuff, on: :destroy running multiple times sounds like a bug, not behavior we need to preserve.

rafaelfranca · 2016-12-06T20:10:58Z

Ah yeah, we should make this behavior the default with no opt-out as we discussed in the original implementation.

matthewd · 2016-12-06T20:31:39Z

activerecord/lib/active_record/persistence.rb

@@ -525,6 +529,12 @@ def touch(*names, time: nil)
      end
    end

+  protected
+
+    def actually_destroyed?


This looks like it can be private

kaspth · 2016-12-06T20:37:23Z

activerecord/lib/active_record/transactions.rb

+              destroyed?
+            else
+              actually_destroyed?
+            end
          when :update
            !(transaction_record_state(:new_record) || destroyed?)


Why won't this destroyed? need to check actually_destroyed??

kaspth · 2016-12-06T20:40:26Z

activerecord/lib/active_record/transactions.rb

@@ -292,9 +292,10 @@ def set_options_for_callbacks!(args, enforced_options = {})

          if options[:on]
            fire_on = Array(options[:on])
+            idempotent = options[:idempotent]


Perhaps it's just because I'm not a native english speaker, but idempotent sounds needlessly technical to me. Like we're barfing out SQL intricacies to end users.

It could just be an unfamiliarity with the word, though I've been trying to read it several times and it still doesn't click.

consider_succeeded: true, might be closer to the SQL-forces-us-to-consider-this-final aspect, though still needs finagling.

kaspth · 2016-12-06T20:46:36Z

activerecord/lib/active_record/persistence.rb

+  protected
+
+    def actually_destroyed?
+      @_actually_destroyed ||= false


It might make sense to clarify @destroyed to e.g. @surreptitiously_destroyed so it doesn't fight for the same meaning as @_actually_destroyed.

We're going to get rid of the method and option so it's all good. :)

matthewd · 2016-12-06T21:54:52Z

@sgrif after_commit :update will still get run even if the update didn't change anything...

stefanmb · 2016-12-07T21:01:44Z

@rafaelfranca @sgrif I've eliminated the idempotent option and made the new callback behaviour the default, as requested.

sgrif · 2016-12-07T21:49:59Z

activerecord/lib/active_record/persistence.rb

@@ -181,7 +181,11 @@ def destroy
      _raise_readonly_record_error if readonly?
      destroy_associations
      self.class.connection.add_transaction_record(self)
-      destroy_row if persisted?
+      @_actually_destroyed = if persisted?


Can this just be changed to

if persisted && !destroyed @destroyed = destroy_row > 0 end

Do we even need to check the return value? Is this really different than destroy_row if persisted && !destroyed?

The issue is somehow flagging whether the SQL DELETE actually affected any rows (so we can trigger the callback then and only then). To do so we need to store the result of destroy_row somewhere. Right now we have:

def persisted? sync_with_transaction_state !(@new_record || @destroyed) end

So by De Morgan's law your suggestion ends up being:

if persisted? @destroyed = destroy_row > 0 end

The problem is that does not work - it will fail the following test: https://github.com/rails/rails/blob/master/activerecord/test/cases/associations/bidirectional_destroy_dependencies_test.rb#L34-L39

De Morgan's law doesn't apply when one of the expressions have side effects.

sgrif · 2016-12-07T21:51:40Z

I really would prefer a solution that doesn't need this @actually_destroyed flag. It reeks of mysql_real_escape_string to me.

stefanmb · 2016-12-07T22:37:05Z

I think the issue here is a breakdown between the ORM model represented by an ActiveRecord object and the underlying SQL data persisting that record.

As shown in the new test case, it's possible to have a race condition between callbacks on two records being destroyed and pointing to the same underlying database row. This situation is something that ActiveRecord seems to allow by design (for example, by permitting one to delete a record, which keeps the Ruby object but not the SQL data).

We use the on: :destroy callbacks for a global count of resources, so we are interested in the underlying SQL data and not (the potentially many) ActiveRecord objects representing it.

Since a record can have destroy called an unlimited number of times we cannot use the @destroyed variable to represent the state of the SQL database. We need some new variable that stores whether or not the current call actually affected any rows.

Perhaps @__actually_destroyed should be called @_rows_deleted?

sgrif · 2016-12-07T23:06:53Z

I think you're missing the point that I'm getting at. When we know that we have destroyed the record, we don't actually need to try again. I don't see any reason that we need to introduce a new flag for this.

stefanmb · 2016-12-07T23:14:12Z

Aren't we then back to the original problem? Each user code call to destroy produces a on: destroy callback to user code (even though the underlying row was deleted only once)?

Is this the desired behaviour in ActiveRecord? I apologize if I'm not following your suggestion.

sgrif · 2016-12-07T23:31:23Z

No, the same way that we don't attempt when you call .save if .changed? is false, we shouldn't attempt to do anything when you call .destroy if .destroyed? is true.

stefanmb · 2016-12-07T23:43:21Z

I get what you're saying, but this is the sequence I see:

Call destroy: Deletes Rows (first time around)
In transaction_include_any_action: returns destroyed? (== true)
User callback occurs.
Call destroy: No-op (second time around)
In transaction_include_any_action: returns destroyed? (== true) <-- Set during (1) above.
User callback occurs. <-- Bug!

matthewd · 2016-12-08T01:26:36Z

@sgrif this problem isn't record.destroy; record.destroy -- it's destroy getting called on two different instances that both point to the same DB row, meaning the second one is newly destroyed in AR-land, but it didn't actually kill a row because the row wasn't there.

sgrif · 2016-12-08T17:59:44Z

@matthewd It looks like both are problems

sgrif · 2016-12-08T18:02:06Z

Let's rename @_actually_destroyed to something like @_deleted_rows_in_db, and check the ivar directly rather than adding another method onto Base

sgrif · 2016-12-08T18:02:57Z

Should we do a similar check on updates? (Much harder race condition to create, would only happen if one process deleted a row while another tried to update it)

sgrif · 2016-12-08T20:50:31Z

activerecord/lib/active_record/transactions.rb

@@ -461,7 +461,7 @@ def transaction_include_any_action?(actions) #:nodoc:
          when :create
            transaction_record_state(:new_record)
          when :destroy
-            destroyed?
+            @_deleted_rows_in_db ||= false


Shouldn't this be || false since the object is frozen at this point?

Oh wait never mind we override freeze to only affect the attributes

The assignment is needed to avoid: warning: instance variable @_deleted_rows_in_db not initialized

Is there a better way to do this?

stefanmb · 2016-12-08T20:54:41Z

Should we do a similar check on updates? (Much harder race condition to create, would only happen if one process deleted a row while another tried to update it)

I don't think the same issue would happen because the callback check is:

          when :update
            !(transaction_record_state(:new_record) || destroyed?)
          end

sgrif · 2016-12-08T20:55:55Z

Right, which would still run callbacks if the same race condition occurred.

stefanmb · 2016-12-08T21:11:41Z

I'm not sure I understand. The destroy race occurred because @destroyed is always true after every call to destroy. The update race cannot occur because the update callback can only trigger if @destroyed == false.

If we are concerned with a record being destroyed during a callback then we need mutual exclusion between (user code) callbacks and the destroy call.

How would the check you are suggesting look? Sorry if I'm misunderstanding.

sgrif · 2016-12-08T21:19:36Z

And the current check for updates is also true, even if the underlying database record was deleted before we tried calling .save. The check would look identical, we would see if the affected row count is greater than zero, and set a flag if so.

stefanmb · 2016-12-08T23:24:52Z

@sgrif I made a rough pass at adding the same check to the update calls. It's a little trickier because a bunch of tests fail since they call save without actually updating any attributes (so the affected row count is 0). I made said tests change some attribute so that they pass. Likewise, the touch method has to set the affected row count.

Do you think this approach is acceptable? If so I'll clean up the PR.

P.S. Thanks again for taking the time to walk me through this.

sgrif · 2016-12-08T23:26:29Z

Hm. If the affected row count is 0 when no attributes were touched my base assumption might have been wrong. I will dig deeper in the morning.

sgrif

Can you add a changelog entry as well?

sgrif · 2016-12-09T16:09:35Z

activerecord/lib/active_record/transactions.rb

          end
        end
      end

-    private


I don't think you meant to delete this

Race conditions can occur when an ActiveRecord is destroyed twice or destroyed and updated. The callbacks should only be triggered once, similar to a SQL database trigger.

rails#29318 rails#27248 introduced `@_trigger_update_callback` but because optimistic locking doesn't call `super` when updating rows, it wasn't being set.

rails-bot assigned sgrif Dec 2, 2016

byroot reviewed Dec 2, 2016

View reviewed changes

maclover7 added the activerecord label Dec 2, 2016

rafaelfranca assigned rafaelfranca and sgrif and unassigned sgrif Dec 6, 2016

matthewd reviewed Dec 6, 2016

View reviewed changes

kaspth reviewed Dec 6, 2016

View reviewed changes

sgrif reviewed Dec 7, 2016

View reviewed changes

sgrif reviewed Dec 8, 2016

View reviewed changes

sgrif reviewed Dec 9, 2016

View reviewed changes

activerecord/lib/active_record/transactions.rb

end

end

end

private

Copy link

Contributor

sgrif Dec 9, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you meant to delete this

Emulate db trigger behaviour for after_commit :destroy, :update

371c083

Race conditions can occur when an ActiveRecord is destroyed twice or destroyed and updated. The callbacks should only be triggered once, similar to a SQL database trigger.

sgrif merged commit a9d72f6 into rails:master Dec 9, 2016

matthewd mentioned this pull request Jun 2, 2017

after_commit doesn't work with optimistic locking #29318

Closed

morganatwishpond mentioned this pull request Jun 2, 2017

Restore after_commit on: :update with optimistic locking #29321

Closed

fatkodima mentioned this pull request Feb 17, 2018

Possible regression of after_rollback on: :update and after_rollback on: :destroy is not working #32035

Closed

bensheldon mentioned this pull request Oct 5, 2022

Don't trigger after_commit :destroy callback again on destroy if record previously was destroyed #46197

Merged

9 tasks

                         end
                       end
                     end
-                  private

Idempotent option for after_commit :destroy callback #27248

Idempotent option for after_commit :destroy callback #27248

Conversation

stefanmb commented Dec 2, 2016

Summary

rails-bot commented Dec 2, 2016

byroot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

byroot commented Dec 2, 2016

stefanmb commented Dec 2, 2016

byroot commented Dec 2, 2016

stefanmb commented Dec 6, 2016

rafaelfranca commented Dec 6, 2016

sgrif commented Dec 6, 2016

rafaelfranca commented Dec 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewd commented Dec 6, 2016

stefanmb commented Dec 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgrif commented Dec 7, 2016

stefanmb commented Dec 7, 2016

sgrif commented Dec 7, 2016

stefanmb commented Dec 7, 2016

sgrif commented Dec 7, 2016

stefanmb commented Dec 7, 2016

matthewd commented Dec 8, 2016

sgrif commented Dec 8, 2016

sgrif commented Dec 8, 2016

sgrif commented Dec 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanmb commented Dec 8, 2016

sgrif commented Dec 8, 2016

stefanmb commented Dec 8, 2016 • edited

sgrif commented Dec 8, 2016

stefanmb commented Dec 8, 2016

sgrif commented Dec 8, 2016

sgrif left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanmb commented Dec 8, 2016 •

edited