Add #create_or_find_by to lean on unique constraints #31989

dhh · 2018-02-13T23:47:21Z

Attempts to create a record with the given attributes in a table that has a unique constraint
on one or several of its columns. If a row already exists with one or several of these
unique constraints, the exception such an insertion would normally raise is caught,
and the existing record with those attributes is sought found using #find_by.

This is similar to #find_or_create_by, but avoids the problem of stale reads, as that methods needs
to first query the table, then attempt to insert a row if none is found. That leaves a timing gap
between the SELECT and the INSERT statements that can cause problems in high throughput applications.

There are several drawbacks to #create_or_find_by, though:

The underlying table must have the relevant columns defined with unique constraints.
A unique constraint violation may be triggered by only one, or at least less than all,
of the given attributes. This means that the subsequent #find_by may fail to find a
matching record, which will then return nil, rather than a record will the given attributes.
It relies on exception handling to handle control flow, which may be marginally slower. And

This method will always returns a record if all given attributes are covered by unique constraints,
but if creation was attempted and failed due to validation errors it won't be persisted, you get what
#create returns in such situation.

nynhex · 2018-02-13T23:49:48Z

This is really good. I have multiple use cases for this. 👍

jeremy · 2018-02-13T23:57:19Z

activerecord/lib/active_record/relation.rb

+    def create_or_find_by!(attributes, &block)
+      create!(attributes, &block)
+    rescue ActiveRecord::RecordNotUnique
+      find_by(attributes)


Should raise in the fallback find case? It'd be surprising to ever get a nil result (for whatever reason, including a missing unique index) from these methods.

Yeah, good point. Let's change that to find_by!.

Better than nil.

rafaelfranca · 2018-02-14T00:11:59Z

activerecord/test/cases/relations_test.rb

+      Subscriber.create_or_find_by(nick: "bob", name: "the cat")
+    end
+  end
+


I think this is missing a test with create_with (to change the behavior of the create) and with where (to change the behavior of find!

I'm not sure if those method should be used with create_or_find_by but right now they can and I have no idea of the effect they would have, so better to test them to make sure it is what we expect.

I don't see how that differs? create_or_find_by uses the same flow as find_or_create_by, except instead of an || we're using a rescue. But the flow is the same.

I mean, it's not that we can't test it, just that I don't think it teaches us anything interesting. The test would be identical to the one for test_find_or_create_by_with_create_with.

If it is the same behavior we want so I guess it is fine being tested in that test

eugeneius · 2018-02-14T00:33:00Z

activerecord/lib/active_record/relation.rb

+    #   matching record, which will then return nil, rather than a record will the given attributes.
+    # * It relies on exception handling to handle control flow, which may be marginally slower. And
+    #
+    # This method will always returns a record if all given attributes are covered by unique constraints,


This isn't totally true: another client could update or delete the row between the rejected INSERT and the subsequent SELECT. This race condition is complementary to the one in #find_or_create_by.

That caveat probably belongs in the "drawbacks" section, but even still I don't think we can make this claim here.

Good point. Will add that point. It's a much rarer race condition in many apps, I'd say.

matthewd · 2018-02-14T00:34:35Z

activerecord/lib/active_record/relation.rb

+    # #create returns in such situation.
+    def create_or_find_by(attributes, &block)
+      create(attributes, &block)
+    rescue ActiveRecord::RecordNotUnique


This needs transaction(requires_new: true) do around the create to work in an ongoing surrounding transaction (on at least PostgreSQL)

Why is this not necessary for find_or_create_by?

Because it doesn't cause an SQL error and then attempt to recover. PostgreSQL remembers when an error has occurred inside a transaction, and disallows all further operations until that transaction has been rolled back.

If we are willing to require PG 9.5 or later for this (when using the PG adapter), we could just stick ON CONFLICT DO NOTHING on the end instead. This would let us wrap the whole thing in a transaction instead of just the create, which eliminates any possibility of race conditions.

matthewd · 2018-02-14T00:38:20Z

activerecord/lib/active_record/relation.rb

+    def create_or_find_by!(attributes, &block)
+      create!(attributes, &block)
+    rescue ActiveRecord::RecordNotUnique
+      find_by!(attributes)


Do we want to retry here instead of raising if the find fails? Seems confusing for create_or_find_by! to raise RecordNotFound, just because of a racing delete.

You can't retry this failure as it should only occur when you use attributes where not all of them are covered by unique constraints. It's more like a INVALID QUERY kind of failure.

Oh right, I was assuming the delete race, rather than operator error. I guess we could retry once, so we're safe from a poorly-timed simple delete. That way we'd only be tripped up by a racing delete; insert; delete.

True. I'm not sure whether having just a single retry is a good warranty, though. Since it's hard for us to tell the difference between legit race condition and invalid query.

dhh · 2018-02-14T00:54:13Z

TIL! Do you have an idea for a good test for that?

…

On Tue, Feb 13, 2018 at 4:49 PM, Matthew Draper ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In activerecord/lib/active_record/relation.rb <#31989 (comment)>: > + # + # There are several drawbacks to #create_or_find_by, though: + # + # * The underlying table must have the relevant columns defined with unique constraints. + # * A unique constraint violation may be triggered by only one, or at least less than all, + # of the given attributes. This means that the subsequent #find_by may fail to find a + # matching record, which will then raise an `ActiveRecord::NotFound` exception, + # rather than a record will the given attributes. + # * It relies on exception handling to handle control flow, which may be marginally slower. And + # + # This method will always returns a record if all given attributes are covered by unique constraints, + # but if creation was attempted and failed due to validation errors it won't be persisted, you get what + # #create returns in such situation. + def create_or_find_by(attributes, &block) + create(attributes, &block) + rescue ActiveRecord::RecordNotUnique Because it doesn't cause an SQL error and then attempt to recover. PostgreSQL remembers when an error has occurred inside a transaction, and disallows all further operations until that transaction has been rolled back. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#31989 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAKtbTMH-JrUyizELV10shNLxOgesnAks5tUi2lgaJpZM4SElV8> .

matthewd · 2018-02-14T01:21:25Z

I think this should do it:

def test_create_or_find_by_within_transaction
  assert_nil Subscriber.find_by(nick: "bob")

  subscriber = Subscriber.create!(nick: "bob")

  Subscriber.transaction do
    assert_equal subscriber, Subscriber.create_or_find_by(nick: "bob")
    assert_not_equal subscriber, Subscriber.create_or_find_by(nick: "cat")
  end
end

tjschuck

Documentation edits.

tjschuck · 2018-02-14T16:55:26Z

activerecord/lib/active_record/relation.rb

+    # * The underlying table must have the relevant columns defined with unique constraints.
+    # * A unique constraint violation may be triggered by only one, or at least less than all, 
+    #   of the given attributes. This means that the subsequent #find_by may fail to find a
+    #   matching record, which will then raise an `ActiveRecord::NotFound` exception,


`ActiveRecord::NotFound` needs <tt> for Rdoc code formatting, not backticks.

tjschuck · 2018-02-14T16:55:59Z

activerecord/lib/active_record/relation.rb

+    # * A unique constraint violation may be triggered by only one, or at least less than all, 
+    #   of the given attributes. This means that the subsequent #find_by may fail to find a
+    #   matching record, which will then raise an `ActiveRecord::NotFound` exception,
+    #   rather than a record will the given attributes.


will => with

tjschuck · 2018-02-14T16:57:01Z

activerecord/lib/active_record/relation.rb

+    # Attempts to create a record with the given attributes in a table that has a unique constraint
+    # on one or several of its columns. If a row already exists with one or several of these
+    # unique constraints, the exception such an insertion would normally raise is caught, 
+    # and the existing record with those attributes is sought found using #find_by.


is sought found using #find_by — "sought" should be deleted.

tjschuck · 2018-02-14T16:58:38Z

activerecord/lib/active_record/relation.rb

+    # and the existing record with those attributes is sought found using #find_by.
+    #
+    # This is similar to #find_or_create_by, but avoids the problem of stale reads between the SELECT 
+    # and the INSERT, as that methods needs to first query the table, then attempt to insert a row 


methods => method

@matthewd

Otherwise PG will complain with "PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block". Thanks @matthewd

@tjschuck

Thanks @tjschuck

AnalyzePlatypus · 2018-02-17T21:00:35Z

Awesome!

subhashsaran · 2018-02-18T18:09:38Z

Good stuff.

dhh · 2018-02-21T23:15:31Z

Happy to explore that as a PG 9.5 level-up somehow. But the whole feature shouldn’t depend on it imo.

…

On Feb 21, 2018, at 12:47, Sean Griffin ***@***.***> wrote: @sgrif commented on this pull request. In activerecord/lib/active_record/relation.rb: > + # + # There are several drawbacks to #create_or_find_by, though: + # + # * The underlying table must have the relevant columns defined with unique constraints. + # * A unique constraint violation may be triggered by only one, or at least less than all, + # of the given attributes. This means that the subsequent #find_by may fail to find a + # matching record, which will then raise an `ActiveRecord::NotFound` exception, + # rather than a record will the given attributes. + # * It relies on exception handling to handle control flow, which may be marginally slower. And + # + # This method will always returns a record if all given attributes are covered by unique constraints, + # but if creation was attempted and failed due to validation errors it won't be persisted, you get what + # #create returns in such situation. + def create_or_find_by(attributes, &block) + create(attributes, &block) + rescue ActiveRecord::RecordNotUnique If we are willing to require PG 9.5 or later for this (when using the PG adapter), we could just stick ON CONFLICT DO NOTHING on the end instead. This would let us wrap the whole thing in a transaction instead of just the create, which eliminates any possibility of race conditions. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

sikachu · 2018-02-21T23:57:17Z

(Hmm, @sgrif's comment was gone?)

@sgrif I'm 👍 on expanding this feature for PG 9.5+ in PG adapter. Other RDBMS can fallback to unique-key-constaint-exception-then-find.

yhirano55 · 2018-04-17T02:43:33Z

I want to use these awesome class methods on Rails 5.2, so much.
Do you have any plans to backport to 5-2-stable?

bogdanvlviv · 2018-04-17T08:12:18Z

I think we don't, "New features are only added to the master branch and will not be made available in point releases." by Maintenance Policy for Ruby on Rails.

yhirano55 · 2018-04-17T08:23:53Z

I see, thank you for your explanation.

djezzzl · 2018-12-07T12:55:53Z

Hi everybody,

As we start relying on the database constraints more, I'd suggest (and few people like the idea) to have the improved versions of validates_uniqueness_of and belongs_to from database_validations gem in the ActiveRecord. I can prepare a PR if needed.

I just opened the issue #34650 where we can discuss the topic.

Any feedback is very appreciated!

create_or_find_by will be implemented in Rails 6.0 (rails/rails#31989) However we've decided to use it before it comes out, and would love it to be in rubocop scope as well. I guess we're not the first ones to do it, so it could benefit to most.

### Description If `LegacyAppeal.find_or_create_by_vacols_id` is hit multiple times for an appeal that doesn't yet exist, it's possible for a race condition to arise where the appeal is saved by one process in between the time another process initializes and attempts to save it. I observed this happening when loading the case details page for appeals that exist in VACOLS but not Caseflow, when `find_or_create_by_vacols_id` is invoked from the appeals controller and the tasks controller in quick succession. When the second process attempts to save its version of the appeal, it fails with `PG::UniqueViolation` / `ActiveRecord::RecordNotUnique`. I changed the method to capture the `ActiveRecord::RecordNotUnique` and return the existing record via `find_by!` (which'll raise a `ActiveRecord::RecordNotFound` if it's unsuccessful), following [this pattern](rails/rails#31989), from the Rails 6 `create_or_find_by` method. It's not ideal to [use an exception as flow control](https://wiki.c2.com/?DontUseExceptionsForFlowControl), but I think it's okay in this case to let the database constraint determine the action.

Add #create_or_find_by to lean on unique constraints

744e37a

dhh requested a review from jeremy February 13, 2018 23:47

jeremy reviewed Feb 13, 2018

View reviewed changes

Raise an exception if no record can be found

ee7a453

Better than nil.

rafaelfranca reviewed Feb 14, 2018

View reviewed changes

eugeneius reviewed Feb 14, 2018

View reviewed changes

matthewd reviewed Feb 14, 2018

View reviewed changes

Address the race condition between INSERT and SELECT

cde5208

tjschuck reviewed Feb 14, 2018

View reviewed changes

dhh added 4 commits February 14, 2018 15:36

Require new transaction in case unique constraint causes rollback

03374e9

Otherwise PG will complain with "PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block". Thanks @matthewd

Documentation fixes

0768324

Thanks @tjschuck

Add changelog entry

72cbe0c

Remove trailing whitespace

ba45394

dhh merged commit fe6adf4 into master Feb 15, 2018

dhh deleted the create_or_find_by branch February 15, 2018 00:51

aried3r mentioned this pull request Aug 16, 2018

Unexpected rake task behavior change in Rails 5.2 #32870

Closed

djezzzl mentioned this pull request Dec 7, 2018

Improve belongs_to and validates_uniqueness_of from database_validations gem #34650

Closed

bogdanvlviv mentioned this pull request Dec 7, 2018

#create_or_find_by/!: add more tests and fix docs #34653

Merged

djezzzl mentioned this pull request Dec 17, 2018

Add rescue from ActiveRecord::RecordNotUnique #34725

Closed

bgeuken mentioned this pull request May 21, 2019

Add commit activitiy to user contribution graph openSUSE/open-build-service#7607

Merged

MaximeLaurenty mentioned this pull request Aug 8, 2019

Add create_or_find_by to save_bang.rb rubocop/rubocop-rails#102

Merged

8 tasks

tomas-nava mentioned this pull request Oct 6, 2020

recover from RecordNotUnique race condition when creating new legacy appeal department-of-veterans-affairs/caseflow#15391

Merged

alexcameron89 mentioned this pull request Jun 21, 2021

create_or_find_by doesn't attempt to find the record when you also have validates_uniqueness_of :field #36027

Closed

Add #create_or_find_by to lean on unique constraints #31989

Add #create_or_find_by to lean on unique constraints #31989

dhh commented Feb 13, 2018

nynhex commented Feb 13, 2018

jeremy Feb 13, 2018

dhh Feb 13, 2018

rafaelfranca Feb 14, 2018

dhh Feb 14, 2018

dhh Feb 14, 2018

rafaelfranca Feb 14, 2018

eugeneius Feb 14, 2018

dhh Feb 14, 2018

matthewd Feb 14, 2018

dhh Feb 14, 2018

matthewd Feb 14, 2018

sgrif Feb 21, 2018

matthewd Feb 14, 2018

dhh Feb 14, 2018

matthewd Feb 14, 2018

dhh Feb 14, 2018

dhh commented Feb 14, 2018 via email

matthewd commented Feb 14, 2018

tjschuck left a comment

tjschuck Feb 14, 2018

tjschuck Feb 14, 2018

tjschuck Feb 14, 2018

tjschuck Feb 14, 2018

AnalyzePlatypus commented Feb 17, 2018

subhashsaran commented Feb 18, 2018

dhh commented Feb 21, 2018 via email

sikachu commented Feb 21, 2018

yhirano55 commented Apr 17, 2018

bogdanvlviv commented Apr 17, 2018

yhirano55 commented Apr 17, 2018

djezzzl commented Dec 7, 2018 •

edited

Add #create_or_find_by to lean on unique constraints #31989

Add #create_or_find_by to lean on unique constraints #31989

Conversation

dhh commented Feb 13, 2018

nynhex commented Feb 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhh commented Feb 14, 2018 via email

matthewd commented Feb 14, 2018

tjschuck left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnalyzePlatypus commented Feb 17, 2018

subhashsaran commented Feb 18, 2018

dhh commented Feb 21, 2018 via email

sikachu commented Feb 21, 2018

yhirano55 commented Apr 17, 2018

bogdanvlviv commented Apr 17, 2018

yhirano55 commented Apr 17, 2018

djezzzl commented Dec 7, 2018 • edited

djezzzl commented Dec 7, 2018 •

edited