Added #or to ActiveRecord::Relation #16052

Closed
wants to merge 1 commit into
from

Projects

None yet
@matthewd
Member
matthewd commented Jul 4, 2014

To complement the default and behaviour while building relations, allow two very similar relations to be combined with an or.

Post.where('id = 1').or(Post.where('id = 2'))
# => SELECT * FROM posts WHERE (id = 1) OR (id = 2)

Unlike previous attempts, we avoid any ambiguity about "how much" we're ORing by exclusively accepting a second relation -- thus, we're ORing whatever conditions differ between the two.

This is a much stricter variant of @gaelmuller's #9052; see also @oelmekki's #10891.

/cc @dhh @jeremy @tenderlove @rafaelfranca

@matthewd matthewd Added #or to ActiveRecord::Relation
    Post.where('id = 1').or(Post.where('id = 2'))
    # => SELECT * FROM posts WHERE (id = 1) OR (id = 2)

[Matthew Draper & Gael Muller]
71ca4ec
@dhh
Member
dhh commented Jul 4, 2014

API-wise, this seems reasonable to me. Others can have a look at the implementation.

@rafaelfranca rafaelfranca commented on the diff Jul 4, 2014
activerecord/lib/active_record/relation/query_methods.rb
+ theirs = theirs.map { |x| String === x ? Arel.sql(x) : x }
+
+ mine = [Arel::Nodes::And.new(mine)] if mine.size > 1
+ theirs = [Arel::Nodes::And.new(theirs)] if theirs.size > 1
+
+ common << Arel::Nodes::Or.new(mine.first, theirs.first)
+ end
+
+ send("#{combining}_values=", common)
+ end
+
+ self
+ end
+
+ def structurally_compatible?(other, allowed_to_vary)
+ Relation::SINGLE_VALUE_METHODS.all? do |name|
@rafaelfranca
rafaelfranca Jul 4, 2014 Member

I'd extract these conditionals to smaller private methods. I know what they are doing but it would be better if they were easily to nay contributor understand what they are doing.

@matthewd
matthewd Jul 5, 2014 Member

Yeah, this ended up a bit uglier than I'd originally anticipated, and I didn't go back and clean it up. Will do.

@rafaelfranca rafaelfranca commented on the diff Jul 4, 2014
activerecord/lib/active_record/relation/query_methods.rb
+
+ send("#{combining}_values=", common)
+ end
+
+ self
+ end
+
+ def structurally_compatible?(other, allowed_to_vary)
+ Relation::SINGLE_VALUE_METHODS.all? do |name|
+ send("#{name}_value") == other.send("#{name}_value")
+ end &&
+ (Relation::MULTI_VALUE_METHODS - [allowed_to_vary, :extending]).all? do |name|
+ send("#{name}_values") == other.send("#{name}_values")
+ end &&
+ (extending_values - [NullRelation]) == (other.extending_values - [NullRelation]) &&
+ !limit_value &&
@rafaelfranca
rafaelfranca Jul 4, 2014 Member

Maybe we could put these simple conditional on the begging of the expression since they are cheaper and Ruby would not evaluate the rest of the expression if any of these are true.

@matthewd
matthewd Jul 5, 2014 Member

An early exit just means we're heading for an exception, so it probably doesn't gain us much.

@egilburg
Contributor
egilburg commented Jul 4, 2014
  1. Is this chainable?

a.or(b).or(c)

  1. If yes, does it combine with and and what's the evaluation order for something like:

a.or(b).and(c).or(d)

@rafaelfranca
Member

@egilburg they only work with relations so they can't be combined with and. We don't even have a and method ๐Ÿ˜„.

They can be chained but with other relation so b and c need to be something like:

User.where('something').or(User.where('something')).or(User.where('something'))

But I believe something like this is still valid:

User.where('something').or(User.where('something')).where(bar: true)
@oelmekki
oelmekki commented Jul 4, 2014

I love this solution, thanks.

I don't see in tests anything specific to handling where.not, does it
works without problems ?

Something like :

Post.where.not('id = 1').or(Post.where.not('id = 2'))

Not that it would be very good code (it's quite hard to understand),
but there probably will be people to try it.

@egilburg
Contributor
egilburg commented Jul 4, 2014

Sorry, instead of and I meant just .where().where().where() which of course acts like AND.

In below example what would be evaluation order be?:

User.where(a).where(b).or(User.where(c))

(A && B) || C

or

A && (B || C)

And in below example:

User.where(a).or(User.where(b)).where(c)

Is it:

(A || B) && C

or

A || (B && C)

@gaelmuller

Obviously I think this is needed, but I don't see what is wrong with my
proposition.

I personnaly find this notation heavier.

Gael Muller

@rafaelfranca
Member
User.where(a).where(b).or(User.where(c))

(A && B) || C

User.where(a).or(User.where(b)).where(c)

(A || B) && C

@egilburg
Contributor
egilburg commented Jul 4, 2014

Ok, so it's always left-to-right. In this case, what ambiguity is solved by using the heavier:

User.where(field1: 'value1').or(User.where(field2: 'value2')).where(field3: 'value3')

As opposed to lighter:

User.where(field1: 'value1').or(field2: 'value2').where(field3: 'value3')

@matthewd
Member
matthewd commented Jul 4, 2014

and to get the other interpretations:

User.where(a).where(b).or(User.where(a).where(c)))
# or: z = User.where(a); z.where(b).or(z.where(c))

A && (B || C)

User.where(a).or(User.where(b).where(c))

A || (B && C)


In general, I anticipate this being used mostly to combine named scopes, likely often inside another named scope (like this): if you're already dealing with direct where conditions, you're probably still better off using straight SQL.

@dhh
Member
dhh commented Jul 4, 2014

User.where(a).or(User.where(b).where(c))
A || (B && C)

On Jul 4, 2014, at 11:05 AM, Eugene Gilburg notifications@github.com wrote:

Ok, so it's always left-to-right. In this case, what ambiguity is solved by using the heavier:

User.where(field1: 'value1').or(User.where(field2: 'value2')).where(field3: 'value3')

As opposed to lighter:

User.where(field1: 'value1').or(field2: 'value2').where(field3: 'value3')

โ€”
Reply to this email directly or view it on GitHub.

@oelmekki
oelmekki commented Jul 4, 2014

@rafaelfranca :

User.where(a).or(User.where(b)).where(c)
(A || B) && C

I think the point is precisely to avoid such order consideration and
only use OR on specific fields :

User.where( foo: 'a' ).or( User.where( foo: 'b' ) ).where( bar: 'c' )

and

User.where( foo: 'a' ).where( bar: 'c' ).or( User.where( foo: 'b' ) )

are both : ( foo = 'a' or foo = 'b' ) and bar = 'c'

Edit : or maybe not, given Matthew answer.

@gaelmuller :

This ordering thing was the problem with your implementation.

@matthewd
Member
matthewd commented Jul 4, 2014

The main ambiguity in that syntax, I think, is:

User.where(a: 1).where(b: 2).or(c: 3)

That will presumably resolve as "(a = 1 AND b = 2) OR c = 3", as the heavier one would... but how do I express the (arguably) more likely "a = 1 AND (b = 2 OR c = 3)"? (given that I'm in a method on the relation User.where(a: 1))

@egilburg
Contributor
egilburg commented Jul 4, 2014

In latter case you'd use:

User.where(a: 1).where(User.where(b: 2).or(c: 3))

So where would allow both a hash syntax or accept a sub-scope inside, allowing combining simple left-to-right interpretations with manual priority order if needed.

@sgrif sgrif commented on the diff Jul 4, 2014
activerecord/lib/active_record/relation/query_methods.rb
+ end
+
+ def or!(other)
+ combining = group_values.any? ? :having : :where
+
+ unless structurally_compatible?(other, combining)
+ raise ArgumentError, 'Relation passed to #or must be structurally compatible'
+ end
+
+ unless other.is_a?(NullRelation)
+ left_values = send("#{combining}_values")
+ right_values = other.send("#{combining}_values")
+
+ common = left_values & right_values
+ mine = left_values - common
+ theirs = right_values - common
@sgrif
sgrif Jul 4, 2014 Member

Is there a particular reason for this? Shouldn't

WHERE a = 1 AND (b = 2 OR c = 3)

be equivalent to

WHERE ((a = 1 AND b = 2) OR (a = 1 AND c = 3))
@matthewd
matthewd Jul 5, 2014 Member
  1. legibility / length of the generated query
  2. keeps the query the same "shape" as the set of relation calls that created it

So, yes.. but it seemed like a nice thing to do, and not unreasonably expensive.

@euwest
euwest Jan 31, 2015

in your example @sgrif if a != 1 wouldn't the first fail after one check whereas the second would still make two checks?

@sgrif sgrif commented on the diff Jul 4, 2014
activerecord/lib/active_record/relation/query_methods.rb
+ combining = group_values.any? ? :having : :where
+
+ unless structurally_compatible?(other, combining)
+ raise ArgumentError, 'Relation passed to #or must be structurally compatible'
+ end
+
+ unless other.is_a?(NullRelation)
+ left_values = send("#{combining}_values")
+ right_values = other.send("#{combining}_values")
+
+ common = left_values & right_values
+ mine = left_values - common
+ theirs = right_values - common
+
+ if mine.any? && theirs.any?
+ mine = mine.map { |x| String === x ? Arel.sql(x) : x }
@sgrif
sgrif Jul 4, 2014 Member

Could this be pulled into a private method?

@sgrif sgrif commented on the diff Jul 4, 2014
activerecord/lib/active_record/relation/query_methods.rb
+ raise ArgumentError, 'Relation passed to #or must be structurally compatible'
+ end
+
+ unless other.is_a?(NullRelation)
+ left_values = send("#{combining}_values")
+ right_values = other.send("#{combining}_values")
+
+ common = left_values & right_values
+ mine = left_values - common
+ theirs = right_values - common
+
+ if mine.any? && theirs.any?
+ mine = mine.map { |x| String === x ? Arel.sql(x) : x }
+ theirs = theirs.map { |x| String === x ? Arel.sql(x) : x }
+
+ mine = [Arel::Nodes::And.new(mine)] if mine.size > 1
@sgrif
sgrif Jul 4, 2014 Member

Does an And node work with a single child? It feels like it should, the rest of this method could get much simpler if it did.

@sgrif sgrif commented on the diff Jul 4, 2014
activerecord/lib/active_record/relation/query_methods.rb
+ unless other.is_a?(NullRelation)
+ left_values = send("#{combining}_values")
+ right_values = other.send("#{combining}_values")
+
+ common = left_values & right_values
+ mine = left_values - common
+ theirs = right_values - common
+
+ if mine.any? && theirs.any?
+ mine = mine.map { |x| String === x ? Arel.sql(x) : x }
+ theirs = theirs.map { |x| String === x ? Arel.sql(x) : x }
+
+ mine = [Arel::Nodes::And.new(mine)] if mine.size > 1
+ theirs = [Arel::Nodes::And.new(theirs)] if theirs.size > 1
+
+ common << Arel::Nodes::Or.new(mine.first, theirs.first)
@sgrif
sgrif Jul 4, 2014 Member

This reads really weird to me, but I don't have a concrete suggestion to improve it.

@sgrif
sgrif Jul 4, 2014 Member

Maybe just renaming common to combined or something. This also could become much simpler if an Or node could work when one of its children were empty, or a specialized node for that case.

@sgrif sgrif commented on the diff Jul 4, 2014
activerecord/lib/active_record/relation/query_methods.rb
+ # The two relations must be structurally compatible: they must be scoping the same model, and
+ # they must differ only by +where+ (if no +group+ has been defined) or +having+ (if a +group+ is
+ # present). Neither relation may have a +limit+, +offset+, or +uniq+ set.
+ #
+ # Post.where("id = 1").or(Post.where("id = 2"))
+ # # SELECT `posts`.* FROM `posts` WHERE (('id = 1' OR 'id = 2'))
+ #
+ def or(other)
+ spawn.or!(other)
+ end
+
+ def or!(other)
+ combining = group_values.any? ? :having : :where
+
+ unless structurally_compatible?(other, combining)
+ raise ArgumentError, 'Relation passed to #or must be structurally compatible'
@sgrif
sgrif Jul 4, 2014 Member

Could we provide more guidance as to why they were structurally incompatible? Perhaps structurally_compatible? could become a structural_incompatibilities method, which returns an array of all of the ways in which they are incompatible, the condition could change to structural_incompatibilities.any?, and we could join the result for the error message.

@matthewd
matthewd Jul 5, 2014 Member

๐Ÿ‘ -- I'd pondered exactly that, so if we've both thought of it, it must be a good idea :)

@sgrif sgrif commented on the diff Jul 5, 2014
activerecord/lib/active_record/relation/query_methods.rb
+ #
+ # Post.where("id = 1").or(Post.where("id = 2"))
+ # # SELECT `posts`.* FROM `posts` WHERE (('id = 1' OR 'id = 2'))
+ #
+ def or(other)
+ spawn.or!(other)
+ end
+
+ def or!(other)
+ combining = group_values.any? ? :having : :where
+
+ unless structurally_compatible?(other, combining)
+ raise ArgumentError, 'Relation passed to #or must be structurally compatible'
+ end
+
+ unless other.is_a?(NullRelation)
@sgrif
sgrif Jul 5, 2014 Member

Would we be able to avoid this is_a? check by defining having_values and where_values on NullRelation?

@matthewd
matthewd Jul 5, 2014 Member

๐Ÿ‘

@sgrif sgrif commented on the diff Jul 5, 2014
activerecord/lib/active_record/null_relation.rb
@@ -77,5 +77,13 @@ def calculate(operation, _column_name, _options = {})
def exists?(_id = false)
false
end
+
+ def or(other)
+ if other.is_a?(NullRelation)
@sgrif
sgrif Jul 5, 2014 Member

Should this just be

def or(other)
  other
end
@matthewd
matthewd Jul 5, 2014 Member

My two considerations here were that we should probably return a new relation (even though mutating a relation is not supported, people do it, so I'm hesitant to introduce the first code that actually breaks), and that we should probably still do the compatibility check, lest we mislead someone who's experimenting.

@al2o3cr
Contributor
al2o3cr commented Jul 6, 2014

This seems to avoid some of the issues from the previous or attempts (for instance, chaining on an existing where doesn't produce unexpected SQL) but introduces a distinction between bound params and SQL strings:

      # works
      base_rel = Post.where(id: 1)
      base_rel.where("title = 'bar'").or(base_rel.where("title = 'baz'")).to_a

      # fails with "ArgumentError: Relation passed to #or must be structurally compatible"
      base_rel = Post.where(id: 1)
      base_rel.where(title: 'bar').or(base_rel.where(title: 'baz')).to_a

This appears to be caused by the two relations being ord having different bind_values in the second case. I'm not sure how to deal with merging bind_values, but this should work or it's going to be very hard to reliably or relations.

@coenert
coenert commented Jul 7, 2014

Maybe you can use a code block as argument to make it simpeler in use:

Post.where(id: 1).or({ where(id: 2) })
@oelmekki

Btw @matthewd , by far the biggest problem I had with
activerecord_any_of was the lost of atypical information (not WHERE,
HAVING or JOIN) from more elaborated queries (that's why I have scoped
it behind WhereChain to make clear you're supposed to use where like
relations, and not just any kind of relation you want).

I've toyed for a time with the idea of using UNION instead of OR for
those queries (which remain fairly rare compared to usual queries):
better dropping perfs (provided everyone is aware of it) than failing.

I've hit two limitations making this a hard task :

  • arel implementation of UNION is tricky and probably needs refactoring
    for it to become a first class keyword
  • sqlite's support of UNION is problematic, for example you can't use a
    dedicated LIMIT for each statement union'd

In case it helps.

@sgrif
Member
sgrif commented Nov 5, 2014

It looks like this will break on queries which use bound parameters, we should probably merge the bind_values and add a test for hash style where, as well.

@sgrif
Member
sgrif commented Jan 28, 2015

Merged in 9e42cf0

@sgrif sgrif closed this Jan 28, 2015
@sealocal
Contributor

๐Ÿ‘

@gabebw
Contributor
gabebw commented Jan 29, 2015

<3 Thank you!

@senny
Member
senny commented Jan 29, 2015

@matthewd @sgrif ๐ŸŽ‰ โค๏ธ

@jrichardlai

Woot! ๐Ÿ‘

@scudelletti

๐Ÿ’œ

@dainmiller

๐Ÿ‘

@lucascaton
Contributor

Nice one! ๐Ÿ‘

@duduribeiro

๐Ÿ‘

@brunojabs

Great! ๐Ÿ‘ ๐ŸŽ‰

@flowerett

๐ŸŽ‰

@deniskorobicyn

๐Ÿ‘

@nurey
nurey commented Feb 3, 2015

โœจ ๐Ÿ‘
any idea which version of Rails this will land in?

@dhh
Member
dhh commented Feb 3, 2015

Rails 5.0.

On Feb 3, 2015, at 11:35, Ilia Lobsanov notifications@github.com wrote:

any idea which version of Rails this will land in?

โ€”
Reply to this email directly or view it on GitHub.

@EppO
EppO commented Feb 6, 2015

Awesome ! ๐Ÿ‘

@dalpo
dalpo commented Feb 16, 2015

โค๏ธ

@dgilperez

Bravo! โค๏ธ

@pinak1180

Great..!!! ๐Ÿ‘

@sergiotapia

Fanastic!

@DNA DNA referenced this pull request in hummingbird-me/hummingbird Jul 20, 2015
Merged

Remove the Watchlists model #607

@MitchellGeere

It's wonderful but I feel like I didn't see an example above that allows for the following example:

 @person = Person.where("(first_name = :forename OR middle_name = :forename OR nickname = :forename) AND last_name = :last_name AND gender = :gender AND birthday = :birthday", forename: first_name, last_name: last_name, gender: gender, birthday: birthday).first

So I would reuse the first name passed in to try an prepare for the chance that someone could have used the persons middle or nickname instead of their full name. Stupid example but on contracts I am David (first name), to the general public I am Mitchell (middle) and to friends I am Mitch (nickname) and I have been searched for on all of them.

Looking forward to getting the or relation!

@albertossilva albertossilva commented on the diff Aug 28, 2015
activerecord/lib/active_record/relation/query_methods.rb
+
+ common << Arel::Nodes::Or.new(mine.first, theirs.first)
+ end
+
+ send("#{combining}_values=", common)
+ end
+
+ self
+ end
+
+ def structurally_compatible?(other, allowed_to_vary)
+ Relation::SINGLE_VALUE_METHODS.all? do |name|
+ send("#{name}_value") == other.send("#{name}_value")
+ end &&
+ (Relation::MULTI_VALUE_METHODS - [allowed_to_vary, :extending]).all? do |name|
+ send("#{name}_values") == other.send("#{name}_values")
@albertossilva
albertossilva Aug 28, 2015

Could you create a method like this?

def validate_strutucture_for(attr)
  lambda { |name| send("#{name}_#{attr}") == other.send("#{name}_#{attr}") }
end

And with that method, you can write

Relation::SINGLE_VALUE_METHODS.all? &validate_strutucture_for("value")

and

(Relation::MULTI_VALUE_METHODS - [allowed_to_vary, :extending]).all? &validate_strutucture_for("values")

What do you think?

@sgrif
sgrif Aug 28, 2015 Member

I think that's a bit harder to read in Ruby than what we have now (though it could be broken up)

@albertossilva albertossilva commented on the diff Aug 28, 2015
activerecord/test/cases/relation/or_test.rb
+ assert_equal expected, Post.where('id = 1').or(Post.containing_the_letter_a)
+ end
+
+ def test_or_inside_named_scope
+ expected = Post.where("body LIKE '\%a\%' OR title LIKE ?", "%'%").order('id DESC').to_a
+ assert_equal expected, Post.order(id: :desc).typographically_interesting
+ end
+
+ def test_or_on_loaded_relation
+ expected = Post.where('id = 1 or id = 2').to_a
+ p = Post.where('id = 1')
+ p.load
+ assert_equal p.loaded?, true
+ assert_equal expected, p.or(Post.where('id = 2')).to_a
+ end
+
@albertossilva
albertossilva Aug 28, 2015

Is this necessary line?

@sgrif
sgrif Aug 28, 2015 Member

The code in this PR differs from what was merged. See
9e42cf0

On Fri, Aug 28, 2015, 12:21 PM Alberto notifications@github.com wrote:

In activerecord/test/cases/relation/or_test.rb
#16052 (comment):

  •  assert_equal expected, Post.where('id = 1').or(Post.containing_the_letter_a)
    
  • end
  • def test_or_inside_named_scope
  •  expected = Post.where("body LIKE '\%a\%' OR title LIKE ?", "%'%").order('id DESC').to_a
    
  •  assert_equal expected, Post.order(id: :desc).typographically_interesting
    
  • end
  • def test_or_on_loaded_relation
  •  expected = Post.where('id = 1 or id = 2').to_a
    
  •  p = Post.where('id = 1')
    
  •  p.load
    
  •  assert_equal p.loaded?, true
    
  •  assert_equal expected, p.or(Post.where('id = 2')).to_a
    
  • end

Is this line necessary?

โ€”
Reply to this email directly or view it on GitHub
https://github.com/rails/rails/pull/16052/files#r38228595.

@bf4
Contributor
bf4 commented Sep 1, 2015

FWIW, I've written up a backport of ActiveRelation#or to Rails 4.2.3

@Eric-Guo
Contributor

I just copy @bf4 file to a new gems called where-or

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment