Fixed #25367 -- Allowed boolean expressions in QuerySet.filter() and exclude().

[schinckel]

Here is a first crack at making it possible to do nice things like:

queryset.filter(Exists(subquery))

Also:

queryset.annotate(foo=Case(When(Exists(subquery), then=...))

[schinckel]

I'm well aware of the superficial issues with this PR (imports and the like). What I want to know is if this is an appropriate approach to implement filter expressions.

@jarshwah and I have been discussing this for some time: we thought there might be a whole lot more work to it than this, but it seems to me that we can just inject some logic in to explicitly handle Expression objects, as long as they have an output_field that is a BooleanField.

[jarshwah]

A few things off the top of my head:

• Does this also work with exclude()?
• Does it handle __related fields?
• Do regular lookups (perhaps by adding a boolean output_field) work in the same way?

This is really cool though. I expected there to be a lot more work involved.

[schinckel]

Yes. .exclude(Exists(foo)), and also .filter(~Exists(...)), and .exclude(~Exists(...)) all appear to work.

I'm not sure about the other two: can you elaborate on what sort of queries you mean? Anything that is an expression that has an output_field that is a boolean should work.

[jarshwah]

2. Never mind - Exists takes a queryset which sets up joins etc.

3. Does using Lookups as Expressions work? See:

from django.db.models.lookups import GreaterThan
# Lookups don't currently define an output_field, but if they did..
Model.objects.filter(GreaterThan('counter', 3))

[schinckel]

Okay, @funkybob just showed me a pair of EXPLAINs that shows an order of magnitude improvement by only having the EXISTS in the WHERE clause, rather than in the select.

I'm surprised, as I thought postgres would do a better job of optimising this.

With EXISTS in SELECT: https://explain.depesz.com/s/xaH

With EXISTS only in WHERE: https://explain.depesz.com/s/nc9m

[schinckel]

@jarshwah Currently Model.objects.filter(GreaterThan('foo', 2)) fails, because the Lookup object doesn't like a string (or an F, actually).

[schinckel]

I think we could possibly avoid the need for a Lookup to define an output_field: wouldn't all lookups imply they are a boolean output? If so, we can allow any Lookup in a filter.

[schinckel]

Hmm. I'm not sure it's possible without quite a lot of work to do anything with a Lookup, as it is passed something as lhs that already has an output_field, and without that, we can't even instantiate a GreaterThan.

[funkybob]

I am a person and I approve of this feature :)

[adamchainz]

Oxford comma. The best type of comma.

👌 I'm a fan, though I'm not sure Django has an official policy.

[jarshwah]

Let's get this one merged I reckon. I've made a suggestion for a small improvement, but that's probably all it needs. Couldn't find the ticket (easily) so a link here or a number would be aces.

[schinckel]

@jarshwah Finally found the issue: https://code.djangoproject.com/ticket/25367

[felixxm]

I merged #11652 (Thanks Simon!) we should be able to get rid of for_where workaround.

[NyanKiyoshi]

The pull request was updated against the latest reviews 👍

[charettes]

It's nice to see that the diff is really small now that preliminary changes have landed 🎉

Thanks to everyone involved so far, really excited so this land in the near future.

[felixxm]

I squashed commits, rebased from master, pushed minor edits, and:

• added support for & operator,
• added some tests,
• moved unrelated docs changes to a separate PR Doc'd for_save argument of Expression.resolve_expression(). #11716.

I'm going to continue this tomorrow to make final edits (in docs and tests), to add more tests, and to check remaining comments.

[NyanKiyoshi]

Thanks @felixxm! I'm very busy so I didn't get any time to make changes on the PR and we really need such feature.

[felixxm]

PR didn't work properly on Oracle because it supports only filters and EXISTS in the WHERE clause, other must be compared with True. I added workaround that IMO is acceptable.

[felixxm]

@jarshwah Can you take a look at a85460e4da5835a3095462b07b4382033d094371?

[felixxm]

I pushed all edits.

[felixxm]

Thanks y'all 🎉 🎊

[charettes]

I was doing a bit of archaeology and I stumbled on #4801 (comment).

We're not currently affected by that since we don't expose expressions such as GreaterThanOrEqual, e.g. lookups are not currently expressions, but if we were to add support for things like #12041 which does exactly that we'd likely be affected by this issue.