Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Predicate functionality for Q objects #388

Closed
wants to merge 46 commits into from

4 participants

Preston Holmes Alex Gaynor Anssi Kääriäinen Florian Apolloner
added some commits September 08, 2012
Preston Holmes initial implementation of predicate into Q 3854ad7
Preston Holmes move manager check, add match_compile api 05dd71e
Preston Holmes typo fix 1551a42
Preston Holmes resolve a circular import 4b80f05
Preston Holmes clean up recursion ebbde80
Preston Holmes fix call to super.__init__ 5612d95
Preston Holmes more correctly match add_filter
since we aren't interested in values during traversal
0a7d769
Preston Holmes handle the case when related objects don't exist a11f1d6
Preston Holmes add initial tests b90bbe1
Preston Holmes Merge branch 'master' into q-predicate
Conflicts:
	django/db/models/sql/query.py
	updates location of lookup_sep constant
9a2febc
Preston Holmes use new lookup_sep constant location 6e92d8b
Preston Holmes fix up traversal 0dc4d52
Preston Holmes add more tests b9101d4
Preston Holmes initial gis test structure f73d053
Preston Holmes start on geodjango tests d031a86
Preston Holmes remove bbox helper 3484332
Preston Holmes updating distance tests with stubs 310b0db
Preston Holmes mostly testing stubs 5a7064c
Preston Holmes remove separate testing app no longer used 66285cc
Preston Holmes gis spatial predicates with tests b51e79d
Preston Holmes start on distances implementation b0ce713
Preston Holmes Fixed #18971 -- added root level CONTRIBUTING doc b9faeb2
Preston Holmes added distance lookups with tests f738405
Preston Holmes Merge branch 'q-predicate' of /Users/preston/Projects/code/django-con…
…tributing/test-vagrant/../../forks/django into q-predicate
cec603a
Preston Holmes add fix and test for bad field name f818406
Preston Holmes added alternate manager tests 052daf0
Preston Holmes initial pass at docs 934f5c6
Preston Holmes added release note f3732bd
Preston Holmes doc revision 9e27855
Preston Holmes Merge branch 'master' into q-predicate d9cf3d3
Preston Holmes add negation handling and tests 003b372
Preston Holmes finished wrong manager tests d742df8
Preston Holmes finished first pass at docs a9dc02e
Preston Holmes pep8 and cleanups b5b4b28
django/contrib/gis/db/models/sql/matching.py
((15 lines not shown))
  15
+def _contains(model, instance_value, value):
  16
+    return instance_value.contains(value)
  17
+
  18
+
  19
+def _contained(model, instance_value, value):
  20
+    instance_bbox = Polygon.from_bbox(instance_value.extent)
  21
+    value_bbox = Polygon.from_bbox(value.extent)
  22
+    return value_bbox.contains(instance_bbox)
  23
+
  24
+
  25
+def _contains_properly(model, instance_value, value):
  26
+    return instance_value.relate_pattern(value, 'T**FF*FF*')
  27
+
  28
+
  29
+def _coveredby(model, instance_value, value):
  30
+    # T*F**F***, *TF**F***, **FT*F***, **F*TF***
2
Alex Gaynor Owner
alex added a note September 22, 2012

This comment is pretty useless

Preston Holmes Owner
ptone added a note September 22, 2012

Yes - it was a note as to the GEO matrix options I wanted to cover, copied and pasted from a postgis or similar docs page - will strike

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Alex Gaynor alex commented on the diff September 22, 2012
django/contrib/gis/db/models/sql/matching.py
((22 lines not shown))
  22
+    return value_bbox.contains(instance_bbox)
  23
+
  24
+
  25
+def _contains_properly(model, instance_value, value):
  26
+    return instance_value.relate_pattern(value, 'T**FF*FF*')
  27
+
  28
+
  29
+def _coveredby(model, instance_value, value):
  30
+    # T*F**F***, *TF**F***, **FT*F***, **F*TF***
  31
+    return any((
  32
+            instance_value.relate_pattern(value, 'T*F**F***'),
  33
+            instance_value.relate_pattern(value, '*TF**F***'),
  34
+            instance_value.relate_pattern(value, '**FT*F***'),
  35
+            instance_value.relate_pattern(value, '**F*TF***'),
  36
+            ))
  37
+
1
Alex Gaynor Owner
alex added a note September 22, 2012

Use or here, not any, same for all the other places.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Alex Gaynor alex commented on the diff September 22, 2012
django/contrib/gis/db/models/sql/query.py
@@ -21,6 +22,13 @@
21 22
             ])
22 23
 ALL_TERMS.update(sql.constants.QUERY_TERMS)
23 24
 
  25
+ALL_MATCHES = sql.matching.match_functions
  26
+
  27
+# we update match functions in the reverse of query_terms, as we want the
  28
+# gis version to be the one in the final lookup ie 'contains' should be gis
  29
+# contains
  30
+ALL_MATCHES.update(match_functions)
  31
+
2
Alex Gaynor Owner
alex added a note September 22, 2012

This is mutating the external primary sql match functions, that shouldn't be necessary.

Preston Holmes Owner
ptone added a note September 22, 2012

is now mutating a local copy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/models/query_utils.py
@@ -41,7 +43,9 @@ class Q(tree.Node):
41 43
     default = AND
42 44
 
43 45
     def __init__(self, *args, **kwargs):
44  
-        super(Q, self).__init__(children=list(args) + list(six.iteritems(kwargs)))
  46
+        super(Q, self).__init__(children=list(args)
  47
+                + list(six.iteritems(kwargs)))
1
Alex Gaynor Owner
alex added a note September 22, 2012

This line spacing is really awkward, please don't make unrelated changes like this, especially when they make stuff uglier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Alex Gaynor alex commented on the diff September 22, 2012
django/db/models/query_utils.py
((15 lines not shown))
  81
+
  82
+                    parent.children.append(branch_root)
  83
+                    descend(branch_root, child.children)
  84
+                else:
  85
+                    # assuming we are in a properly formed Q, could only be
  86
+                    # a tuple
  87
+                    child_le = LookupExpression(expr=child, manager=manager)
  88
+                    parent.children.append(child_le)
  89
+
  90
+        root = LookupExpression(connector=self.connector, manager=manager)
  91
+        descend(root, self.children)
  92
+        root.negated = self.negated
  93
+
  94
+        self._compiled_matcher = root
  95
+
  96
+    def matches(self, instance, manager=None):
2
Alex Gaynor Owner
alex added a note September 22, 2012

Whatever manager is needs to be documented, or it shouldn't be a parameter on a public API.

Preston Holmes Owner
ptone added a note September 22, 2012

yes - it is documented

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/models/query_utils.py
((34 lines not shown))
  100
+        if manager is None:
  101
+            manager = instance._default_manager
  102
+
  103
+        if not self._compiled_matcher:
  104
+            # we are evaluating the first model, or are uncompiled
  105
+            self._compile_matcher(manager)
  106
+        if self._compiled_matcher.manager != manager:
  107
+            # the pre-compiled matcher was compiled for a different manager
  108
+            self._compile_matcher(manager)
  109
+        return_val = self._compiled_matcher.matches(instance)
  110
+        if self.negated:
  111
+            # It is extremely unlikely (impossible?) to end up with a root
  112
+            # Q object that is negated, given the implementation of __invert__
  113
+            # we should be able to return _compiled_matcher.matches() directly
  114
+            # but in case we encounter an unforseen edgecase, we check again
  115
+            return not return_val
1
Alex Gaynor Owner
alex added a note September 22, 2012

Either it's possible or it's not, if you can write a test case for it to happen it's possible and this comment is confusing, if not this condition should be removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/models/query_utils.py
((58 lines not shown))
  244
+                    self.lookup_type = parts.pop()
  245
+                    return
  246
+                # Unless we're at the end of the list of lookups, let's attempt
  247
+                # to continue traversing relations.
  248
+                if (counter + 1) < num_parts:
  249
+                    try:
  250
+                        lookup_model = lookup_field.rel.to
  251
+                    except AttributeError:
  252
+                        # Not a related field. Bail out.
  253
+                        self.lookup_type = parts.pop()
  254
+                        return
  255
+        else:
  256
+            # presumably we have a simple <field>=x with no lookup term
  257
+            # so we just use our default of exact
  258
+            self.attr_route.append(parts[0])
  259
+            return
1
Alex Gaynor Owner
alex added a note September 22, 2012

This function should return some values to be checke by its caller, not set things on self.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/models/query_utils.py
((80 lines not shown))
  266
+
  267
+    def matches(self, instance):
  268
+        """
  269
+        Evaluates an instance against the lookup we were created with.
  270
+        Return true if the instance matches the condiiton.
  271
+        """
  272
+        if not isinstance(instance, self.manager.model):
  273
+            raise ValueError("invalid manager given for {}".format(instance))
  274
+
  275
+        evaluators = {"AND": all, "OR": any}
  276
+        evaluator = evaluators[self.connector]
  277
+        return_val = None
  278
+        if self.children:
  279
+            return_val = (
  280
+                    evaluator(c.matches(instance) for c in self.children)
  281
+                    )
1
Alex Gaynor Owner
alex added a note September 22, 2012

Drop the parens and put this all on one line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/models/query_utils.py
((95 lines not shown))
  281
+                    )
  282
+        else:
  283
+            try:
  284
+                instance_value = self.get_instance_value(instance)
  285
+                return_val = self.lookup_function(instance, instance_value,
  286
+                        self.value)
  287
+            except AttributeError:
  288
+                # this is raised when we were not able to traverse the full
  289
+                # attribute route. In nearly all cases this means the match
  290
+                # failed as it specified a longer relationship chain then
  291
+                # exists for this instance.
  292
+                if (hasattr(self.lookup_function, 'none_is_true')
  293
+                    and self.lookup_function.none_is_true):
  294
+                    return_val = True
  295
+                else:
  296
+                    return_val = False
1
Alex Gaynor Owner
alex added a note September 22, 2012

return_val = getattr(self.lookup_function, 'none_is_true', False)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/models/sql/matching.py
((70 lines not shown))
  70
+    return instance_value.day == value
  71
+
  72
+
  73
+def _week_day(model, instance_value, value):
  74
+    return instance_value.weekday() == value
  75
+
  76
+
  77
+def _isnull(model, instance_value, value):
  78
+    if value:
  79
+        return instance_value is None
  80
+    else:
  81
+        return instance_value is not None
  82
+
  83
+# This is a special attr/flag to designate that when None is the instance_value
  84
+# due to an inability to follow a set of relationships, True should be returned
  85
+# for the match, as in most cases, the match would be considered False
1
Alex Gaynor Owner
alex added a note September 22, 2012

I can't understand this comment. If I can't I doubt anyone else will be able to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Alex Gaynor
Owner

Overall the patch is ok, but ISTM this should really wait for a cleanup of the __ lookup stuff, because right now this is just adding more code that only works with the fixed set of builtin lookups, rather than moving us towards being able to have custom lookups, as it is the GIS stuff is a bit of a hack just monkeypatching the builtin dict of matchers.

docs/releases/1.5.txt
@@ -90,6 +90,13 @@ In all :doc:`generic class-based views </topics/class-based-views/index>`
90 90
 (or any class-based view inheriting from ``ContextMixin``), the context dictionary
91 91
 contains a ``view`` variable that points to the ``View`` instance.
92 92
 
  93
+Use of Q objects as a predicate test for model instances
  94
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  95
+
  96
+Q objects now provide a :meth:`~django.db.models.Q.matches()
  97
+<django.db.models.Q.matches>` method that will determine whether matches the
1
Florian Apolloner Owner

"Whether matches the" -- missing "the instance" there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/topics/q-objects.txt
((127 lines not shown))
  127
+
  128
+.. method:: Q.matches(instance, [manager=None])
  129
+
  130
+    Returns ``True`` or ``False`` based on whether the instance matches the
  131
+    condition specified by the Q object. If an object specifies multiple
  132
+    managers, a specific manager can be specified, otherwise the default
  133
+    manager is used. The need to specify a specific manager is uncommon - see
  134
+    details on alternate managers below.
  135
+
  136
+.. note::
  137
+
  138
+    When negating a Q object, the result is a new Q object, it is not negated
  139
+    in place. That means it is invalid to both negate a Q object and call
  140
+    matches on it in a simple statement. ie this won't work as expected:
  141
+    ``~my_q.matches(some_instance)``. Instead, assign the negated Q to a new
  142
+    local variable, then call the matches method on that.
2
Florian Apolloner Owner

I don't like assigning to a new variable, (~qobject).matches should work too. And the issue is IMO not that a new Q object is created but that attribute look up comes before bitwise not in the operator precedence list.

Preston Holmes Owner
ptone added a note September 23, 2012

agree - revising

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/topics/q-objects.txt
((146 lines not shown))
  146
+    When to use Q objects to test against some set of objects instead of
  147
+    issuing a separate query to the database will depend on a number of
  148
+    factors, including the number of different conditions you are testing, the
  149
+    number of objects to be tested, and whether a QuerySet may already be
  150
+    evaluated. Profiling and architectural necessity will be your best guides
  151
+    in making the right choice.
  152
+    (https://docs.djangoproject.com/en/dev/ref/models/querysets/#when-querysets-are-evaluated)
  153
+
  154
+Match Compiling and alternate managers
  155
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  156
+
  157
+.. note::
  158
+
  159
+    This section covers API details that aren't commonly used.
  160
+
  161
+Testing whether an instance matches)a condition requires evaluating the lookups
1
Florian Apolloner Owner

s/)/ /

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Anssi Kääriäinen
Owner

Some quick comments about this feature on topical level:

  • In general it will be impossible to make sure that filtering in Python works exactly like it does in SQL. Things like case insensitive matches are locale and DB vendor specific.
  • Many fields/lookups do type conversion to values. An example is Q(date=date.today()).matches(SomeModel(ldate=datetime.now())). If the SomeModel is saved to DB, the model matches (the datetime is converted to date on save). But, as a Python lookup it doesn't.
  • The usual suspects: How does this work for reverse relations, generic foreign keys and models with parents/childs? How about timezone stuff?

In general I am suspicious of claiming that this does the same thing as the lookup would do in ORM. For that reason I would separate this from Q-object. You would compile an obj-matcher from the q-object, and do the matching using that. The matcher object could be documented as doing best-effort matching in Python. The biggest reason for the separation is that this way the expectation that compile_matcher(q).matches(a_model) does the exact same thing as the actual ORM implementation is not so strong.

As for the implementation: The biggest worry for me is the lookup traversal code. I am surprised if that does what the ORM lookup traversal does. Also, this should be tied more tightly to fields so that one could do the type conversions per field.

My initial opinion is that we might want this into core, but this needs to be clearly documented as something not even trying to do the exact same thing as the lookup will do when ran through ORM. We will never get this to do the exact same thing as the ORM lookup does, so lets not claim that.

Preston Holmes
Owner

responding to @akaariai:

First, thanks for the review. I agree that this should be clearly documented as not mirroring the backend behavior 100%. This was originally something outside of Q, but it was suggested by both @jacobian and @malcolmt that it probably belongs as a feature of Q. Because this deals with Django models, but not with any db backend, this exists as a bit of a chimera inside the ORM, but doesn't have anywhere better to live in Django, as for the most part, model = ORM.

The datetime vs date conversion is an issue - but my hunch is that there should be a way to call the field's to_python method - I'm looking into it, and have a failing test described by your situation to work with.

Ideally the value of the field as accessed by simple attribute should be the same on the newly created instance, as for the fetched one. So the situation you raise I think is problematic for more situations than just Q as predicate, but for other potential situations. In general the model should give you the right value when asked - so in the case of your 'usual suspects' it shouldn't be the concern of the q-lookup code, that should be properly handled by the attribute access on the model. The compiling step of Q simply validates a chain of relationships and field name as being a valid attribute traversal path.

added some commits September 26, 2012
Preston Holmes use more field centric conversions
for both the lookup value, and the instance value
field specific methods are used to effect any conversions
7bdf334
Preston Holmes start on reverse relations ba58c8f
Anssi Kääriäinen
Owner

If others have voted to keep this as q-obj functionality, then lets do that. The main point is that lets not advertise this as interchangeable of doing the matching in the DB.

As for reverse lookups, I don't believe this works yet as ORM lookups do. If you have a lookup like Q(friends__age__gte=20)&Q(friends__age__lte=30) the filter must target the same friend of the possibly many friends. This is something the patch should handle, or explicitly throw an error and say that this isn't handled yet. I am not sure if this already works, but at least it looks like this isn't tested. (This is a somewhat strange interpretation, if you have two friends, one aged 19, other 31, the filters will match individually, but not ANDed... but that is what the ORM does)

I am fine with leaving some of the functionality for later, as long as the non-working cases are errors and documented.

I wonder if to_python() is the correct method to use for type conversions. IIRC it can throw ValidationError which seems like a strange error to get from q_object.matches(someobj). Maybe catch and throw TypeError? A better approach might be to define a field method purely for this. This could of course hook to existing methods by default.

In general if this gets merged I expect this to cause a lot of little bugs in type conversions and in trying to match exactly what the ORM does. This feature is likely worth the needed fixing effort.

My feeling is that this will likely not make it into 1.5. I am not saying it is impossible, but to me it seems there is still too much to do, and in addition review & committer time is in short supply.

Preston Holmes
Owner

@akaariai thanks again for the valuable feedback. I have what I think is a technically functional strategy for the reverse relations part - but it is poorly optimized and I'd be hard pressed to think of a case where it would be preferred over testing membership of your instance in a queryset. An alternate method to to_python might make sense, but could probably come up with a cleanup of the way fields and lookups work in general.

So yes - starting to get the feeling that this is perhaps in the "close, but no cigar" category for 1.5 :-/

Preston Holmes
Owner

Postponed for now - pending improvements in field based lookups

Preston Holmes ptone closed this October 24, 2012
Preston Holmes ptone referenced this pull request in ptone/django-predicate October 21, 2013
Open

Merge downstream changes into django-predicate #3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 46 unique commits by 1 author.

Sep 08, 2012
Preston Holmes initial implementation of predicate into Q 3854ad7
Preston Holmes move manager check, add match_compile api 05dd71e
Preston Holmes typo fix 1551a42
Preston Holmes resolve a circular import 4b80f05
Sep 09, 2012
Preston Holmes clean up recursion ebbde80
Preston Holmes fix call to super.__init__ 5612d95
Preston Holmes more correctly match add_filter
since we aren't interested in values during traversal
0a7d769
Preston Holmes handle the case when related objects don't exist a11f1d6
Preston Holmes add initial tests b90bbe1
Preston Holmes Merge branch 'master' into q-predicate
Conflicts:
	django/db/models/sql/query.py
	updates location of lookup_sep constant
9a2febc
Preston Holmes use new lookup_sep constant location 6e92d8b
Sep 10, 2012
Preston Holmes fix up traversal 0dc4d52
Preston Holmes add more tests b9101d4
Preston Holmes initial gis test structure f73d053
Sep 11, 2012
Preston Holmes start on geodjango tests d031a86
Preston Holmes remove bbox helper 3484332
Preston Holmes updating distance tests with stubs 310b0db
Preston Holmes mostly testing stubs 5a7064c
Preston Holmes remove separate testing app no longer used 66285cc
Sep 16, 2012
Preston Holmes gis spatial predicates with tests b51e79d
Sep 17, 2012
Preston Holmes start on distances implementation b0ce713
Sep 19, 2012
Preston Holmes Fixed #18971 -- added root level CONTRIBUTING doc b9faeb2
Preston Holmes added distance lookups with tests f738405
Preston Holmes Merge branch 'q-predicate' of /Users/preston/Projects/code/django-con…
…tributing/test-vagrant/../../forks/django into q-predicate
cec603a
Sep 20, 2012
Preston Holmes add fix and test for bad field name f818406
Sep 21, 2012
Preston Holmes added alternate manager tests 052daf0
Preston Holmes initial pass at docs 934f5c6
Preston Holmes added release note f3732bd
Preston Holmes doc revision 9e27855
Sep 22, 2012
Preston Holmes Merge branch 'master' into q-predicate d9cf3d3
Preston Holmes add negation handling and tests 003b372
Preston Holmes finished wrong manager tests d742df8
Preston Holmes finished first pass at docs a9dc02e
Preston Holmes pep8 and cleanups b5b4b28
Preston Holmes removed some extraneous comments 006721b
Preston Holmes make a local copy before modifying with .update 7afdab4
Preston Holmes chill out on the 80 cols strictness f38b01d
Preston Holmes refactor logic that drifted into unnecessarily complex ef0fa06
Preston Holmes modified traverse_lookup to return value instead of modifying self b5737da
Preston Holmes clarified and tested a negation edge case ff45770
Preston Holmes clarified comment on none_is_true flag 919d303
Sep 23, 2012
Preston Holmes fixed and improved release note 9e65a45
Preston Holmes clarify note on interactions between ~ and matches 0405446
Preston Holmes whitespace cleanup cf18b74
Sep 26, 2012
Preston Holmes use more field centric conversions
for both the lookup value, and the instance value
field specific methods are used to effect any conversions
7bdf334
Preston Holmes start on reverse relations ba58c8f
Something went wrong with that request. Please try again.