Skip to content

Fixed #20625 -- Chainable Manager/QuerySet methods. #1328

Closed
wants to merge 1 commit into from

8 participants

@loic
Django member
loic commented Jul 3, 2013

No description provided.

@mjtamlyn mjtamlyn commented on an outdated diff Jul 4, 2013
docs/topics/db/managers.txt
+ def get_queryset(self):
+ return PersonQuerySet()
+
+ def male(self):
+ return self.get_queryset().male()
+
+ def female(self):
+ return self.get_queryset().female()
+
+ class Person(models.Model):
+ first_name = models.CharField(max_length=50)
+ last_name = models.CharField(max_length=50)
+ sex = models.CharField(max_length=1, choices=(('M', 'Male'), ('F', 'Female')))
+ people = PersonManager()
+
+This example allows you to call both ``men()`` and ``women()`` directly from the manager ``Person.people``.
@mjtamlyn
Django member
mjtamlyn added a note Jul 4, 2013

Text wrapping

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@mjtamlyn mjtamlyn commented on an outdated diff Jul 4, 2013
docs/topics/db/managers.txt
+ opted_out_public_method.manager = False
+
+ # Available on both Manager and QuerySet.
+ def _opted_in_private_method(self):
+ return
+ _opted_in_private_method.manager = True
+
+If you would like to use both a custom ``Manager`` and a custom ``QuerySet``
+you do the following::
+
+ class CustomManager(models.Manager):
+ def manager_only_method(self):
+ return
+
+ class CustomQuerySet(models.QuerySet):
+ base_manager_class = CustomManager
@mjtamlyn
Django member
mjtamlyn added a note Jul 4, 2013

two spaces after =

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@mjtamlyn mjtamlyn and 1 other commented on an outdated diff Jul 4, 2013
django/db/models/query.py
@@ -41,6 +45,48 @@ def __init__(self, model=None, query=None, using=None):
self._prefetch_done = False
self._known_related_objects = {} # {rel_field, {pk: rel_obj}}
+ @classmethod
+ def get_manager_class(cls, base_class=None):
+ """
+ Creates a manager class for this QuerySet class.
+ """
+
+ if base_class is None:
+ base_class = cls.base_manager_class
+ if base_class is None:
+ from django.db.models.manager import Manager as base_class
+
+ def create_method(name):
+ def manager_copy(self, *args, **kwargs):
+ return getattr(self.get_queryset(), name)(*args, **kwargs)
+ manager_copy.__name__ = name
@mjtamlyn
Django member
mjtamlyn added a note Jul 4, 2013

Might be nice to copy __doc__ here as well

@loic
Django member
loic added a note Jul 4, 2013

Good idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@mjtamlyn mjtamlyn and 2 others commented on an outdated diff Jul 4, 2013
django/db/models/manager.py
@@ -55,12 +55,13 @@ class RenameManagerMethods(RenameMethodsBase):
)
-class Manager(six.with_metaclass(RenameManagerMethods)):
+class _Manager(six.with_metaclass(RenameManagerMethods)):
@mjtamlyn
Django member
mjtamlyn added a note Jul 4, 2013

Could we call this BaseManager instead? That's what it is now...

@loic
Django member
loic added a note Jul 4, 2013

I don't feel strongly about this, but I see _Manager as an implementation detail of Manager; calling it BaseManager makes it look like it's part of a public API.

@akaariai named that class, let's see what he thinks.

@aaugustin
Django member
aaugustin added a note Jul 22, 2013

Django doesn't use _-prefixes consistently in general, and never uses them for class names.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@mjtamlyn
Django member
mjtamlyn commented Jul 4, 2013

Overall I love this change!

Just to check cos I'm not that clear on how inspect is working - are classmethods copied over here? I feel perhaps they should not be, but maybe we should be checking that. Similarly what happens to properties - I'm looking at ordered and db in particular.

@loic
Django member
loic commented Jul 4, 2013

@mjtamlyn for classmethods it depends. In PY3 these are not included, in PY2 they are if they are public and their manager attribute is not False. A minor issue is that to set the manager attribute we need to use old style decorator (as_manager = classmethod(as_manager) instead of @classmethod). properties are not included.

I've added a new commit with a test that ensures that the stock Manager gets exactly the same proxy methods as before. I'll leave it in a seperate commit for now, I'll squash the commit when we are close from completion.

Btw thanks for the review. I think I've already made most of the changes you've suggested.

@loic loic commented on an outdated diff Jul 5, 2013
django/db/models/query.py
+ # Copy the method onto the manager.
+ new_methods[name] = create_method(name, method)
+ return new_methods
+
+ @classmethod
+ def _get_manager_class(cls, base_class=None):
+ if base_class is None:
+ base_class = cls.base_manager_class
+ if base_class is None:
+ from django.db.models.manager import Manager as base_class
+ new_methods = cls._get_manager_methods(base_class)
+ manager_cls = type(cls.__name__ + 'Manager', (base_class,), new_methods)
+ manager_cls._queryset_class = cls
+ return manager_cls
+
+ def as_manager(cls, base_class=None):
@loic
Django member
loic added a note Jul 5, 2013

Should we keep the base_class argument for this method?

As documented it's already possible to set the attribute QuerySet.base_manager_class, so it effectively introduces more than one way to do it.

@akaariai: I leave it up to you since it was part of your original proposal.

@loic
Django member
loic added a note Jul 6, 2013

I removed it, I think it makes for a cleaner API. It wasn't documented by an example anyway because I didn't want to show 2 different ways of doing the same thing.

In any case, it would be a lot easier to add it back if people feel a need for it than it would be to deprecate it down the road.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham and 1 other commented on an outdated diff Jul 9, 2013
docs/ref/models/querysets.txt
@@ -1815,6 +1813,15 @@ DO_NOTHING do not prevent taking the fast-path in deletion.
Note that the queries generated in object deletion is an implementation
detail subject to change.
+as_manager
+~~~~~~~~~~
+
+.. classmethod:: as_manager()
+
+Class method that returns a subclass of :class:`~django.db.models.Manager`
+with a copy of the ``QuerySet`` methods. See
+:ref:`automatically create Manager with QuerySet methods <automatically-create-manager-with-queryset-methods>`.
@timgraham
Django member
timgraham added a note Jul 9, 2013

would you be opposed to dropping "automatically" from the discussion of this feature? (just seems a bit verbose and already implied by "generating" / "creating")

@loic
Django member
loic added a note Jul 9, 2013

I don't mind, I'll update all occurrences of "automatically".

@loic
Django member
loic added a note Jul 9, 2013

It seems we use "automatically" together with a variation of "create" quite extensively in managers.txt. We even give the following definition:

Throughout this section, we will use the term "automatic manager" to mean a
manager that Django creates for you...

Do you still think I should make the change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham commented on an outdated diff Jul 9, 2013
docs/releases/1.7.txt
@@ -17,6 +17,13 @@ deprecation process for some features`_.
What's new in Django 1.7
========================
+Automatically generate ``Manager`` subclasses with custom ``QuerySet`` methods.
@timgraham
Django member
timgraham added a note Jul 9, 2013

I don't think headings usually end with periods

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham and 1 other commented on an outdated diff Jul 9, 2013
docs/topics/db/managers.txt
@@ -97,6 +97,105 @@ that list of ``OpinionPoll`` objects with ``num_responses`` attributes.
Another thing to note about this example is that ``Manager`` methods can
access ``self.model`` to get the model class to which they're attached.
+.. _calling-custom-queryset-methods-from-manager:
+
+Calling custom ``QuerySet`` methods from the ``Manager``.
+---------------------------------------------------------
+
+While most methods from the standard ``QuerySet`` are accessible directly
+from the ``Manager``, it's only the case for the extra methods defined on a
+custom ``QuerySet`` if you also implement them on the ``Manager``::
+
+ class PersonQuerySet(models.QuerySet):
@timgraham
Django member
timgraham added a note Jul 9, 2013

there have been commits in the past to not use gender in examples, e.g. 7edf231

@loic
Django member
loic added a note Jul 9, 2013

I was aware of another similar commit 72efdc4; I actually raised that concern on IRC a couple of days ago but didn't get any feedback.

I didn't want to introduce any more models to this document and the Person model was the only one that made sense to show the feature. I think it would be best to have another commit later that cleans at once all such examples from the docs. What do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham commented on an outdated diff Jul 9, 2013
docs/topics/db/managers.txt
@@ -97,6 +97,105 @@ that list of ``OpinionPoll`` objects with ``num_responses`` attributes.
Another thing to note about this example is that ``Manager`` methods can
access ``self.model`` to get the model class to which they're attached.
+.. _calling-custom-queryset-methods-from-manager:
+
+Calling custom ``QuerySet`` methods from the ``Manager``.
+---------------------------------------------------------
+
+While most methods from the standard ``QuerySet`` are accessible directly
+from the ``Manager``, it's only the case for the extra methods defined on a
@timgraham
Django member
timgraham added a note Jul 9, 2013

it's -> this is

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham commented on an outdated diff Jul 9, 2013
docs/topics/db/managers.txt
+ def female(self):
+ return self.get_queryset().female()
+
+ class Person(models.Model):
+ first_name = models.CharField(max_length=50)
+ last_name = models.CharField(max_length=50)
+ sex = models.CharField(max_length=1, choices=(('M', 'Male'), ('F', 'Female')))
+ people = PersonManager()
+
+This example allows you to call both ``men()`` and ``women()`` directly from
+the manager ``Person.people``.
+
+.. _automatically-create-manager-with-queryset-methods:
+.. versionadded:: 1.7
+
+The factory :meth:`QuerySet.as_manager() <django.db.models.query.QuerySet.as_manager>`
@timgraham
Django member
timgraham added a note Jul 9, 2013

remove "the factory"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham commented on an outdated diff Jul 9, 2013
docs/topics/db/managers.txt
+ first_name = models.CharField(max_length=50)
+ last_name = models.CharField(max_length=50)
+ sex = models.CharField(max_length=1, choices=(('M', 'Male'), ('F', 'Female')))
+ people = PersonManager()
+
+This example allows you to call both ``men()`` and ``women()`` directly from
+the manager ``Person.people``.
+
+.. _automatically-create-manager-with-queryset-methods:
+.. versionadded:: 1.7
+
+The factory :meth:`QuerySet.as_manager() <django.db.models.query.QuerySet.as_manager>`
+can be used to create automatically a subclass of ``Manager`` with a copy of the
+``QuerySet`` methods::
+
+ class PersonQuerySet(models.QuerySet):
@timgraham
Django member
timgraham added a note Jul 9, 2013

don't need to reiterate this from the example above IMO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham commented on an outdated diff Jul 9, 2013
docs/topics/db/managers.txt
+.. _automatically-create-manager-with-queryset-methods:
+.. versionadded:: 1.7
+
+The factory :meth:`QuerySet.as_manager() <django.db.models.query.QuerySet.as_manager>`
+can be used to create automatically a subclass of ``Manager`` with a copy of the
+``QuerySet`` methods::
+
+ class PersonQuerySet(models.QuerySet):
+ def male(self):
+ return self.filter(sex='M')
+
+ def female(self):
+ return self.filter(sex='F')
+
+ class Person(models.Model):
+ first_name = models.CharField(max_length=50)
@timgraham
Django member
timgraham added a note Jul 9, 2013

just use ... instead of repeating the fields and just show the 1 new line

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham commented on an outdated diff Jul 9, 2013
tests/basic/tests.py
+ 'only',
+ 'using',
+ 'exists',
+ '_update',
+ ]
+
+ def test_manager_methods(self):
+ """
+ This test ensures that the correct set of methods from `QuerySet`
+ are copied onto `Manager`.
+
+ It's particularly useful to prevent accidentally leaking new methods
+ into `Manager`. New `QuerySet` methods that should also be copied onto
+ `Manager` will need to be added to `ManagerTest.QUERYSET_PROXY_METHODS`.
+ """
+
@timgraham
Django member
timgraham added a note Jul 9, 2013

I think we usually omit a newline after docstrings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@loic
Django member
loic commented Jul 9, 2013

@timgraham: I made most of the suggested changes.

The comment regarding the gender issue has been collapsed because of the updated diff but I replied to it.

There is also a pending remark regarding the use of "automatically".

@mjtamlyn
Django member
mjtamlyn commented Jul 9, 2013

@loic FYI this is no longer merging cleanly. We can probably fix it when we merge it though

@loic
Django member
loic commented Jul 9, 2013

@mjtamlyn thanks for the heads up; I've rebased to the latest master.

@timgraham timgraham commented on the diff Jul 10, 2013
docs/topics/db/managers.txt
+ def male(self):
+ return self.get_queryset().male()
+
+ def female(self):
+ return self.get_queryset().female()
+
+ class Person(models.Model):
+ first_name = models.CharField(max_length=50)
+ last_name = models.CharField(max_length=50)
+ sex = models.CharField(max_length=1, choices=(('M', 'Male'), ('F', 'Female')))
+ people = PersonManager()
+
+This example allows you to call both ``men()`` and ``women()`` directly from
+the manager ``Person.people``.
+
+.. _create-manager-with-queryset-methods:
@timgraham
Django member
timgraham added a note Jul 10, 2013

I'd add a section heading here "Creating Manager instances with QuerySet methods" (you can then remove the custom text from the :ref: links as well) and make it more clear that this approach is intended to replace the above: "In lieu of the above approach which requires duplicating methods on the both the QuerySet and the Manager, ...."

@loic
Django member
loic added a note Jul 10, 2013

Done even though GH doesn't collapse this outdated diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham commented on an outdated diff Jul 10, 2013
docs/topics/db/managers.txt
+
+ class Person(models.Model):
+ first_name = models.CharField(max_length=50)
+ last_name = models.CharField(max_length=50)
+ sex = models.CharField(max_length=1, choices=(('M', 'Male'), ('F', 'Female')))
+ people = PersonManager()
+
+This example allows you to call both ``men()`` and ``women()`` directly from
+the manager ``Person.people``.
+
+.. _create-manager-with-queryset-methods:
+.. versionadded:: 1.7
+
+:meth:`QuerySet.as_manager() <django.db.models.query.QuerySet.as_manager>`
+can be used to create an instance of ``Manager`` with a copy of a custom
+``QuerySet`` methods::
@timgraham
Django member
timgraham added a note Jul 10, 2013

QuerySet's methods

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham and 1 other commented on an outdated diff Jul 10, 2013
docs/topics/db/managers.txt
+.. _create-manager-with-queryset-methods:
+.. versionadded:: 1.7
+
+:meth:`QuerySet.as_manager() <django.db.models.query.QuerySet.as_manager>`
+can be used to create an instance of ``Manager`` with a copy of a custom
+``QuerySet`` methods::
+
+ class Person(models.Model):
+ ...
+ people = PersonQuerySet.as_manager()
+
+The ``Manager`` instance created by :meth:`QuerySet.as_manager()
+<django.db.models.query.QuerySet.as_manager>` will be virtually
+identical to the ``PersonManager`` from the previous example.
+
+Not every ``QuerySet`` method makes sense at the ``Manager`` level, for
@timgraham
Django member
timgraham added a note Jul 10, 2013

For instance should be a separate sentence or use ; instead of ,

@timgraham
Django member
timgraham added a note Jul 10, 2013

don't want to bikeshed, but this detail of which methods are copied seems a little abstract for the topic guide and may be more appropriate for the reference section. I imagine most users won't care about these details to achieve the common use case described above, what do you think? If you think most users will care about this, then could the example be made more concrete?

@loic
Django member
loic added a note Jul 10, 2013

I think it's important to document it somewhere, especially to remove all impression of "magic". I see your point though, but since there is no existing ref for Manager it would be a fair bit of work. A more concrete example would be a challenge since it's really a set of rules.

Btw I'm struggling to get back on #django-dev but NickServ seems to be down.

@timgraham
Django member
timgraham added a note Jul 10, 2013

I thought it could go under the ref entry for QuerySet.as_manager -- having the same problem with IRC.

@loic
Django member
loic added a note Jul 10, 2013

I think it would be out of context over there because that document is already pretty heavy and very much QuerySet orientated. I think the QuerySet.delete() example is sufficient to make that section a bit less abstract.

Without the rules people might be in for a surprise regarding what happens to private methods for example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham timgraham and 1 other commented on an outdated diff Jul 10, 2013
docs/releases/1.7.txt
@@ -17,6 +17,14 @@ deprecation process for some features`_.
What's new in Django 1.7
========================
+Calling custom ``QuerySet`` methods from the ``Manager``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`QuerySet.as_manager() <django.db.models.query.QuerySet.as_manager>`
+class method has been added to :ref:`create Manager instances with QuerySet
+methods <create-manager-with-queryset-methods>`.
+See :ref:`calling-custom-queryset-methods-from-manager` for more details.
@timgraham
Django member
timgraham added a note Jul 10, 2013

do we need this second link? (it links to the section above what's linked in the previous sentence and isn't something new in 1.7)

@loic
Django member
loic added a note Jul 10, 2013

I agree, fixed it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin and 1 other commented on an outdated diff Jul 22, 2013
django/db/models/query.py
@@ -45,6 +55,52 @@ def __init__(self, model=None, query=None, using=None):
self._prefetch_done = False
self._known_related_objects = {} # {rel_field, {pk: rel_obj}}
+ @classmethod
+ def _get_manager_methods(cls, base_class):
+ def create_method(name, method):
+ def manager_method(self, *args, **kwargs):
+ return getattr(self.get_queryset(), name)(*args, **kwargs)
+ manager_method.__name__ = method.__name__
+ manager_method.__doc__ = method.__doc__
+ return manager_method
+
+ new_methods = {}
+ predicate = inspect.isfunction
@aaugustin
Django member
aaugustin added a note Jul 22, 2013

This would be more readable as a one-liner:
predicate = inspect.isfunction if six.PY3 else inspect.ismethod

@loic
Django member
loic added a note Jul 22, 2013

No worries, will update promptly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin commented on the diff Jul 22, 2013
docs/ref/models/querysets.txt
@@ -1866,6 +1864,15 @@ DO_NOTHING do not prevent taking the fast-path in deletion.
Note that the queries generated in object deletion is an implementation
detail subject to change.
+as_manager
+~~~~~~~~~~
+
@aaugustin
Django member
aaugustin added a note Jul 22, 2013

.. versionadded: 1.7

@loic
Django member
loic added a note Jul 22, 2013

Done even though GH doesn't collapse this outdated diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin and 1 other commented on an outdated diff Jul 22, 2013
docs/topics/db/managers.txt
+ opted_out_public_method.manager = False
+
+ # Available on both Manager and QuerySet.
+ def _opted_in_private_method(self):
+ return
+ _opted_in_private_method.manager = True
+
+If you would like to use both a custom ``Manager`` and a custom ``QuerySet``
+you do the following::
+
+ class CustomManager(models.Manager):
+ def manager_only_method(self):
+ return
+
+ class CustomQuerySet(models.QuerySet):
+ base_manager_class = CustomManager
@aaugustin
Django member
aaugustin added a note Jul 22, 2013

I don't like this API: you're putting a class-level attribute in a QuerySet class that will only be used in its as_manager() method. Why not make it an argument of the as_manager()method instead?

@loic
Django member
loic added a note Jul 22, 2013

It was a as_manager() argument at some point but later removed: github comment, trac comment:22, trac comment:23 and trac comment:25.

Having both a custom Manager and a custom QuerySet is a bit of an edge case, especially now that we remove the need for a custom Manager for most cases, so I thought it was better to make the as_manager() signature as simple as possible.

Also, I was leaving room for a as_manager_class() method if the need arises.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin and 1 other commented on an outdated diff Jul 22, 2013
docs/topics/db/managers.txt
+
+The ``Manager`` instance created by :meth:`QuerySet.as_manager()
+<django.db.models.query.QuerySet.as_manager>` will be virtually
+identical to the ``PersonManager`` from the previous example.
+
+Not every ``QuerySet`` method makes sense at the ``Manager`` level; for
+instance we intentionally prevent the :meth:`QuerySet.delete()
+<django.db.models.query.QuerySet.delete>` method from being copied onto
+the ``Manager`` class.
+
+Methods are copied according to the following rules:
+
+- Public methods are copied by default.
+- Private methods (starting with an underscore) are not copied by default.
+- Methods with a `manager` attribute set to `True` are always copied.
+- Methods with a `manager` attribute set to `False` are never copied.
@aaugustin
Django member
aaugustin added a note Jul 22, 2013

Slapping attributes on functions is often a poor API. In this case, the manager attribute is provided for the sole use of the as_manager() method. Why not provide include / exclude arguments to as_manager() that would be list of names of method to copy / not to copy ?

@loic
Django member
loic added a note Jul 22, 2013

I don't like the include / exclude argument idea because it wouldn't work well with inheritance.

@loic
Django member
loic added a note Jul 22, 2013

Talked to @akaariai and he also thinks method attributes are the best option here.

If you really don't want any reference to "manager" from within the QuerySet class, we could rename it as queryset_only and queryset_only = True would be equivalent to manager = False.

Edit: Did the rename in ad64369.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin commented on the diff Jul 22, 2013
docs/topics/db/managers.txt
+ return self.filter(sex='F')
+
+ class PersonManager(models.Manager):
+ def get_queryset(self):
+ return PersonQuerySet()
+
+ def male(self):
+ return self.get_queryset().male()
+
+ def female(self):
+ return self.get_queryset().female()
+
+ class Person(models.Model):
+ first_name = models.CharField(max_length=50)
+ last_name = models.CharField(max_length=50)
+ sex = models.CharField(max_length=1, choices=(('M', 'Male'), ('F', 'Female')))
@aaugustin
Django member
aaugustin added a note Jul 22, 2013

We've removed every example based on gender from the docs, because not every human fits in the male or female category. Please chose another theme.

@loic
Django member
loic added a note Jul 22, 2013

Please see this comment thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin and 1 other commented on an outdated diff Jul 22, 2013
django/db/models/manager.py
def all(self):
+ # This method can't be proxied to QuerySet, as prefetch_related is lost
+ # on clone().
@aaugustin
Django member
aaugustin added a note Jul 22, 2013

You're obviously trying to explain something useful but the comment is still too terse for me to understand it.

@loic
Django member
loic added a note Jul 22, 2013

I originally worded the comment "If we proxy to Queryset.all() we bust the cache used by prefetch_related.", @akaariai edited it. Was it more helpful?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin and 1 other commented on an outdated diff Jul 22, 2013
django/db/models/query.py
+ manager_method.__name__ = method.__name__
+ manager_method.__doc__ = method.__doc__
+ return manager_method
+
+ new_methods = {}
+ predicate = inspect.isfunction
+ if not six.PY3:
+ # Refs http://bugs.python.org/issue1785
+ predicate = inspect.ismethod
+ for name, method in inspect.getmembers(cls, predicate=predicate):
+ # Only copy missing methods.
+ if hasattr(base_class, name):
+ continue
+ # Only copy public methods or methods with the attribute `manager=True`.
+ should_copy = getattr(method, 'manager', None)
+ if should_copy is False or should_copy is None and name.startswith('_'):
@aaugustin
Django member
aaugustin added a note Jul 22, 2013

Please add parentheses around the and block, precedence isn't obvious.

@loic
Django member
loic added a note Jul 22, 2013

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin and 1 other commented on an outdated diff Jul 22, 2013
django/db/models/query.py
+ # Only copy public methods or methods with the attribute `manager=True`.
+ should_copy = getattr(method, 'manager', None)
+ if should_copy is False or should_copy is None and name.startswith('_'):
+ continue
+ # Copy the method onto the manager.
+ new_methods[name] = create_method(name, method)
+ return new_methods
+
+ @classmethod
+ def _get_manager_class(cls, base_class=None):
+ if base_class is None:
+ base_class = cls.base_manager_class
+ if base_class is None:
+ from django.db.models.manager import Manager as base_class
+ new_methods = cls._get_manager_methods(base_class)
+ manager_cls = type(cls.__name__ + 'Manager', (base_class,), new_methods)
@aaugustin
Django member
aaugustin added a note Jul 22, 2013

This inner import is a syndrom of the circular dependency between Manager and QuerySet.

Could we avoid it by moving this code inside _Manager? Manager would know how to create a subclass of itself with the methods of a given QuerySet.

@loic
Django member
loic added a note Jul 22, 2013

Let me toy with the idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@loic
Django member
loic commented Jul 22, 2013

I tested the __new__ idea from @akaariai on the ML, the conclusion is that it'll break inheritance, just like __getattr__ and __getattribute__ did.

The only way to support inheritance is by generating real classes ahead of time, a class factory can do that, the Manager(CustomQueryset) API can't.

Between CustomQuerySet.as_manager() and Manager.from_queryset(CustomQuerySet), I prefer the former by a big margin. I've been writing custom Manager classes extensively and with this patch in core, I don't plan on ever writing a custom Manager. Manager.from_queryset(CustomQuerySet) is more verbose and requires importing a Manager class that we are not going use.

Also I'd like to point out that both base_manager_class and the manager method attribute are there to support advanced usage, most people will never have to worry about them.

@loic
Django member
loic commented Jul 22, 2013

Added two commits which hopefully address the concerns of making QuerySet overly dependent on the Manager class.

I still like better the end user API of CustomQuerySet.as_manager(), but Manager.from_queryset(CustomQuerySet) does have the advantage of removing the need for the base_manager_class attribute. To be completely fair to the as_manager() factory, CustomQueryset.as_manager(CustomManager) could have had the same effect.

I've wanted this general feature pretty much since I've started using Django, so if there is consensus on from_queryset(), count me in.

Edit: One more reason I'm not so keen on the Manager(CustomQuerySet) idea is that we add the burden of handling the queryset_class argument onto anyone with a custom Manager.__init__(), especially considering it's a kwarg that will generally be used as an arg.

@akaariai
Django member

Both as_manager() and from_queryset() work and have some advantages each. as_manager() is slightly shorter and requires one less import in the common case, while from_queryset() is better if you need both custom queryset and manager classes, and doesn't have the circular dependency problem.

My preference is as_manager() because it makes the common case shortest. But, I can go with from_queryset(), too. Most of all I don't want to stall the patch on this issue. Seems like overall there is a slight preference for from_queryset(), so if no further ideas or opinions are presented lets just go with from_queryset().

@loic
Django member
loic commented Jul 23, 2013

Maybe there is a middle ground:

  • Make Manager.from_queryset() return a class, therefore replacing Manager.create_from_manager().
  • Make QuerySet.as_manager() a shortcut for the common case, therefore removing support for custom base Manager which eliminates the need of handling __init__() arguments.

So effectivelly you could do:

class BaseManager(Manager):
    def __init__(self, *args, **kwargs):
        pass

CustomManager = BaseManager.from_queryset(CustomQuerySet)

# For the common case where you only need a custom `QuerySet`:
manager = CustomQuerySet.as_manager()

# When you need a custom `Manager`:
manager = CustomManager(*args, **kwargs)
# or
manager = BaseManager.from_queryset(CustomQuerySet)(*args, **kwargs)

Edit: One thing I like about this approach is that we make it elegant to create a CustomManager class, which will become useful later on when tickets like #14891 are fixed.

Edit 2: d8d3a60 implements this. I'm now nearly convinced this is the way to go.

Edit 3: Touched up the docs which should be mostly accurate again, we may need to update the release notes to mention from_queryset() depending on how much we want to promote the advanced usage.

@carljm
Django member
carljm commented Jul 23, 2013

What is the argument for QuerySet.as_manager() (vs just using Manager.from_queryset in all cases)? Simply to save the user one import? It doesn't even do that; in your models file you almost certainly have from django.db import models already, so it's just models.Manager.from_queryset(MyCustomQuerySet)() with no additional import needed. Is it to save the additional parentheses?

Personally I don't see sufficient justification there for adding manager-specific API to queryset and making the dependency between those classes bidirectional.

@loic
Django member
loic commented Jul 23, 2013

My argument is that this feature will pretty much deprecate the concept of manager as we know it. And when most of the time all we need is MyQueryset.as_manager(), we will be wondering why the API is as clunky as models.Manager.from_queryset(MyQueryset)().

This new feature opens up a lot of new possibilities, QuerySets that generate disposable Managers are much more flexible than custom Managers ever were. Look at how many projects provide custom Managers, yet they are inflexible and can't work with each other.

With this new feature we can do:

class CompositeQuerySet(RatedQuerySet, PublishedQuerySet, SiteRelatedQuerySet):
    pass
objects = CompositeQuerySet.as_manager()

Which replaces:

class RatingManager(Manager):
    ...
class PublishingManager(Manager):
    ...
class SiteManager(Manager):
    ...
objects = ? # Pick one only.

Combining multiple QuerySets, even if they are provided by different apps, makes them work together; you can effectively chain the filtering provided by each of them and start the chain all the way from the model manager instance. I don't see why we should put the legacy class in the middle of this API.

Yet this patch ensures that anyone who really want to create a custom Manager type from a custom QuerySet can do so in an elegant way, but we have to realize that it's going to be an exception, not the norm.

Edit: Reworked the example and changed wording to some extent.

Edit 2: For the record, I do not suggest we deprecate the Manager class, nor do I think it'll become obsolete, quite the contrary. If anything I want to see more places where we can customize them. What I believe will become obsolete is the way we've been defining them and using them.

@shaib shaib and 2 others commented on an outdated diff Jul 24, 2013
django/db/models/manager.py
+ continue
+ # Only copy public methods or methods with the attribute `queryset_only=False`.
+ queryset_only = getattr(method, 'queryset_only', None)
+ if queryset_only is True or (queryset_only is None and name.startswith('_')):
+ continue
+ # Copy the method onto the manager.
+ new_methods[name] = create_method(name, method)
+ return new_methods
+
+ @classmethod
+ def from_queryset(cls, queryset_class):
+ class_dict = {
+ '_queryset_class': queryset_class,
+ }
+ class_dict.update(cls._get_queryset_methods(queryset_class))
+ return type(cls.__name__, (cls,), class_dict)
@shaib
shaib added a note Jul 24, 2013

I'd try to build a more informative name, using also the queryset_class's name; ("%sFrom%s" % (cls.__name__,queryset_class.__name__)) seems reasonable. True, normally you won't see the class name in code, but it will be helpful in tracebacks and Django debug screens where locals are printed.

@carljm
Django member
carljm added a note Jul 24, 2013

I agree with this suggestion; it's a small change but it could really help debugging. Without it, all manager classes created from querysets will just be called "Manager", which could be quite confusing if you're working with more than one of them in a PDB session or a traceback.

@loic
Django member
loic added a note Jul 24, 2013

Yep, I'm on it already, running the test suite.

@loic
Django member
loic added a note Jul 24, 2013

Done. Please review the implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shaib shaib and 1 other commented on an outdated diff Jul 24, 2013
django/db/models/manager.py
+ def manager_method(self, *args, **kwargs):
+ return getattr(self.get_queryset(), name)(*args, **kwargs)
+ manager_method.__name__ = method.__name__
+ manager_method.__doc__ = method.__doc__
+ return manager_method
+
+ new_methods = {}
+ # Refs http://bugs.python.org/issue1785.
+ predicate = inspect.isfunction if six.PY3 else inspect.ismethod
+ for name, method in inspect.getmembers(queryset_class, predicate=predicate):
+ # Only copy missing methods.
+ if hasattr(cls, name):
+ continue
+ # Only copy public methods or methods with the attribute `queryset_only=False`.
+ queryset_only = getattr(method, 'queryset_only', None)
+ if queryset_only is True or (queryset_only is None and name.startswith('_')):
@shaib
shaib added a note Jul 24, 2013

Why the limitation to is True and not just if queryset_only?
And then, why the limitation to queryset_only is None (which even contradicts the comment two lines up)?
The line should just read if queryset_only or name.startswith('_'):, IMO

@loic
Django member
loic added a note Jul 24, 2013

Right, this variable used to be called "manager" and worked the other way around, I just changed the False to True, but now we can simplify it.

@shaib
shaib added a note Jul 24, 2013

Following your note on _update below -- keep the queryset_only is None part; otherwise you can't override the default to avoid copying private methods.

@loic
Django member
loic added a note Jul 24, 2013

That would be a good way to test the test suite :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shaib shaib commented on the diff Jul 24, 2013
django/db/models/manager.py
self._set_creation_counter()
self.model = None
self._inherited = False
self._db = None
+ @classmethod
+ def _get_queryset_methods(cls, queryset_class):
+ def create_method(name, method):
+ def manager_method(self, *args, **kwargs):
+ return getattr(self.get_queryset(), name)(*args, **kwargs)
+ manager_method.__name__ = method.__name__
+ manager_method.__doc__ = method.__doc__
@shaib
shaib added a note Jul 24, 2013

Decorate manager_method with functools.wraps instead of these 2 lines? Note that as the code stands, the method argument is unused. So, I suggest:

    def create_method(method):
         @functools.wraps(method)
         def manager_method(self, *args, **kwargs):
             return method(self.get_queryset(), *args, **kwargs)
         return manager_method

Is there an edge-case that is better served by getattr?

@loic
Django member
loic added a note Jul 24, 2013

Good suggestion.

@loic
Django member
loic added a note Jul 24, 2013

It creates a failing test in custom_managers.tests.CustomManagerTests.test_manager.

If you want to investigate, that might be one of those edge-cases.

@loic
Django member
loic added a note Jul 24, 2013

I investigated and the reason this doesn't work is that create_method() is only called when the method is missing. So basically if you have CustomQuerySet that overrides filter() and you call method rather than targeting it by name you'll always call QuerySet.filter() and completely bypass CustomQuerySet.filter().

I hope that makes sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shaib shaib and 2 others commented on an outdated diff Jul 24, 2013
django/db/models/manager.py
def raw(self, raw_query, params=None, *args, **kwargs):
return RawQuerySet(raw_query=raw_query, model=self.model, params=params, using=self._db, *args, **kwargs)
+Manager = BaseManager.from_queryset(QuerySet)
@shaib
shaib added a note Jul 24, 2013

Is the separation between Manager and BaseManager necessary?

Why would I chose to inherit directly from BaseManager?

@loic
Django member
loic added a note Jul 24, 2013

Here we are basically dogfooding the new feature to remove all the legacy proxy methods.

BaseManager is missing all the QuerySet proxy method so you shouldn't inherit from it. It was called _Manager at some point to highlight that it's not part of a public API, but at least 2 core devs didn't like it.

@shaib
shaib added a note Jul 24, 2013

Well then, might be bikeshedding, but in that case I'd just use the name Manager all the way through; Manager = Manager.from_queryset(QuerySet) should work fine, and will avoid the wrong message about a Manager class without queryset methods.

@loic
Django member
loic added a note Jul 24, 2013

That will make testing more difficult, see test in tests/basic/tests.py.

@shaib
shaib added a note Jul 24, 2013

How would that change anything there?

@loic
Django member
loic added a note Jul 24, 2013

I wouldn't be able to import BaseManager anymore, and calling Manager._get_queryset_methods(QuerySet) wouldn't yield any result since all the methods would already be in place.

@carljm
Django member
carljm added a note Jul 24, 2013

FWIW, I think it would be a bad idea to have two different classes in the same module called Manager, with the definition of the second overriding the first. Separate names is much clearer as to what's going on. If we don't document BaseManager I see no significant risk of confusion; Django's codebase is full of undocumented BaseFoo classes that are an implementation detail, and I have never seen user confusion as a result.

@shaib
shaib added a note Jul 24, 2013

Yes, I realized that a little after I posted but I had to run off.
I suspect this points to a worthwhile refactoring of _get_queryset_methods to a function, say get_exportable_methods, that is defined next to the QuerySet class, and takes a QuerySet class and a filter. That would keep all handling of the queryset_only flags in django.db.models.query, which I'd find preferable.
But I wouldn't push for that too hard.

@loic
Django member
loic added a note Jul 24, 2013

That's exactly where it was before...

@shaib
shaib added a note Jul 24, 2013

Ok, Carl's argument is convincing enough for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shaib shaib and 1 other commented on an outdated diff Jul 24, 2013
django/db/models/query.py
@@ -30,10 +30,17 @@
EmptyResultSet = sql.EmptyResultSet
+def _pickle_queryset(class_bases, class_dict):
+ new = Empty()
@shaib
shaib added a note Jul 24, 2013

The use of Empty that is imported from models.fields looks very weird until one looks at its definition.
Can't you use object() directly?

Edit: No, you can't, Python2 won't let you. Still, the use of a class imported from fields feels hackish.

@loic
Django member
loic added a note Jul 24, 2013

Nop you can't. Just like other basic types like int or dict you can't add attributes dynamically to object.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shaib shaib commented on the diff Jul 24, 2013
django/db/models/query.py
@@ -30,10 +30,17 @@
EmptyResultSet = sql.EmptyResultSet
+def _pickle_queryset(class_bases, class_dict):
@shaib
shaib added a note Jul 24, 2013

Some explanation on why this is needed would be nice

@loic
Django member
loic added a note Jul 24, 2013

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shaib shaib commented on the diff Jul 24, 2013
django/db/models/query.py
@@ -45,6 +52,13 @@ def __init__(self, model=None, query=None, using=None):
self._prefetch_done = False
self._known_related_objects = {} # {rel_field, {pk: rel_obj}}
+ def as_manager(cls):
+ # Address the circular dependency between `Queryset` and `Manager`.
+ from django.db.models.manager import Manager
+ return Manager.from_queryset(cls)()
+ as_manager.queryset_only = True
+ as_manager = classmethod(as_manager)
@shaib
shaib added a note Jul 24, 2013

Why not use decorator syntax for classmethod?
While you're at it, why not hide the queryset_only attribute in a decorator:

def queryset_only(method):
     method.queryset_only = True
     return method
@loic
Django member
loic added a note Jul 24, 2013

"Why not use decorator syntax for classmethod?", because it won't work on python 2, it'll breaks when you try to assign as_manager.queryset_only = True

For the queryset_only() decorator, I'd like to avoid adding extra cruft, it will require documenting it and importing it before use.

@shaib
shaib added a note Jul 24, 2013

That's not "cruft" -- that's making user code look like it was written after Python 2.4 was released.
And also,

@classmethod
@queryset_only
def as_manager(...

does work.

@loic
Django member
loic added a note Jul 24, 2013

TBH I think this is bikeshedding, queryset_only usage should be very rare in practice.

@loic
Django member
loic added a note Jul 25, 2013

@aaugustin thought I dismissed your idea a bit hastily, so I gave it another thought. The problem with @queryset_only is that we'll need the opposite decorator for the opt-in mechanism. Or it could take a boolean argument @queryset_only(False) but IMO a real boolean attribute is more explicit. Also I think we can easily add it later shall people want it since it's just syntactic sugar around a method attribute anyway. Finally I'd like to stress how uncommon it should be, there are very few public QuerySet methods that don't make sense at the Manager level: out of 36 methods that we've removed from Manager, we only use this twice and that's for the base implementation. That said, if more people feel strongly about this, I'll add it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shaib shaib commented on the diff Jul 24, 2013
django/db/models/query.py
@@ -567,6 +595,7 @@ def _update(self, values):
self._result_cache = None
return query.get_compiler(self.db).execute_sql(None)
_update.alters_data = True
+ _update.queryset_only = False
@shaib
shaib added a note Jul 24, 2013

Not required (starts with _)

@loic
Django member
loic added a note Jul 24, 2013

This method should be copied over, hence queryset_only set to False.

@shaib
shaib added a note Jul 24, 2013

Right. I corrected my oversimplifying comment above accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shaib shaib and 1 other commented on an outdated diff Jul 24, 2013
django/db/models/manager.py
+ if hasattr(cls, name):
+ continue
+ # Only copy public methods or methods with the attribute `queryset_only=False`.
+ queryset_only = getattr(method, 'queryset_only', None)
+ if queryset_only or (queryset_only is None and name.startswith('_')):
+ continue
+ # Copy the method onto the manager.
+ new_methods[name] = create_method(name, method)
+ return new_methods
+
+ @classmethod
+ def from_queryset(cls, queryset_class):
+ # Since we are dogfooding `from_queryset()` to create `Manager` we need
+ # a special case here to ensure it retains its original "Manager" name
+ # rather than "BaseManagerFromQuerySet".
+ if cls is BaseManager and queryset_class is QuerySet:
@shaib
shaib added a note Jul 24, 2013

Instead of this special-casing, take an optional class_name argument. Might turn out useful for other people.

(for me, this comment is evidence for the value of dogfooding)

@loic
Django member
loic added a note Jul 24, 2013

Good idea, I made the change. I'm not sure I understand your comment about dogfooding.

@shaib
shaib added a note Jul 24, 2013

Never mind the comment. Looking good now.

@loic
Django member
loic added a note Jul 24, 2013

Thinking about it a bit more, I could have just set Manager.__name__ = 'Manager' after the Manager = BaseManager.from_queryset() call. Are you convinced we need the extra argument?

@shaib
shaib added a note Jul 24, 2013

I think it's a better API; a custom manager could be using its name in a __new__ method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@davidszotten davidszotten and 1 other commented on an outdated diff Jul 24, 2013
docs/topics/db/managers.txt
+
+The ``Manager`` instance created by :meth:`QuerySet.as_manager()
+<django.db.models.query.QuerySet.as_manager>` will be virtually
+identical to the ``PersonManager`` from the previous example.
+
+Not every ``QuerySet`` method makes sense at the ``Manager`` level; for
+instance we intentionally prevent the :meth:`QuerySet.delete()
+<django.db.models.query.QuerySet.delete>` method from being copied onto
+the ``Manager`` class.
+
+Methods are copied according to the following rules:
+
+- Public methods are copied by default.
+- Private methods (starting with an underscore) are not copied by default.
+- Methods with a `queryset_only` attribute set to `True` are always copied.
+- Methods with a `queryset_only` attribute set to `False` are never copied.
@davidszotten
davidszotten added a note Jul 24, 2013

are these the wrong way around?

@loic
Django member
loic added a note Jul 25, 2013

Good catch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@loic loic Fixed #20625 -- Chainable Manager/QuerySet methods.
Additionally this patch solves the orthogonal problem that specialized
`QuerySet` like `ValuesQuerySet` didn't inherit from the current `QuerySet`
type. This wasn't an issue until now because we didn't officially support
custom `QuerySet` but it became necessary with the introduction of this new
feature.

Thanks aaugustin, akaariai, carljm, charettes, mjtamlyn, shaib and timgraham
for the reviews.
4604ef2
@timgraham
Django member

merged in 31fadc1

@timgraham timgraham closed this Jul 29, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.