Skip to content

Fixed #901 -- Added Model.refresh_from_db() method #2882

Closed
wants to merge 2 commits into from

8 participants

@akaariai
Django member
akaariai commented Jul 5, 2014

No description provided.

@slurms slurms and 1 other commented on an outdated diff Jul 23, 2014
django/db/models/base.py
@@ -546,6 +547,68 @@ def _set_pk_val(self, value):
pk = property(_get_pk_val, _set_pk_val)
+ def get_deferred_fields(self):
+ """
+ Return a set containing names of deferred fields for this instance.
+ """
+ return set(
+ f.attname for f in self._meta.concrete_fields
+ if isinstance(self.__class__.__dict__.get(f.attname), DeferredAttribute))
+
+ def refresh_from_db(self, using=None, fields=None, **kwargs):
+ """
+ Reloads field values from the database.
+
+ By default the reloading happens from the database this instance was
+ realoaded, or by using read router if this instance wasn't loaded from
@slurms
slurms added a note Jul 23, 2014

Small typo (realoaded -> reloaded) and some grammatical fixes perhaps?
"By default, the reloading happens from the database this instance was loaded from, or by the using read router if this instance wasn't loaded from the database. The using parameter will override the default."

@carljm
Django member
carljm added a note Nov 19, 2014

Looks like this still needs to be addressed? The suggested re-wording is an improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@dbrgn
dbrgn commented Sep 24, 2014

Any news on this? Would love this feature to be in a future Django version.

@akaariai
Django member

buildbot, retest this please

@dbrgn dbrgn commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
@@ -116,6 +116,68 @@ The example above shows a full ``from_db()`` implementation to clarify how that
is done. In this case it would of course be possible to just use ``super()`` call
in the ``from_db()`` method.
+Refreshing objects from database
+================================
+
+.. method:: Model.refresh_from_db(using=None, fields=None)
+
+.. versionadded:: 1.8
+
+If you need to reload a model's values from the database you can use the
+``reload_from_db`` method. When this method is called without arguments the
@dbrgn
dbrgn added a note Nov 19, 2014

Shouldn't this be refresh instead of reload?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@dbrgn dbrgn commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+ obj = MyModel.objects.create(val=1)
+ MyModel.objects.filter(pk=obj.pk).update(val=F('val') + 1)
+ # At this point obj.val is still 1, but the value in the database
+ # was updated to 2. The object's updated value needs to be reloaded
+ # from the database.
+ obj.refresh_from_db()
+ self.assertEqual(obj.val, 2)
+
+Note that when deferred fields are accessed, the loading of the deferred
+field's value happens through this method. Thus it is possible to customize
+the way deferred loading happens. The example below shows how one can reload
+all of the instance's fields when a deferred field is reloaded::
+
+
+ class ExampleModel(models.Model):
+ def reload_from_db(self, using=None, fields=None):
@dbrgn
dbrgn added a note Nov 19, 2014

reload -> refresh again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@dbrgn dbrgn commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+
+ def test_update_result(self):
+ obj = MyModel.objects.create(val=1)
+ MyModel.objects.filter(pk=obj.pk).update(val=F('val') + 1)
+ # At this point obj.val is still 1, but the value in the database
+ # was updated to 2. The object's updated value needs to be reloaded
+ # from the database.
+ obj.refresh_from_db()
+ self.assertEqual(obj.val, 2)
+
+Note that when deferred fields are accessed, the loading of the deferred
+field's value happens through this method. Thus it is possible to customize
+the way deferred loading happens. The example below shows how one can reload
+all of the instance's fields when a deferred field is reloaded::
+
+
@dbrgn
dbrgn added a note Nov 19, 2014

Even though it's a small detail, one of these blank lines could probably be removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
@@ -116,6 +116,68 @@ The example above shows a full ``from_db()`` implementation to clarify how that
is done. In this case it would of course be possible to just use ``super()`` call
in the ``from_db()`` method.
+Refreshing objects from database
+================================
+
+.. method:: Model.refresh_from_db(using=None, fields=None)
+
+.. versionadded:: 1.8
+
+If you need to reload a model's values from the database you can use the
+``reload_from_db`` method. When this method is called without arguments the
+following is done:
+
+1. All non deferred fields of the model are loaded from the database to a
@carljm
Django member
carljm added a note Nov 19, 2014

I would try to re-word this without reference to the "new instance", which is an internal implementation detail which may confuse users; no "new instance" is part of the public API of this method. Perhaps combine 1 and 2 into: "All non-deferred fields of the model are updated to the values currently present in the database."?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
@@ -116,6 +116,68 @@ The example above shows a full ``from_db()`` implementation to clarify how that
is done. In this case it would of course be possible to just use ``super()`` call
in the ``from_db()`` method.
+Refreshing objects from database
+================================
+
+.. method:: Model.refresh_from_db(using=None, fields=None)
+
+.. versionadded:: 1.8
+
+If you need to reload a model's values from the database you can use the
+``reload_from_db`` method. When this method is called without arguments the
+following is done:
+
+1. All non deferred fields of the model are loaded from the database to a
+ new instance.
+2. Field values from the new instance are copied to the reloaded instance.
+3. Those loaded related instance for which the relation's value isn't valid
@carljm
Django member
carljm added a note Nov 19, 2014

instances

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+Refreshing objects from database
+================================
+
+.. method:: Model.refresh_from_db(using=None, fields=None)
+
+.. versionadded:: 1.8
+
+If you need to reload a model's values from the database you can use the
+``reload_from_db`` method. When this method is called without arguments the
+following is done:
+
+1. All non deferred fields of the model are loaded from the database to a
+ new instance.
+2. Field values from the new instance are copied to the reloaded instance.
+3. Those loaded related instance for which the relation's value isn't valid
+ any more are removed from the reloaded instance. For example if you have
@carljm
Django member
carljm added a note Nov 19, 2014

Comma after 'example'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+================================
+
+.. method:: Model.refresh_from_db(using=None, fields=None)
+
+.. versionadded:: 1.8
+
+If you need to reload a model's values from the database you can use the
+``reload_from_db`` method. When this method is called without arguments the
+following is done:
+
+1. All non deferred fields of the model are loaded from the database to a
+ new instance.
+2. Field values from the new instance are copied to the reloaded instance.
+3. Those loaded related instance for which the relation's value isn't valid
+ any more are removed from the reloaded instance. For example if you have
+ foreign key from reloaded instance to other model with name ``author``,
@carljm
Django member
carljm added a note Nov 19, 2014

if you have a foreign key named "author" from the reloaded instance to _an_other model

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+
+.. method:: Model.refresh_from_db(using=None, fields=None)
+
+.. versionadded:: 1.8
+
+If you need to reload a model's values from the database you can use the
+``reload_from_db`` method. When this method is called without arguments the
+following is done:
+
+1. All non deferred fields of the model are loaded from the database to a
+ new instance.
+2. Field values from the new instance are copied to the reloaded instance.
+3. Those loaded related instance for which the relation's value isn't valid
+ any more are removed from the reloaded instance. For example if you have
+ foreign key from reloaded instance to other model with name ``author``,
+ then if ``obj.author_id != obj.author.id``, then obj.author will be thrown
@carljm
Django member
carljm added a note Nov 19, 2014

remove ", then"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+.. method:: Model.refresh_from_db(using=None, fields=None)
+
+.. versionadded:: 1.8
+
+If you need to reload a model's values from the database you can use the
+``reload_from_db`` method. When this method is called without arguments the
+following is done:
+
+1. All non deferred fields of the model are loaded from the database to a
+ new instance.
+2. Field values from the new instance are copied to the reloaded instance.
+3. Those loaded related instance for which the relation's value isn't valid
+ any more are removed from the reloaded instance. For example if you have
+ foreign key from reloaded instance to other model with name ``author``,
+ then if ``obj.author_id != obj.author.id``, then obj.author will be thrown
+ away, and it will be reloaded when accessed with the new ``obj.author_id``
@carljm
Django member
carljm added a note Nov 19, 2014

"and when next accessed it will be reloaded with"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+If you need to reload a model's values from the database you can use the
+``reload_from_db`` method. When this method is called without arguments the
+following is done:
+
+1. All non deferred fields of the model are loaded from the database to a
+ new instance.
+2. Field values from the new instance are copied to the reloaded instance.
+3. Those loaded related instance for which the relation's value isn't valid
+ any more are removed from the reloaded instance. For example if you have
+ foreign key from reloaded instance to other model with name ``author``,
+ then if ``obj.author_id != obj.author.id``, then obj.author will be thrown
+ away, and it will be reloaded when accessed with the new ``obj.author_id``
+ value.
+
+The reloading happens from the database the instance was loaded from, or from
+default database if the instance wasn't loaded from database. The ``using``
@carljm
Django member
carljm added a note Nov 19, 2014

the default database

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+2. Field values from the new instance are copied to the reloaded instance.
+3. Those loaded related instance for which the relation's value isn't valid
+ any more are removed from the reloaded instance. For example if you have
+ foreign key from reloaded instance to other model with name ``author``,
+ then if ``obj.author_id != obj.author.id``, then obj.author will be thrown
+ away, and it will be reloaded when accessed with the new ``obj.author_id``
+ value.
+
+The reloading happens from the database the instance was loaded from, or from
+default database if the instance wasn't loaded from database. The ``using``
+argument can be used to force the database used for reloading.
+
+It is possible to force the set of fields to be loaded by using the ``fields``
+argument.
+
+For example, to test that an ``update()`` call resulted in expected update,
@carljm
Django member
carljm added a note Nov 19, 2014

in the expected update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm carljm commented on an outdated diff Nov 19, 2014
docs/ref/models/instances.txt
+
+
+ class ExampleModel(models.Model):
+ def reload_from_db(self, using=None, fields=None):
+ if fields is not None:
+ fields = set(fields)
+ deferred_fields = self.get_deferred_fields()
+ # If any deferred field is going to be loaded
+ if set(fields).intersection(deferred_fields):
+ # then load all of them
+ fields = fields.union(deferred_fields)
+ super(ExampleModel, self).refresh_from_db(using, fields, **kwargs)
+
+.. method:: Model.get_deferred_fields()
+
+A helper method that returns the attribute names of all those fields that
@carljm
Django member
carljm added a note Nov 19, 2014

clarify that the returned value is a set?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carljm
Django member
carljm commented Nov 19, 2014

Code looks great to me! Some cleanup needed in docstrings and docs.

@aaugustin aaugustin and 1 other commented on an outdated diff Nov 20, 2014
docs/releases/1.8.txt
@@ -99,6 +99,14 @@ Minor features
<django.contrib.admin.ModelAdmin.show_full_result_count>` to control whether
or not the full count of objects should be displayed on a filtered admin page.
+* It is now possible to refresh :class:`~django.db.models.Model` instances by
+ using
+ :meth:`Model.refresh_from_db <django.db.models.Model.refresh_from_db>`
+
+* You can now get the set of deferred fields for a :class:`~django.db.models.Model`
+ from
+ :meth:`Model.get_deferred_fields <django.db.models.Model.get_deferred_fields>`
@aaugustin
Django member
aaugustin added a note Nov 20, 2014

I wouldn't document this API until we've seen the results of the Model._meta GSoC.

@timgraham
Django member
timgraham added a note Nov 20, 2014

Nothing comes to find here from what I've reviewed so far, but \cc @PirosB3. _meta seems more like "static info" about model fields rather than something that might vary for each instance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aaugustin aaugustin and 1 other commented on an outdated diff Nov 20, 2014
tests/basic/tests.py
@@ -713,3 +713,60 @@ def get_queryset(self):
asos.save(update_fields=['pub_date'])
finally:
Article._base_manager.__class__ = orig_class
+
+
+class ModelRefreshTests(TestCase):
+ def _truncate_ms(self, val):
+ # MySQL removes microseconds from the datetimes which can cause problems
+ # when comparing the original value to that loaded from DB.
+ return val - timedelta(microseconds=val.microsecond)
@aaugustin
Django member
aaugustin added a note Nov 20, 2014

If memory serves, we fixed that recently. Can you check if it's still required?

@timgraham
Django member
timgraham added a note Nov 20, 2014

Yes, but we still support older versions of MySQL so this doesn't hurt. I am adding a comment with the version of MySQL it's applicable for so we can drop it someday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham
Django member

Ready to commit with minor edits, pending confirmation of get_deferred_fields() API.

diff --git a/django/db/models/base.py b/django/db/models/base.py
index 8111d19..b43fa24 100644
--- a/django/db/models/base.py
+++ b/django/db/models/base.py
@@ -555,35 +555,36 @@ class Model(six.with_metaclass(ModelBase)):

     def get_deferred_fields(self):
         """
-        Return a set containing names of deferred fields for this instance.
+        Returns a set containing names of deferred fields for this instance.
         """
-        return set(
+        return {
             f.attname for f in self._meta.concrete_fields
-            if isinstance(self.__class__.__dict__.get(f.attname), DeferredAttribute))
+            if isinstance(self.__class__.__dict__.get(f.attname), DeferredAttribute)
+        }

     def refresh_from_db(self, using=None, fields=None, **kwargs):
         """
         Reloads field values from the database.

-        By default the reloading happens from the database this instance was
-        realoaded, or by using read router if this instance wasn't loaded from
-        database. The using parameter can be used to override the default.
+        By default, the reloading happens from the database this instance was
+        loaded from, or by the `using` read router if this instance wasn't
+        loaded from the database. The using parameter will override the default.

-        Fields can be used to specify which fields to reload. The fields
-        should be an iterable of field attnames. If fields is None, then
-        all non-deferred fields are reloaded.
+        `fields` can be used to specify which fields to reload. It should be an
+        iterable of field attnames. If fields is None, then all non-deferred
+        fields are reloaded.

-        When accessing deferred field of an instance, the deferred loading
-        of the field will call this method when executing the loading.
+        When accessing deferred fields of an instance, the deferred loading
+        of the field will call this method.
         """
         if fields is not None:
             if len(fields) == 0:
                 return
             if any(LOOKUP_SEP in f for f in fields):
                 raise ValueError(
-                    'Found "%s" in fields argument. Relations and transforms not '
-                    'allowed in fields!' % LOOKUP_SEP)
-
+                    'Found "%s" in fields argument. Relations and transforms '
+                    'are not allowed in fields.' % LOOKUP_SEP
+                )
         db = using if using is not None else self._state.db
         if self._deferred:
             non_deferred_model = self._meta.proxy_for_model
diff --git a/docs/ref/models/instances.txt b/docs/ref/models/instances.txt
index 400ccc6..f95c2e4 100644
--- a/docs/ref/models/instances.txt
+++ b/docs/ref/models/instances.txt
@@ -119,25 +119,25 @@ in the ``from_db()`` method.
 Refreshing objects from database
 ================================

-.. method:: Model.refresh_from_db(using=None, fields=None)
+.. method:: Model.refresh_from_db(using=None, fields=None, **kwargs)

 .. versionadded:: 1.8

-If you need to reload a model's values from the database you can use the
-``refresh_from_db`` method. When this method is called without arguments the
+If you need to reload a model's values from the database, you can use the
+``refresh_from_db()`` method. When this method is called without arguments the
 following is done:

 1. All non-deferred fields of the model are updated to the values currently
    present in the database.
-2. Those loaded related instances for which the relation's value isn't valid
-   any more are removed from the reloaded instance. For example, if you have
-   foreign key from the reloaded instance to other model with name ``author``,
-   then if ``obj.author_id != obj.author.id``, obj.author will be thrown
-   away, and when next accessed it will be reloaded with ``obj.author_id``
-   value.
+2. The previously loaded related instances for which the relation's value is no
+   longer valid are removed from the reloaded instance. For example, if you have
+   a foreign key from the reloaded instance to another model with name
+   ``Author``, then if ``obj.author_id != obj.author.id``, ``obj.author`` will
+   be thrown away, and when next accessed it will be reloaded with the value of
+   ``obj.author_id``.

 The reloading happens from the database the instance was loaded from, or from
-the default database if the instance wasn't loaded from database. The
+the default database if the instance wasn't loaded from the database. The
 ``using`` argument can be used to force the database used for reloading.

 It is possible to force the set of fields to be loaded by using the ``fields``
@@ -161,20 +161,22 @@ the way deferred loading happens. The example below shows how one can reload
 all of the instance's fields when a deferred field is reloaded::

     class ExampleModel(models.Model):
-        def refresh_from_db(self, using=None, fields=None):
+        def refresh_from_db(self, using=None, fields=None, **kwargs):
             # fields contains the name of the deferred field to be
             # loaded.
             if fields is not None:
                 fields = set(fields)
                 deferred_fields = self.get_deferred_fields()
                 # If any deferred field is going to be loaded
-                if set(fields).intersection(deferred_fields):
+                if fields.intersection(deferred_fields):
                     # then load all of them
                     fields = fields.union(deferred_fields)
             super(ExampleModel, self).refresh_from_db(using, fields, **kwargs)

 .. method:: Model.get_deferred_fields()

+.. versionadded:: 1.8
+
 A helper method that returns a set containing the attribute names of all those
 fields that are currently deferred for this model.

diff --git a/docs/releases/1.8.txt b/docs/releases/1.8.txt
index c9fd427..e88d8a5 100644
--- a/docs/releases/1.8.txt
+++ b/docs/releases/1.8.txt
@@ -100,12 +100,11 @@ Minor features
   or not the full count of objects should be displayed on a filtered admin page.

 * It is now possible to refresh :class:`~django.db.models.Model` instances by
-  using
-  :meth:`Model.refresh_from_db <django.db.models.Model.refresh_from_db>`
+  using :meth:`Model.refresh_from_db() <django.db.models.Model.refresh_from_db>`.

-* You can now get the set of deferred fields for a :class:`~django.db.models.Model`
-  from
-  :meth:`Model.get_deferred_fields <django.db.models.Model.get_deferred_fields>`
+* You can now get the set of deferred fields for a
+  :class:`~django.db.models.Model` from :meth:`Model.get_deferred_fields()
+  <django.db.models.Model.get_deferred_fields>`.

 :mod:`django.contrib.admindocs`
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/tests/basic/tests.py b/tests/basic/tests.py
index 4db793b..b242649 100644
--- a/tests/basic/tests.py
+++ b/tests/basic/tests.py
@@ -717,8 +717,8 @@ class SelectOnSaveTests(TestCase):

 class ModelRefreshTests(TestCase):
     def _truncate_ms(self, val):
-        # MySQL removes microseconds from the datetimes which can cause problems
-        # when comparing the original value to that loaded from DB.
+        # MySQL < 5.6.4 removes microseconds from the datetimes which can cause
+        # problems when comparing the original value to that loaded from DB.
         return val - timedelta(microseconds=val.microsecond)

     def test_refresh(self):
diff --git a/tests/defer/models.py b/tests/defer/models.py
index 9072c58..ecf69c0 100644
--- a/tests/defer/models.py
+++ b/tests/defer/models.py
@@ -43,6 +43,6 @@ class RefreshPrimaryProxy(Primary):
         if fields is not None:
             fields = set(fields)
             deferred_fields = self.get_deferred_fields()
-            if set(fields).intersection(deferred_fields):
+            if fields.intersection(deferred_fields):
                 fields = fields.union(deferred_fields)
         super(RefreshPrimaryProxy, self).refresh_from_db(using, fields, **kwargs)
diff --git a/tests/defer/tests.py b/tests/defer/tests.py
index 0df33cc..597f871 100644
--- a/tests/defer/tests.py
+++ b/tests/defer/tests.py
@@ -211,7 +211,7 @@ class DeferTests(TestCase):
         rf.value = 'new bar'
         rf.save()
         with self.assertNumQueries(1):
-            # Customized referesh_from_db reloads all deferred fields on
+            # Customized refresh_from_db() reloads all deferred fields on
             # access of any of them.
             self.assertEqual(rf2.name, 'new foo')
             self.assertEqual(rf2.value, 'new bar')
@dfunckt
dfunckt commented Nov 20, 2014

This looks awesome, however I don't think it handles annotated fields. Is this deliberate?

@PirosB3
@akaariai
Django member

We will of course wait.

Personally I don't think get_deferred_fields() belongs to _meta. The problem is that this information really is per instance. When accessing a deferred field, that instance's deferred fields will change, but _meta of course doesn't change. We could push this method to meta with instance as an argument, but I don't see the point of it.

Annotations will not be refreshed, and that is deliberate. We could refresh annotations, too, if we remembered the original queryset the instance came from. Another possibility is to add "using_queryset" argument to the refresh_from_db() method. This would allow one to refresh annotations if wanted.

@dfunckt
dfunckt commented Nov 21, 2014

Hi Anssi, yes, annotations require knowledge of the queryset that fetched the instance, and your suggestion to provide a custom queryset sounds like a good solution -- it's simple and can also be used in other cases. Regardless whether we go that route, I think it's useful to have a note in docs about what's going to happen to annotations when the instance is refreshed.

@timgraham
Django member

Anssi, shall we add a sentence to the docs to mention that annotations are not refreshed?

@akaariai akaariai Fixed #901 -- Added Model.refresh_from_db() method
Thanks to github aliases dbrgn, carljm, slurms, dfunckt and timgraham
for reviews.
649729e
@akaariai
Django member

There is now a small mention that annotations are not refreshed.

I think the get_deferred_fields() method is fine as is. This really is per-instance method, not per-class, and thus belongs to the instance instead of _meta.

@timgraham
Django member

merged in c7175fc.

@timgraham timgraham closed this Nov 28, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.