Skip to content

Loading…

Create non-existent instances #6

Merged
merged 1 commit into from

5 participants

@agonzalezro

It's possible that the object related with the cache model that we are
trying to access doesn't exist yet.

If we had created our caching model with create=True, example:

cache_relation(User.settings, create=True)

the instance for User.settings is going to be created in caste that it
was not found.

Added some spaces to be Flake8 compliant.

@lamby

I completely understand how useful this would be. However, this dramatically changes the API and expectations of what a "get" should do (ie. it should never do a write!) and feel very very strongly this should not be a default behaviour.

Also this introduces/inherits subtle transaction issues from get_or_create

@leepa

Consider creating get_or_create_instance()?

@agonzalezro

I completely understand your concerns @lamby. I didn't think at all in other use-cases non related with ours (my mistake).

I am going to create a function get_or_create_instance that is going to solve our problem and it's not going to change the API.

Please, feel free of taking a look to coming PR and comment if you are happy (or not).

Thanks for your feedback!

@giftig giftig commented on an outdated diff
cache_toolbox/core.py
@@ -58,7 +64,14 @@ def get_instance(model, instance_or_pk, timeout=None, using=None):
cache.delete(key)
# Use the default manager so we are never filtered by a .get_query_set()
- instance = model._default_manager.using(using).get(pk=pk)
+ # It's possible that the related object didn't exist yet
+ try:
+ instance = model._default_manager.using(using).get(pk=pk)
+ except model.DoesNotExist as exc:
+ if create:
+ instance, _ = model.objects.get_or_create(pk=pk)
+ else:
+ raise exc
@giftig
giftig added a note

This obscures the stack trace. Just use raise, and don't bother to name the exception.

Yep, good point I will change this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@giftig giftig commented on an outdated diff
cache_toolbox/relation.py
@@ -76,7 +76,9 @@ class Foo(models.Model):
from .core import get_instance, delete_instance
-def cache_relation(descriptor, timeout=None):
+def cache_relation(descriptor, timeout=None, create=False):
+ # If create is True the instance is going to be create if it doesn't exist
@giftig
giftig added a note

created. And this should be a docstring, not a code comment.

We don't have docstrings with :param: anywhere else. I don't think that we should add it just for this function. Anyway, I will add an example to the module documentation as it's everywhere else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@lamby

Thinking about this a little more, without proper UPSERT support this entire idea is really not very robust. I prefer not to have essentially broken-by-design options (even if disabled by default) as it taints the entire library.

@giftig giftig commented on an outdated diff
cache_toolbox/core.py
@@ -58,7 +64,13 @@ def get_instance(model, instance_or_pk, timeout=None, using=None):
cache.delete(key)
# Use the default manager so we are never filtered by a .get_query_set()
- instance = model._default_manager.using(using).get(pk=pk)
+ # It's possible that the related object didn't exist yet
+ try:
+ instance = model._default_manager.using(using).get(pk=pk)
+ except model.DoesNotExist:
+ if not create:
+ raise
+ instance, _ = model.objects.get_or_create(pk=pk)
@giftig
giftig added a note

You're using objects instead of _default_manager here; why? Also you've just failed to get it and now you're trying to get or create it. You should either just use create here, or you should do something like:

   if create:
       instance, _ = models._default_manager.get_or_create(pk=pk)
   else:
       instance = models._default_manager.get(pk=pk)

Also what happens if you are trying to create a model which has required fields without defaults? You're not providing any defaults and you'd presumably have to take these into the function to allow this to work. I also find it odd to try to create something and provide it with a pk, given that pk is an autoincrement field by default.

@lamby
lamby added a note

given that pk is an autoincrement field by default.

The reason for this generally is so that you can use user_id as your primary key; it thus needs to be deterministic and therefore pk needs to be passed in.

create a model which has required fields without defaults

You have that problem anyway; I'm not sure it's a concern of cache_toolbox.

(Agree with all the rest)

@lamby
lamby added a note

agonzalezro, you've now dropped the using(using) call for get_or_create and added a bunch of useless and distracting whitespace changes everywhere.

Please slow down and take more care/pride.

@giftig
giftig added a note

@lamby those whitespace changes are to make the module flake8-compliant.

@leepa
leepa added a note

The commit message should say that's been done as part of the change.

@giftig
giftig added a note

+1

@lamby
lamby added a note

I don't mind whitespace changes if you can justify them, but they obviously shouldn't be done in the same commit as something important...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@giftig

Also please update commit message to indicate flake8 fixes per @leepa

@giftig

You haven't updated the commit message.

@giftig giftig commented on an outdated diff
cache_toolbox/relation.py
@@ -94,7 +99,9 @@ def get(self):
except AttributeError:
pass
- instance = get_instance(rel.model, self.pk, timeout)
+ instance = get_instance(
+ rel.model, self.pk, timeout, create=create, defaults=defaults or {}
@giftig
giftig added a note

Just use defaults=defaults here; you don't need the 'or {}' in both places.

get_instance can be called from everywhere else, not only from core.py.

@giftig
giftig added a note

So? You're only using defaults here to pass it into get_instance; let get_instance handle what it should do with it. This should just pass it straight in.

Yeah, I didn't realise that this file was relation.py. Will change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@giftig giftig commented on an outdated diff
cache_toolbox/relation.py
@@ -20,12 +20,15 @@ class Foo(models.Model):
related_name='foo',
)
- name = models.CharField(max_length=20)
+ name = models.CharField(max_length=20, create=True)
@giftig
giftig added a note

create=True shouldn't be here...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@giftig giftig commented on an outdated diff
cache_toolbox/relation.py
@@ -20,12 +20,15 @@ class Foo(models.Model):
related_name='foo',
)
- name = models.CharField(max_length=20)
+ name = models.CharField(max_length=20, create=True)
cache_relation(User.foo)
@giftig
giftig added a note

...it should be here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@giftig

You also made some changes to make it meet our coding standards, which you haven't mentioned in the commit messages (the local imports etc. aren't flake8 fixes)

@giftig giftig commented on an outdated diff
cache_toolbox/relation.py
@@ -22,10 +22,13 @@ class Foo(models.Model):
name = models.CharField(max_length=20)
- cache_relation(User.foo)
+ cache_relation(User.foo, create=True)
@giftig
giftig added a note

You should also document the 'defaults' param.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@giftig

Mostly looks OK to me now but a couple of minor comments left.

@agonzalezro agonzalezro Create non-existent instances is create=True
It's possible that the object related with the cache model that we are
trying to access doesn't exist yet.

If we had created our caching model with create=True, example:

    cache_relation(User.settings, create=True)

the instance for User.settings is going to be created in caste that it
was not found.

Added some spaces to be Flake8 compliant.
dfe2d0f
@giftig

Still haven't mentioned standards fixes as well as flake8 ones in commit message, but meh.

This has a +1 from me.

@agonzalezro

I did it on the commit message time ago @giftig, but the description here doesn't get updated.

I've removed the import changes because that is more related with our coding standards than with this project. If I add some gmg coding standards here I would need to add some tests and documentation though. We should create a ENGDEBT card to do that.

@Evolter

I wonder about having flag in settings for choosing strategy (get or raise exception and get or create), but this looks fine for me.
+1

@lamby

^ I didn't see any comment on the unsafe nature of this architecture re. UPSERT

@giftig

Ship it

@lamby

Classy comment re-ordering there.

@agonzalezro

Sorry @lamby but we need this patchset merged to keep our packages and this synced.

If you really think that UPSERT could be a problem (which wasn't for few years), please provide a PR which introduces a suitable test to demonstrate that it fails, and we would be happy to provide a solution if you won't.

I would be happy of testing that behaviour on this package but the lack of tests make it really difficult. We were using our test suit on playfire to test it, and it seems that it works completely fine.

@agonzalezro agonzalezro merged commit 69576c9 into playfire:master
@lamby

As I explained previously it is the idea/architecture behind using sparse models that this changeset encourages that is faulty, so not only would it be an abstraction-level violation to test it at the code level, it can demonstrate nothing whatsover of value as we can already deduce such a test will fail.

On a practical level, the test would also be cumbersome in the Django framework as requires simultaneous transactions.

http://lucumr.pocoo.org/2014/2/16/a-case-for-upserts/ should be sufficient background documentation if you are not immediately familiar with this general problem area.

(As a meta comment, it's an obvious non-sequitor to respond to a purely technical point with "no, because $process".)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 28, 2014
  1. @agonzalezro

    Create non-existent instances is create=True

    agonzalezro committed
    It's possible that the object related with the cache model that we are
    trying to access doesn't exist yet.
    
    If we had created our caching model with create=True, example:
    
        cache_relation(User.settings, create=True)
    
    the instance for User.settings is going to be created in caste that it
    was not found.
    
    Added some spaces to be Flake8 compliant.
This page is out of date. Refresh to see the latest.
Showing with 28 additions and 5 deletions.
  1. +16 −2 cache_toolbox/core.py
  2. +12 −3 cache_toolbox/relation.py
View
18 cache_toolbox/core.py
@@ -13,7 +13,11 @@
from . import app_settings
-def get_instance(model, instance_or_pk, timeout=None, using=None):
+
+def get_instance(
+ model, instance_or_pk,
+ timeout=None, using=None, create=False, defaults=None
+):
"""
Returns the ``model`` instance with a primary key of ``instance_or_pk``.
@@ -23,6 +27,9 @@ def get_instance(model, instance_or_pk, timeout=None, using=None):
If omitted, the timeout value defaults to
``settings.CACHE_TOOLBOX_DEFAULT_TIMEOUT`` instead of 0 (zero).
+ If ``create`` is True, we are going to create the instance in case that it
+ was not found.
+
Example::
>>> get_instance(User, 1) # Cache miss
@@ -58,7 +65,12 @@ def get_instance(model, instance_or_pk, timeout=None, using=None):
cache.delete(key)
# Use the default manager so we are never filtered by a .get_query_set()
- instance = model._default_manager.using(using).get(pk=pk)
+ queryset = model._default_manager.using(using)
+ if create:
+ # It's possible that the related object didn't exist yet
+ instance, _ = queryset.get_or_create(pk=pk, defaults=defaults or {})
+ else:
+ instance = queryset.get(pk=pk)
data = {}
for field in instance._meta.fields:
@@ -82,6 +94,7 @@ def get_instance(model, instance_or_pk, timeout=None, using=None):
return instance
+
def delete_instance(model, *instance_or_pk):
"""
Purges the cache keys for the instances of this model.
@@ -89,6 +102,7 @@ def delete_instance(model, *instance_or_pk):
cache.delete_many([instance_key(model, x) for x in instance_or_pk])
+
def instance_key(model, instance_or_pk):
"""
Returns the cache key for this (model, instance) pair.
View
15 cache_toolbox/relation.py
@@ -22,10 +22,16 @@ class Foo(models.Model):
name = models.CharField(max_length=20)
- cache_relation(User.foo)
+ cache_relation(User.foo, create=True, defaults={})
(``primary_key`` being ``True`` is currently required.)
+With ``create=True`` we force the creation of an instance of `Foo` in case that
+we are trying to access to user.foo_cache but ``user.foo`` doesn't exist yet.
+
+If ``create=True`` we are going to pass the default to the get_or_create
+function.
+
::
>>> user = User.objects.get(pk=1)
@@ -76,7 +82,8 @@ class Foo(models.Model):
from .core import get_instance, delete_instance
-def cache_relation(descriptor, timeout=None):
+
+def cache_relation(descriptor, timeout=None, create=False, defaults=None):
rel = descriptor.related
related_name = '%s_cache' % rel.field.related_query_name()
@@ -94,7 +101,9 @@ def get(self):
except AttributeError:
pass
- instance = get_instance(rel.model, self.pk, timeout)
+ instance = get_instance(
+ rel.model, self.pk, timeout, create=create, defaults=defaults
+ )
setattr(self, '_%s_cache' % related_name, instance)
Something went wrong with that request. Please try again.