Django 1.6 support

[ebrelsford]

It would be great to have support for Django 1.6. Most obvious change is that django.core.cache.backends.memcached.CacheClass was renamed BaseMemcachedCache. Not sure about the rest.

[BertrandBordage]

See also #36.

[BertrandBordage]

See also #36.

[finder]

Working on it this weekend, we have about 20 failing tests so there's a lot of new issues with 1.6, particularly around transaction management, so it may be a bit.

[finder]

Working on it this weekend, we have about 20 failing tests so there's a lot of new issues with 1.6, particularly around transaction management, so it may be a bit.

[bjornlilja]

Any updates on this?

[bjornlilja]

Any updates on this?

[chadmasso]

bump?

[chadmasso]

bump?

[finder]

There's been fixes to branch django-1.6 that brings the test failures to about 3, all transaction based. I haven't had a chance to revisit it though, or to test multi-db. Pull requests are welcome if you can fix the failing tests or test multi-db. :) Once all are complete we can do a release, until then I wouldn't feel comfortable releasing it for a production environment. jmoiron's travelling and I'll be traveling next week, so I can say it won't be this weekend otherwise, perhaps by Christmas.

[finder]

There's been fixes to branch django-1.6 that brings the test failures to about 3, all transaction based. I haven't had a chance to revisit it though, or to test multi-db. Pull requests are welcome if you can fix the failing tests or test multi-db. :) Once all are complete we can do a release, until then I wouldn't feel comfortable releasing it for a production environment. jmoiron's travelling and I'll be traveling next week, so I can say it won't be this weekend otherwise, perhaps by Christmas.

[rh0dium]

Status please !! Happy Holidays!!

[rh0dium]

Status please !! Happy Holidays!!

[Andrioden]

Not sure if good practice to do but: +1

Keep up the good work

[Andrioden]

Not sure if good practice to do but: +1

Keep up the good work

[vishen]

Has anybody taken a look at how to handle the new Atomic class in django-1.6 (https://github.com/django/django/blob/master/django/db/transaction.py#L221)? It doesn't use the public api that johnny patches, but uses the connection directly.

[vishen]

Has anybody taken a look at how to handle the new Atomic class in django-1.6 (https://github.com/django/django/blob/master/django/db/transaction.py#L221)? It doesn't use the public api that johnny patches, but uses the connection directly.

[tigrus]

Status?

[tigrus]

Status?

[gdub]

Created a branch at https://github.com/gdub/johnny-cache/tree/django-1.6 that has start of some tests for the new Atomic class. With a patched django 1.6 (https://github.com/gdub/django/tree/1.6.x-atomic-patch):

• Python 3.4: All tests pass except for the johnny.tests.cache.AtomicTest.test_savepoint_rollback test using django.db.backends.postgresql_psycopg2 and patched mysql.connector.django (https://github.com/gdub/mysql-connector-python) backends.
• Python 2.7: All tests pass with django.db.backends.postgresql_psycopg2 and django.db.backends.sqlite3 backends, but getting several encoding failures with the mysql.connector.django backend.

If the Django patch can make it into the 1.7 release, then perhaps johnny-cache can skip support for Django 1.6 and jump to supporting 1.7 instead.

Note: I've pushed the tox setup I was using for the above tests to https://github.com/gdub/johnny-cache/tree/tox-configs

[gdub]

Created a branch at https://github.com/gdub/johnny-cache/tree/django-1.6 that has start of some tests for the new Atomic class. With a patched django 1.6 (https://github.com/gdub/django/tree/1.6.x-atomic-patch):

• Python 3.4: All tests pass except for the johnny.tests.cache.AtomicTest.test_savepoint_rollback test using django.db.backends.postgresql_psycopg2 and patched mysql.connector.django (https://github.com/gdub/mysql-connector-python) backends.
• Python 2.7: All tests pass with django.db.backends.postgresql_psycopg2 and django.db.backends.sqlite3 backends, but getting several encoding failures with the mysql.connector.django backend.

If the Django patch can make it into the 1.7 release, then perhaps johnny-cache can skip support for Django 1.6 and jump to supporting 1.7 instead.

Note: I've pushed the tox setup I was using for the above tests to https://github.com/gdub/johnny-cache/tree/tox-configs

[EmilStenstrom]

@gdub Where is the Django ticket for merging your pull request? Do I understand it correctly that this is what's blocking further progress on this ticket?

[EmilStenstrom]

@gdub Where is the Django ticket for merging your pull request? Do I understand it correctly that this is what's blocking further progress on this ticket?

[aaugustin]

(Author of Django's new transaction handling here.)

I had a quick look at johnny/transaction.py. It seems that johnny-caches uses savepoints and relies on the fact that Django doesn't use them. This isn't true anymore in Django 1.6. I suspect mixing nested atomic contexts and johnny-cache will fail rather badly. Can someone confirm?

I'm not convinced the changes suggested in Django are a good idea, they make the implementation less consistent in order to support monkey-patching that may break the guarantees of atomic. Providing a mixin for the database backends would be a much more robust way to customize transaction handling than monkey-patching django.db.transactions.

[aaugustin]

(Author of Django's new transaction handling here.)

I had a quick look at johnny/transaction.py. It seems that johnny-caches uses savepoints and relies on the fact that Django doesn't use them. This isn't true anymore in Django 1.6. I suspect mixing nested atomic contexts and johnny-cache will fail rather badly. Can someone confirm?

I'm not convinced the changes suggested in Django are a good idea, they make the implementation less consistent in order to support monkey-patching that may break the guarantees of atomic. Providing a mixin for the database backends would be a much more robust way to customize transaction handling than monkey-patching django.db.transactions.

[gdub]

The Django ticket, and linked pull request, is here: https://code.djangoproject.com/ticket/22802

@aaugustin, Not sure on the comment about johnny using savepoints and relies on the fact of Django not using them. johnny/transaction.py still calls the original django functions, wrapping them to do some of its own house-cleaning with its cache store.

I'm also not a fan of the monkey patching, though I also think it shows that Django does not provide easy enough hooks to into the transaction behavior. Unless the base Django database classes provided a way to mixin a transaction-management class or a way to specify a desired class via a database option, one would have to create a set of python packages and class hierarchy to parallel each existing database backend.

Perhaps one of the johnny-cache authors would have more insight into the decisions made, or possible long-term solutions that would make extension easier. I threw out some ideas in the comments of the linked Django ticket.

[gdub]

The Django ticket, and linked pull request, is here: https://code.djangoproject.com/ticket/22802

@aaugustin, Not sure on the comment about johnny using savepoints and relies on the fact of Django not using them. johnny/transaction.py still calls the original django functions, wrapping them to do some of its own house-cleaning with its cache store.

I'm also not a fan of the monkey patching, though I also think it shows that Django does not provide easy enough hooks to into the transaction behavior. Unless the base Django database classes provided a way to mixin a transaction-management class or a way to specify a desired class via a database option, one would have to create a set of python packages and class hierarchy to parallel each existing database backend.

Perhaps one of the johnny-cache authors would have more insight into the decisions made, or possible long-term solutions that would make extension easier. I threw out some ideas in the comments of the linked Django ticket.

[EmilStenstrom]

@gdub Since that ticket was wontfixed I guess the way forward is to find a way to let people install a custom database backend. Is providing patched versions of the existing backends such a big deal? I see the configuration options needed to be:

DATABASES = {
    'default': {
        'ENGINE': johnny.backends.mysql',
        ...
    }
}

... and a mixin for people that want to use johnny cache for their own database backend.

[EmilStenstrom]

@gdub Since that ticket was wontfixed I guess the way forward is to find a way to let people install a custom database backend. Is providing patched versions of the existing backends such a big deal? I see the configuration options needed to be:

DATABASES = {
    'default': {
        'ENGINE': johnny.backends.mysql',
        ...
    }
}

... and a mixin for people that want to use johnny cache for their own database backend.

[amirrustam]

Hi all. What is the status of Django 1.6 support. From browsing online, it seems like johnny-cache is being used with Django 1.6, and also issue #29 is more evidence of that. So is johnny an option for 1.6? If yes, does the django-1.6 branch (10 month old) have to be used?

The alternatives are django-cache-machine and django-cacheops which are nice, but they either don't appropriately invalidate on INSERT, don't use Django's cache settings, or Redis only. Hopefully I have not misunderstood these alternatives. Any input on this matter is appreciated.

[amirrustam]

Hi all. What is the status of Django 1.6 support. From browsing online, it seems like johnny-cache is being used with Django 1.6, and also issue #29 is more evidence of that. So is johnny an option for 1.6? If yes, does the django-1.6 branch (10 month old) have to be used?

The alternatives are django-cache-machine and django-cacheops which are nice, but they either don't appropriately invalidate on INSERT, don't use Django's cache settings, or Redis only. Hopefully I have not misunderstood these alternatives. Any input on this matter is appreciated.

[rh0dium]

Thanks Amir,

We too are pondering whether on this as well and would really love to know of a branch that is working effectively with MySQL and Django >= 1.6. For the reasons you state I am unwilling to move - I am doing some benchmarks to see if I can bring straight MySQL caching to the performance that Johnny Cache can produce but haven't achieved it as of now.

[rh0dium]

Thanks Amir,

We too are pondering whether on this as well and would really love to know of a branch that is working effectively with MySQL and Django >= 1.6. For the reasons you state I am unwilling to move - I am doing some benchmarks to see if I can bring straight MySQL caching to the performance that Johnny Cache can produce but haven't achieved it as of now.

[BertrandBordage]

As far as I know, there is still a lot of cache invalidation issues with Django 1.6.
I tried to fix them, but I didn't manage to fix all of them yet :(

[BertrandBordage]

As far as I know, there is still a lot of cache invalidation issues with Django 1.6.
I tried to fix them, but I didn't manage to fix all of them yet :(

[BertrandBordage]

I'm currently writing a successor for johnny-cache, built from scratch for Django 1.6 & 1.7.

I just finished a prototype for Redis only, it works well! It consists in 105 lines of code and you just have to add a line to INSTALLED_APPS in order to install it :)

I’ll share it as soon as possible!

[BertrandBordage]

I'm currently writing a successor for johnny-cache, built from scratch for Django 1.6 & 1.7.

I just finished a prototype for Redis only, it works well! It consists in 105 lines of code and you just have to add a line to INSTALLED_APPS in order to install it :)

I’ll share it as soon as possible!

[vishen]

Would love to see it when your ready @BertrandBordage !

[vishen]

Would love to see it when your ready @BertrandBordage !

[jmoiron]

Me too. Frankly I do not have the time, energy, or desire to maintain johnny-cache properly, and that's been clear for a while. I have dragged my feet in admitting it and trying to find a maintainer, which is a major failing on my part. In the interim:

• johnny has gained a lot of cruft from being originally designed in a world with only django1.1 and python2
• django has moved on 6 major revisions
• django has changed its orm machinery quite often, alternately making some aspects of johnny more difficult but also providing more hooks for something like it

It's about time for a rewrite which doesn't have these burdens. If it can be made compatible with the interface and strict guarantees johnny has set out in the past, I don't mind handing over the project and pypi repos.

Otherwise, I will accept any patches that improve compatibility and hand out repos access to people willing to patch it.

[jmoiron]

Me too. Frankly I do not have the time, energy, or desire to maintain johnny-cache properly, and that's been clear for a while. I have dragged my feet in admitting it and trying to find a maintainer, which is a major failing on my part. In the interim:

• johnny has gained a lot of cruft from being originally designed in a world with only django1.1 and python2
• django has moved on 6 major revisions
• django has changed its orm machinery quite often, alternately making some aspects of johnny more difficult but also providing more hooks for something like it

It's about time for a rewrite which doesn't have these burdens. If it can be made compatible with the interface and strict guarantees johnny has set out in the past, I don't mind handing over the project and pypi repos.

Otherwise, I will accept any patches that improve compatibility and hand out repos access to people willing to patch it.

[BertrandBordage]

@jmoiron Yes, the project wanted to be compatible with too many versions of Django & Python. It became a bit of a mess and now it’s too hard to clean everything, even after hours of work. However, I really want to thank you for sharing johnny-cache, it helped me a lot in the past three years!

And so, I started a new project called django-cachalot! I copied on johnny-cache the idea of monkey patching the ORM. Everyone can give it a try, but please don’t use it in production before version 1.0!

[BertrandBordage]

@jmoiron Yes, the project wanted to be compatible with too many versions of Django & Python. It became a bit of a mess and now it’s too hard to clean everything, even after hours of work. However, I really want to thank you for sharing johnny-cache, it helped me a lot in the past three years!

And so, I started a new project called django-cachalot! I copied on johnny-cache the idea of monkey patching the ORM. Everyone can give it a try, but please don’t use it in production before version 1.0!

[vishen]

@BertrandBordage How do you plan to handle the atomic problem?

[vishen]

@BertrandBordage How do you plan to handle the atomic problem?

[BertrandBordage]

@vishen First easy solution, simply avoid caching read queries when connection.in_atomic_block.

But we can cache what happens in transaction, we simply need to use a local caching (using a dict for reads and a set of invalidated tables) for each atomic block. When an atomic block fails, we delete the dict & set, otherwise we merge them to the local cache of the parent atomic block. If there's no parent atomic block, we "commit" by applying changes to the real cache backend.

All this is quite easy using connection.savepoint_ids. However, it’s not thread-safe. But it’s a problem with the current transaction system, it’s not done to be thread-safe if I understand well. I’ll talk about this with Aymeric Augustin, the author of the atomic system.

I’ll work on this tonight.

[BertrandBordage]

@vishen First easy solution, simply avoid caching read queries when connection.in_atomic_block.

But we can cache what happens in transaction, we simply need to use a local caching (using a dict for reads and a set of invalidated tables) for each atomic block. When an atomic block fails, we delete the dict & set, otherwise we merge them to the local cache of the parent atomic block. If there's no parent atomic block, we "commit" by applying changes to the real cache backend.

All this is quite easy using connection.savepoint_ids. However, it’s not thread-safe. But it’s a problem with the current transaction system, it’s not done to be thread-safe if I understand well. I’ll talk about this with Aymeric Augustin, the author of the atomic system.

I’ll work on this tonight.

[BertrandBordage]

The transaction issue was a bit more complicated than I thought. What I said above didn’t work exactly as expected. Instead of relying on connection.savepoint_ids, I had to make my own stack of "atomic caches".

So yes, django-cachalot properly handles transactions since a few minutes :)

[BertrandBordage]

The transaction issue was a bit more complicated than I thought. What I said above didn’t work exactly as expected. Instead of relying on connection.savepoint_ids, I had to make my own stack of "atomic caches".

So yes, django-cachalot properly handles transactions since a few minutes :)

[rh0dium]

Hey @BertrandBordage

I spent this morning reviewing your code and I like where your head is. I have a couple questions though. At the expense of looking like a idiot I'll ask them because I really want a solid successor to johnny.

It's not entirely clear how you are performing cache invalidation. Perhaps you could explain this a bit to me (us).

To summarize the creating of a cache key (on a read) you do this

• The cache key is created via the query.as_sql() method https://github.com/BertrandBordage/django-cachalot/blob/master/cachalot/monkey_patch.py#L159

To invalidate a cache key (on a write) you do this

• Get a list of all tables on the query https://github.com/BertrandBordage/django-cachalot/blob/master/cachalot/monkey_patch.py#L52
• Get from the cache all the queries with keys matching those tables L53
• Then something goes on - you are creating the queries by iterating over the table names by getting the associated cache keys?
• Then you delete them.

But here is where my confusion steps in (perhaps it redis only but) when you do a cache.get_many https://github.com/django/django/blob/master/django/core/cache/backends/base.py#L135 aren't you only getting those cache keys which match explicitly. So you never really get all of the keys you only get those keys that match the table names - which shouldn't exist?

I suppose I'm in error on this but could you clarify this a bit.

Edit: It looks like you are storing two queries one to hold the query and another to store the queries whch are under the table. Is this correct?

Thanks

[rh0dium]

Hey @BertrandBordage

I spent this morning reviewing your code and I like where your head is. I have a couple questions though. At the expense of looking like a idiot I'll ask them because I really want a solid successor to johnny.

It's not entirely clear how you are performing cache invalidation. Perhaps you could explain this a bit to me (us).

To summarize the creating of a cache key (on a read) you do this

• The cache key is created via the query.as_sql() method https://github.com/BertrandBordage/django-cachalot/blob/master/cachalot/monkey_patch.py#L159

To invalidate a cache key (on a write) you do this

• Get a list of all tables on the query https://github.com/BertrandBordage/django-cachalot/blob/master/cachalot/monkey_patch.py#L52
• Get from the cache all the queries with keys matching those tables L53
• Then something goes on - you are creating the queries by iterating over the table names by getting the associated cache keys?
• Then you delete them.

But here is where my confusion steps in (perhaps it redis only but) when you do a cache.get_many https://github.com/django/django/blob/master/django/core/cache/backends/base.py#L135 aren't you only getting those cache keys which match explicitly. So you never really get all of the keys you only get those keys that match the table names - which shouldn't exist?

I suppose I'm in error on this but could you clarify this a bit.

Edit: It looks like you are storing two queries one to hold the query and another to store the queries whch are under the table. Is this correct?

Thanks

[BertrandBordage]

@rh0dium, could you ask your question on the django-cachalot Google group? We’re a bit off-topic now.

[BertrandBordage]

@rh0dium, could you ask your question on the django-cachalot Google group? We’re a bit off-topic now.

[vijaykatam]

We have been using johnny-cache heavily for models that don't change often and this issue was preventing us from upgrading django. So went ahead and implemented an alternative that uses custom model manager and querysets, see https://github.com/vijaykatam/django-cache-manager. We are currently swapping out johnny-cache in favor of this.

[vijaykatam]

We have been using johnny-cache heavily for models that don't change often and this issue was preventing us from upgrading django. So went ahead and implemented an alternative that uses custom model manager and querysets, see https://github.com/vijaykatam/django-cache-manager. We are currently swapping out johnny-cache in favor of this.

[cabello]

Thanks for sharing, we could start some kind of guideline for READMEs in django projects, since there are so many majors versions around and alive, something like:

• Supported - Django 1.x;
• Expected To Work / Untested - Django 1.x--;
• Not Working - Django 1.x----.

Anyway this idea may be way too off topic, we stopped using johnny cache in my current project after upgrading to 1.6 and I will wait just a bit before trying a new cache solution again. Hope your project get mature and stable. :)

[cabello]

Thanks for sharing, we could start some kind of guideline for READMEs in django projects, since there are so many majors versions around and alive, something like:

• Supported - Django 1.x;
• Expected To Work / Untested - Django 1.x--;
• Not Working - Django 1.x----.

Anyway this idea may be way too off topic, we stopped using johnny cache in my current project after upgrading to 1.6 and I will wait just a bit before trying a new cache solution again. Hope your project get mature and stable. :)

[vijaykatam]

@cabello good point, will add django versions.

[vijaykatam]

@cabello good point, will add django versions.