TypeError: 'NoneType' object is not iterable

[shakey-ltd]

running django-cachalot with Django 1.8, memcached and ElastiCache

File "/opt/python/run/venv/lib/python2.7/site-packages/cachalot/monkey_patch.py", line 57, in _get_result_or_execute_query
new_table_cache_keys.difference_update(data)
TypeError: 'NoneType' object is not iterable

[BertrandBordage]

Probably coming from ElastiCache that chose to return None instead of an empty dict. When ElastiCache will follow the rules, we’ll maybe add it. But for now, it’s an invalid issue.

[dan-passaro]

This is not invalid; I get the same problem with django.core.cache.backends.memcached.MemcachedCache on Django 1.8. Attempting to look into it now but the problem is the line timestamp, result = data.pop(cache_key). Somehow a value in that data dict is None. I am currently trying to isolate why so I can give you steps to reproduce.

[BertrandBordage]

@dan-passaro The error message may be the same, but the problem is different:

• In @shakey-uk’s problem, the ElastiCache backend was returning None to a cache.get_many(cache_keys) call. It’s totally abnormal since the Django cache API specifies that a call to cache.get_many always returns a dict, and an empty dict if no key has been found.
• In your case, something like {cache_key1: None, cache_key2: None} is returned to the same call, leading to an error in timestamp, result = data.pop(cache_key). According to the Django cache API, it can only been returned if a key was previously set to None. Since django-cachalot never sets a key to None, there is not much possibility:
  • You won at the hardest lottery in the universe: another application set a cache key to None and django-cachalot generated a key that’s accidentally the same. Since django-cachalot uses sha1 to generate its cache keys, you have a 1 / 16⁴⁰ chance (multiplied by the chance of having a None value instead of anything else). So that’s more impossible than suddenly waking up tomorrow on Jupiter in a healthy environment.
  • You’re using a broken install of memcached or python-memcached.

Anyway, I’ve been testing django-cachalot against a ton of supported softwares (see https://travis-ci.org/BertrandBordage/django-cachalot) for a year, and at no moment I could see something like that.

Please send your Python, Django, memcached, python-memcached, and OS versions, it may help us find the reason.

[dan-passaro]

Python 2.7.9
Django 1.8.5
Memcached 1.4.14
python-memcached 1.54

I have been looking into it more, and if the only calls to cache are all in cachalot.monkey_patch it does seem to be some other component causing the value to become None. No other app is running that would generate keys of that kind so I'll try and dig around more and try to figure out what's going on. This stack was (or at least appeared to be) working smoothly under Django 1.7.

EDIT:

Looks like cache.set() is failing for that specific result (which is admittedly quite large):

(Pdb) cache.set(cache_key, (time(), result), None)
(Pdb) cache.get_many([cache_key])
{'03afedf1445f306fa09a5af2cbf65124893c7b34': None}
(Pdb) cache.set(cache_key, (time(), 'ay'), None)
(Pdb) cache.get_many([cache_key])
{'03afedf1445f306fa09a5af2cbf65124893c7b34': (1444919918.691598, 'ay')}

This PDB is in cachalot.monkey_patch._get_result_or_execute_query just above the final cache.set() call.

Curiously it fails differently with other large data, e.g.:

(Pdb) cache.set(cache_key, 'a' * 10000000000, None)  # this froze my laptop for a few seconds
(Pdb) cache.get_many([cache_key])
{}

[dan-passaro]

I guess the underlying bug is in python-memcached (or memcached itself but I feel that's impossibly unlikely)...

(Pdb) cache._cache.set(cache_key, (time(), result))
True
(Pdb) cache._cache.get(cache_key)
(Pdb) cache._cache.set(cache_key, (time(), 'foo'))
True
(Pdb) cache._cache.get(cache_key)
(1444933450.983374, 'foo')

Note it returns True even when it fails to store anything.

FWIW this "only shows up on Django 1.8" because I forgot that as part of moving to Django 1.8, we also had to move from django-pyodbc to django-pyodbc-azure. I'm not exactly sure why that would trigger this bug but I can confirm this same issue happens with our 1.7 stack using django-pyodbc-azure and it doesn't happen when using django-pyodbc. I'm guessing the value of result is different and too large or something like that when using django-pyodbc-azure (although again, since it's the same query, it is puzzling).

[dan-passaro]

I am unable to reproduce this bug outside the pdb context. I serialized result and tried to reproduce in a standalone script but now it either succeeds or fails incorrectly, where set() returns 0 (indicating failure) but get() returns the data anyway (I don't even).

I'm convinced it's a bug with python-memcache not correctly determining timeouts on set. Switching to pylibmc consistently causes a timeout exception to be raised instead, after which subsequent requests to the page in question generate the same TypeError in cachalot. This suggests to me that independent of client, a set operation that times out leaves a dangling, null key in memcached.

I've unfortunately sunk too much time trying to figure this problem out to get specific reproduction steps or to properly identify the issue. As there are other pressing things I need to do here at work, I'm going to fork cachalot and duct-tape over the problem by handling/skipping None values where it's currently crashing. Apologies for noise in the wrong ticket, and not resolving this properly.

[BertrandBordage]

Thanks for trying to solve this, I don’t really understand either what’s causing this.
But what I can tell you is that you shouldn’t use django-cachalot: you’re using django-pyodbc-azure as your database backend, and it’s not supported by django-cachalot (and will never be).

By the way, are you using Windows? It may be the cause of the issue, I don’t know how memcached behaves on Windows and most Django developers are Linux or MacOS users.

[dan-passaro]

No, the servers use FreeTDS to connect to a legacy SQL Server DB. We're
slowly migrating to Postgres. I think the servers run on Ubuntu.

I understand not wanting to support SQL Server, but I think the code in
question is relatively backend agnostic. At least in my case the data at
that point was all built in Python types... The issue seems to be that the
result set is just too large for Memcached to handle well.

And even without support, Cachalot works amazingly well! Thanks for putting
so much work into this awesome package :)

[BertrandBordage]

Yes, that’s why I said I don’t understand ;) I know it’s not DB related in this case, but anyway, it’s not totally safe to use it with an unsupported DB. In django-cachalot 1.1, a system check error will prevent you from using this unsafe combination. However, you’ll be able to skip this error using SILENCED_SYSTEM_CHECKS.

What’s weird about your large object issue is that it should throw the error described in the limits of memcached. Try to tweak the settings as I suggested in that piece of doc, it may fix the issue.

Glad you enjoy Cachalot besides that :D

[karthikbgl]

I ran into this exact issue. I just went into the shell, cleared the cache and the issue was resolved. (Using Python 3.5, Django 1.10 and Memcached)