memory backend objects persist unwanted data

[sqlalchemy-bot]

Migrated issue, originally created by jvanasco (jvanasco)

I ran into an issue while troubleshooting my read-through cache.

in Production, we use pylibmc ; in Development we usually use dbm , but sometimes switch to memory as it is much faster. our cached data has quite a bit of processing applied to it, and on a recent test run -- unwanted 'per-request' data was somehow persisting. i spent a few hours upgrading logging, and trying to track down phantom cache "sets".

then i realized what was going on -- we weren't calling a set, but the data was persisting because of the memory backend. we weren't pulling something out of a cache, it was the same object each time.

in order to get our tests to pass and ensure some amount of parity between dev and production, i put together a quick "MemoryPickle" backend. I couldn't think of a better way to handle this; it's the memory backend , with get/set wrapped in pickle loads/gets.

this does nothing but ensure that you get an item out of the cache that only has "set" data. there are probably better ways to handle this , which is why I'm just proposing this with a gist, and not doing a pull request.

this sort of thing could be handled with a wrapper or custom backend. i think something like this is very useful and belongs in core, only because it allows users the ability to leverage the speed of a memory backend with the same behavior of external backends ( dbm , pylibmc , redis , etc )... and it would only require a 1 line config change.

here's the working copy.

https://gist.github.com/jvanasco/7651368

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

I've literally spent all day on a bug here that is essentially the same issue. What I'm doing at the moment, since it's an ORM/Session cache, is loading the objects with a different Session, so that when they get merged into the return value, they are different from those the default Session would load. Over here I think I'm going to stick with that since it doesnt have the significant overhead and error-prone-ness of pickle.

Anyway, I think the most straightforward system would be to just add a flag "pickle" to the existing "memory" backend. Send me a pullreq.

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

added pickle to the memory backend in 53f8a07

[sqlalchemy-bot]

Changes by Michael Bayer (zzzeek):

• changed status to closed