New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory backend objects persist unwanted data #51

Closed
sqlalchemy-bot opened this Issue Nov 26, 2013 · 3 comments

Comments

Projects
None yet
1 participant
@sqlalchemy-bot

sqlalchemy-bot commented Nov 26, 2013

Migrated issue, originally created by jvanasco (jvanasco)

I ran into an issue while troubleshooting my read-through cache.

in Production, we use pylibmc ; in Development we usually use dbm , but sometimes switch to memory as it is much faster. our cached data has quite a bit of processing applied to it, and on a recent test run -- unwanted 'per-request' data was somehow persisting. i spent a few hours upgrading logging, and trying to track down phantom cache "sets".

then i realized what was going on -- we weren't calling a set, but the data was persisting because of the memory backend. we weren't pulling something out of a cache, it was the same object each time.

in order to get our tests to pass and ensure some amount of parity between dev and production, i put together a quick "MemoryPickle" backend. I couldn't think of a better way to handle this; it's the memory backend , with get/set wrapped in pickle loads/gets.

this does nothing but ensure that you get an item out of the cache that only has "set" data. there are probably better ways to handle this , which is why I'm just proposing this with a gist, and not doing a pull request.

this sort of thing could be handled with a wrapper or custom backend. i think something like this is very useful and belongs in core, only because it allows users the ability to leverage the speed of a memory backend with the same behavior of external backends ( dbm , pylibmc , redis , etc )... and it would only require a 1 line config change.

here's the working copy.

https://gist.github.com/jvanasco/7651368

@sqlalchemy-bot

This comment has been minimized.

sqlalchemy-bot commented Nov 26, 2013

Michael Bayer (zzzeek) wrote:

I've literally spent all day on a bug here that is essentially the same issue. What I'm doing at the moment, since it's an ORM/Session cache, is loading the objects with a different Session, so that when they get merged into the return value, they are different from those the default Session would load. Over here I think I'm going to stick with that since it doesnt have the significant overhead and error-prone-ness of pickle.

Anyway, I think the most straightforward system would be to just add a flag "pickle" to the existing "memory" backend. Send me a pullreq.

@sqlalchemy-bot

This comment has been minimized.

sqlalchemy-bot commented Jan 9, 2014

Michael Bayer (zzzeek) wrote:

added pickle to the memory backend in 53f8a07

@sqlalchemy-bot

This comment has been minimized.

sqlalchemy-bot commented Jan 9, 2014

Changes by Michael Bayer (zzzeek):

  • changed status to closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment