Making mongoengine gevent "threadsafe"

[egbutter]

Mongofanatics:

I am playing around with mongoengine in an "asynchronous" (gevent) application and just found out that as my threads finish making queries, I need to tell each thread individually to release its connection to mongodb back into its threadpool.

Reference: (http://stackoverflow.com/questions/7166998/pymongo-gevent-throw-me-a-banana-and-just-monkey-patch/7169174#7169174).

Is there a reason why mongoengine does not call pymongo.Connection().end_request() right now? If this requires a patch, I am more than willing to submit one! Cheers,

Eric

[rozza]

@hmarr - Any thoughts?

[dcrosta]

See http://groups.google.com/group/mongodb-user/browse_thread/thread/90b71fa3dc5b6b5f/328cd0a4ebc40cd3. 10gen doesn't support using pymongo with asynchronous frameworks at this time.

[egbutter]

@dcrosta - thanks for pointing me back to my own question (the email must have slipped by me).

just because 10gen does not support it does not mean it cannot be done. e.g., there is a pymongo patch by Matt Good https://github.com/mgood/mongo-python-driver that handles gevent "nicely" and here http://code.activestate.com/recipes/577490-mongodb-pool-for-gevent-and-pymongo-packages/ is a different patch.

[rozza]

patches always welcome!

[rozza]

Another patch: https://gist.github.com/1184264

[andrew-azarov]

Any updates on this?

[rozza]

It should work with pymongo 2.2.1 - http://api.mongodb.org/python/current/examples/gevent.html?highlight=gevent

I haven't tested it - but the connection part should be fine! You'll need to monkey patch all before importing the MongoEngine / registering the connections

[andrew-azarov]

I just have this bottlepy plugin to do the endrequest cleanup, so I'm thinking about what you have found.
Should I move out the endrequest check? (The connection is made global in bottlepy)

[andrew-azarov]

here's the code

class mongoengineplugin(object):
    name = 'mongoengine'
    api = 2

    def __init__(self, db, keyword='db', alias='default',**kwargs):
        self.db = connect(db,alias,**kwargs)
        self.keyword = keyword
    def setup(self, app):
        for other in app.plugins:
            if not isinstance(other, mongoengineplugin): continue
            if other.keyword == self.keyword:
                raise PluginError("Found another mongoengine plugin with "\
                "conflicting settings (non-unique keyword).")

    def apply(self, callback, context):
        conf = context.config.get('mongoengine') or {}
        db = conf.get('db', self.db)
        keyword = conf.get('keyword',self.keyword)
        args = inspect.getargspec(context.callback)[0]
        if keyword not in args:
            return callback
        def wrapper(*a, **ka):
            ka[keyword] = db
            rv = callback(*a,**ka)
            db.end_request()
            return rv
        return wrapper

do i need to keep this db.end_request() ?
it is working at the end of the wsgi processing