How to sync_to_async(executor=gevent.threadpool.ThreadPoolExecutor)?

[allen-munsch]

Is this doable?

sync_to_async(
    Books.objects.all, 
    thread_sensitive=False, 
    executor=gevent.threadpool.ThreadPoolExecutor(max_workers=1)
)

My guess, is that doing all the messy patchwork stuff probably won't be supported officially as part of asgiref, however, it'd be nice as a user to have an interface that shims between the two worlds. However messy it is under the hood.

Use case:

Mounting an ASGI application inside of a WSGI app ( which uses psycogreen, gevent, eventlet, etc ) ensuring all the production core code for a bigger app remains performant, while at the same time enabling users to begin experimenting/mounting, newer ASGI apps, without having to re-write all of the django orm code using turtleorm, sqlalchemy, etc.

Example scenario: "How can I mount a FastAPI app to the inside of Django?"

• https://forum.djangoproject.com/t/django-async-orm/8036
• https://forum.djangoproject.com/t/asynchronous-orm/5925
• replace the default ThreadPoolExecutor with gevent.threadpool.ThreadPoolExecutor rednaks/django-async-orm#7
• psycogreen, gevent, eventlet, psycopg2 and django async orm rednaks/django-async-orm#9
• https://stackoverflow.com/questions/67735453/django-async-orm-sync-to-async-executor-swap-out-concurrent-futures-threadpoolex

Something really wacky along these lines:

try:
    from gevent.threadpool import ThreadPoolExecutor as GThreadPoolExecutor
    from django.conf import settings
    if settings.GEVENT_DJANGO_ASYNC_ORM:
        from gevent import monkey
        monkey.patch_all()
        def monkey_patch_the_monkey_patchers(ex):
            from .patch_gevent import _FutureProxy
            def submit(ex, fn, *args, **kwargs): # pylint:disable=arguments-differ
                print(fn, *args, **kwargs)
                with ex._shutdown_lock: # pylint:disable=not-context-manager
                    if ex._shutdown:
                        raise RuntimeError('cannot schedule new futures after shutdown')
                    future = ex._threadpool.spawn(fn, *args, **kwargs)
                    proxy_future = _FutureProxy(future)

                    # just fake it, maybe?
                    proxy_future.__class__ = concurrent.futures.Future

                    return proxy_future
            ex.submit = submit
            return ex
        MonkeyPoolExecutor = monkey_patch_the_monkey_patchers(GThreadPoolExecutor)
        conf = {"thread_sensitive": False, "executor": MonkeyPoolExecutor(max_workers=1)}
        executor_ = MonkeyPoolExecutor
except Exception as e:
    print('uhoh', e)
    pass
    
all_the_books = await sync_to_async(
    Books.objects.all, 
    **conf
)()

[andrewgodwin]

Yeah, unfortunately gevent has such a different execution model that I don't think we can sensibly support interoperability with it - at least not without someone paid full-time to figure all this out.

You can likely get it working in a hacky way, but making so it works reliably, propagates exceptions properly, handles threadlocals and locking correctly? Probably a mammoth task.