Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[RFC] Option 2 for Libcloud async support #1027
This is the second design concept for async support in Libcloud.
This will only support Python 3.5+ (but cleanly handle older versions)
The concept here is to have Asynchronous Mixins,
The drivers then use this mixin for their custom connection classes, e.g.
They then inherit from
Now the consumer can more or less do this:
from libcloud.storage.providers import get_driver from libcloud.storage.types import Provider import asyncio import sys KEY = sys.getenv('GOOGLE_KEY') SECRET = sys.getenv('GOOGLE_SECRET') GoogleStorageDriver = get_driver(Provider.GOOGLE_STORAGE) driver = GoogleStorageDriver(key=KEY, secret=SECRET) def do_stuff_with_object(obj): print(obj) async def run(): tasks =  async for container in driver.iterate_containers_async(): async for obj in driver.iterate_container_objects_async(container): tasks.append(asyncio.ensure_future(do_stuff_with_object(obj))) await asyncio.gather(*tasks) loop = asyncio.get_event_loop() loop.run_until_complete(run()) loop.close()
FYI, this currently doesn't work, it's a rough sketch.
Ideally here, the driver methods should not need to change, the async methods should patch out calls to
This looks good. Thanks for the detailed sketch!
However, I'm worried about needing to reimplement every driver method as async. You're saying the drivers methods should not need to change. But even if
People who have more familiarity with async in Python than us continue to think about adding sync/async APIs to Python APIs. For exemple, there's ongoing work on urllib3. When urllib3 is ready, then the work will start on requests. At that point we will be in a better position to add an async API. Until then, I think it's not realistic and we should close those two async PRs.