[RFC] Option 2 for Libcloud async support #1027

Closed
wants to merge 7 commits into
from

Conversation

Projects
None yet
2 participants
@tonybaloney
Contributor

tonybaloney commented Apr 7, 2017

This is the second design concept for async support in Libcloud.

This will only support Python 3.5+ (but cleanly handle older versions)

The concept here is to have Asynchronous Mixins, LibcloudConnection uses requests and LibcloudAsyncConnection uses aiohttp for async transport.

https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/connection_async.py#L22-L42

The LibcloudAsyncConnection is an implementation detail of AsyncConnection, which is the API for the drivers to consume.
https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/common/base.py#L742-L778

The drivers then use this mixin for their custom connection classes, e.g.
https://github.com/tonybaloney/libcloud/blob/b0e06700bd206b1af10663e2f8c006303536bf4b/libcloud/storage/drivers/google_storage.py#L77

They then inherit from StorageAsyncDriver, which uses a new set of base methods, e.g. iterate_containers_async and can be implemented like this:

https://github.com/tonybaloney/libcloud/blob/b0e06700bd206b1af10663e2f8c006303536bf4b/libcloud/storage/drivers/google_storage.py#L215-L224

Now the consumer can more or less do this:

from libcloud.storage.providers import get_driver
from libcloud.storage.types import Provider

import asyncio

import sys

KEY = sys.getenv('GOOGLE_KEY')
SECRET = sys.getenv('GOOGLE_SECRET')

GoogleStorageDriver = get_driver(Provider.GOOGLE_STORAGE)
driver = GoogleStorageDriver(key=KEY, secret=SECRET)

def do_stuff_with_object(obj):
    print(obj)

async def run():
    tasks = []
    async for container in driver.iterate_containers_async():
        async for obj in driver.iterate_container_objects_async(container):
            tasks.append(asyncio.ensure_future(do_stuff_with_object(obj)))
    await asyncio.gather(*tasks)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())
loop.close()

FYI, this currently doesn't work, it's a rough sketch.
Also, it hides some of the imports for aiohttp and asyncio so that existing use cases don't break for Python 3.5<

tonybaloney added some commits Apr 2, 2017

@tonybaloney

This comment has been minimized.

Show comment
Hide comment
@tonybaloney

tonybaloney Apr 7, 2017

Contributor

Ideally here, the driver methods should not need to change, the async methods should patch out calls to self.connection.request with self.connection.request_async.
Also I haven't exposed the ability to provide the event loop with the driver constructor, but this would be simple to achieve.

Contributor

tonybaloney commented Apr 7, 2017

Ideally here, the driver methods should not need to change, the async methods should patch out calls to self.connection.request with self.connection.request_async.
Also I haven't exposed the ability to provide the event loop with the driver constructor, but this would be simple to achieve.

@pquentin

This comment has been minimized.

Show comment
Hide comment
@pquentin

pquentin Apr 7, 2017

Contributor

This looks good. Thanks for the detailed sketch! 👍

However, I'm worried about needing to reimplement every driver method as async. You're saying the drivers methods should not need to change. But even if self.connection.request in monkeypatched to be made async, you still need await to call it, which does not work without modifying iterate_containers, right? Am I missing something?

Contributor

pquentin commented Apr 7, 2017

This looks good. Thanks for the detailed sketch! 👍

However, I'm worried about needing to reimplement every driver method as async. You're saying the drivers methods should not need to change. But even if self.connection.request in monkeypatched to be made async, you still need await to call it, which does not work without modifying iterate_containers, right? Am I missing something?

@pquentin

This comment has been minimized.

Show comment
Hide comment
@pquentin

pquentin Sep 28, 2017

Contributor

People who have more familiarity with async in Python than us continue to think about adding sync/async APIs to Python APIs. For exemple, there's ongoing work on urllib3. When urllib3 is ready, then the work will start on requests. At that point we will be in a better position to add an async API. Until then, I think it's not realistic and we should close those two async PRs.

Contributor

pquentin commented Sep 28, 2017

People who have more familiarity with async in Python than us continue to think about adding sync/async APIs to Python APIs. For exemple, there's ongoing work on urllib3. When urllib3 is ready, then the work will start on requests. At that point we will be in a better position to add an async API. Until then, I think it's not realistic and we should close those two async PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment