New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(Optionally) Return futures #327
Comments
I wrote a terrible way to do this when working with Tornado: class AsyncDockerClient():
'''Completely ridiculous wrapper for a Docker client that returns futures
on every single docker method called on it, configured with an executor.
If no executor is passed, it defaults to ThreadPoolExecutor(max_workers=2).
'''
def __init__(self, docker_client, executor=None):
if executor is None:
executor = ThreadPoolExecutor(max_workers=2)
self._docker_client = docker_client
self.executor = executor
def __getattr__(self, name):
'''Creates a function, based on docker_client.name that returns a
Future. If name is not a callable, returns the attribute directly.
'''
fn = getattr(self._docker_client, name)
# Make sure it really is a function first
if not callable(fn):
return fn
def method(*args, **kwargs):
return self.executor.submit(fn, *args, **kwargs)
return method The right way to do this would be to just extend your base class and change the underlying client. /cc @minrk |
+1 |
2 similar comments
+1 |
+1 |
Is there any status for adding futures support to docker-py? Would like to see a solution within docker-py that isn't just utilizing asyncio.run_in_executor(INSERT DOCKER-PY COMMAND). EDIT: perhaps this could be a separate project? The "simple" path would be duplication of code, but i could see how that would be messy... |
Here is my approach using requests_futures:
Returning a future at the request level mucks up everything but at least this way I get co-operative threading. Edit: My understanding is that this is still blocking at the IO level. |
Hi everyone. I'm thinking of this too and wonder if it would be a good idea to have the Client class to use aiohttp's Session class instead of requests? The idea is that requests doesn't handle async, so it would be nice to get async methods for every action. (Does that make make sense?) |
https://pypi.python.org/pypi/yieldfrom.requests is one option The solution for all these blocking libraries, is to use async/await coroutines, but they will not be usable until 2020-01-01 when all currently supported Python versions without async/await are both EOL. Ideally requests, docker-py, etc etc will create an asyncio version of their libraries then import it into a legacy blocking version. |
twisted now supports asyncio, and so you can create code that works with https://github.com/twisted/treq instead. This way there can be a blocking api that wraps the Twisted calls, and there can be a non-blocking api using Twisted or asyncio via Twisted on Python 3. |
However it might be better to use the https://sans-io.readthedocs.io/ approach, then people can plug their own session in. |
I think a BYO-Netlib could be good way to deal with this, though I know that would be a much larger effort. My guess is that we would still need an asynchronous docker client class to enable async/await for all the functions |
you could use pypi https://pypi.python.org/pypi/deferred just need to poke @hawkowl or @glyph to upload a fresh one |
this way you can write a blocking request session wrapper that takes deferreds and the aiodocker-py library can replace it |
If this would be useful we'd be quite happy to help out; just let us know! However, |
@glyph well I'm really looking to have the async/await support here: twisted/twisted#489 also wheels |
@graingert Oh! Well in that case, yes, we should definitely do that :). |
and maybe get it off the launchpad and into Git(Hu|La)b. |
It's currently on GH as https://github.com/mikeal/deferred |
(The plan is to move it into the Twisted org, of course.) |
ah I found it on https://launchpad.net/deferred |
Yeah that was the original location. |
FWIW, there is also txaio which would allow an implementation that runs on both Twisted and asyncio (user choice) .. Disclosure: I am affiliated with txaio |
@cecton have a look at this: https://github.com/aio-libs/aiodocker |
@graingert Thx! I already saw it (too late) but I wanted something closer to docker-py and more official (because then Docker maintain it). But my approach is bad too... I end up adapting all the code to put "async" and "await" keywords everywhere. This is definitely not what I wanted. I wanted some kind of wrapper on docker-py that would make the call asynchronous but now I kinda realize it's impossible or I totally missed something. .... actually.... I just realize now that it is under aio-libs so it's maintain by the devs of aiohttp. 😕 great lol |
To whom is interested: I managed to wrap docker-py: https://github.com/tenforce/docker-py-aiohttp It can certainly be ported to other asyncio libraries using the same principles. The core of the principles are located here: https://github.com/cecton/requests2aiohttp In brief: I made a class that is meant to override It's not state of the art but I added integration tests for the same Docker versions tested by docker-py and it seems very stable. |
In order for me to use this in tornado and other asynchronous frameworks, I either have to make direct calls to the Docker API (using an async http client) or wrap a threadpool executor around this.
I'd like to see if there's a way for us to provide both a synchronous and asynchronous version of docker-py.
The text was updated successfully, but these errors were encountered: