Async in process in Threads in Core

[misterpilou]

Okay, i'm building a web crawler in python and after using aiomultiprocessing, i wanted to wrap it up with stackless python. But i think dask can fit my needs perfectly. Just wondering if i could do this with Dask, i mean wrap asyncio tasks in process, process in threads (actually just realize that i'm surely worng about that considering what i read) and then wrapping threads worker into cores. All that being possibly distributed. Most resources on the web say that dask is for scientific computation, ML etc... I will do things that later in my crawler but i think like dask is far more than that from what i saw

[martindurant]

It is a little hard for me to imagine quite what your code will be like - seems to be quite a lot of nested concepts here.
However, the short answer is "yes", Dask can be used to parallelise many workloads, not only scientific/numerical ones. The distributed client supports the concurrent futures API (e.g., https://distributed.readthedocs.io/en/latest/api.html#distributed.Client.submit ), which may well be enough to get you going, but can also be explicitly async (https://distributed.readthedocs.io/en/latest/api.html#asynchronous-methods , https://distributed.readthedocs.io/en/latest/asynchronous.html

[misterpilou]

Yes as in python you can bind asynchronous worker to ProcessPoolExecutor and i see David Beazley wrapping it in ThreadPoolExecutor i was wondering if something like this could be made with Dask. (The thing being distributed is even more evil). Maybe it's not even necessary. I will do some tests

[misterpilou]

I wanted to do some tests but how should i run my async function (async def ...). Maybe i didn't search enought but it feel like map and submit can't run async def even with AioClient. I tried to map an async def with a list of url, parse it the same way then client.gather() but i've got a "can't pickle coroutine objects". What would be the workaround for that. I know i'm out of subject from the current issue. I could open another one if you want

[martindurant]

If you wanted to have async tasks running on the worker side, you'd have to pass a function that calls them and explicitly put them on the event loop that exists in the worker. Something like

def runner():
    loop = get-my-loop
    return loop.run(coroutine_func)   # or sync()

Since you're not actually achieving asyncronous operation here, I wonder whether there's any point to this.