Work with files asynchronously

[asvetlov]

There is a very old tech debt in aiohttp: the library uses blocking synchronous calls to work with files.

It should be fixed by replacement such direct calls by loop.run_in_executor().

Affected parts:

• web_fileresponse.py
• payload.py

Splitting the task into two pull requests makes sense in term of easier review.

The issue is pretty easy and straightforward, it waits for a champion.

[aio-libs-bot]

GitMate.io thinks possibly related issues are #764 (Too many open files), #20 (Refactor files handling.), #1067 (will there be huge file upload support ?), #72 (HTTP POST not working), and #44 (Please document UnixConnection work).

[socketpair]

Do not forget to open() files also in a thread.

I think, internal API should be the following:

import aiofiles

file = await aiofiles.open()
result = await file.read()
result = await file.write()
await file.chmod()
await file.fsync()
...

UPD: https://github.com/Tinche/aiofiles buth this solution is overingeneered. I would write my own with implementing each method explicitly.

[asvetlov]

I don't want to use aiofiles.
run_in_executor is just enough for the library needs.
aiohttp calls open() in file_response.py only, client API accepts already opened regular IO objects.
Wrapping them into aiofiles looks more complex than explicit thread pool usage.

[chirgjn]

👋 would love to pick this up, if not fixed already

[asvetlov]

Please go ahead!

[kxepal]

This could raise a holywar, but I'll risk to ask you: why not aiofiles?

[asvetlov]

Did you read messages above?
I can send a ball back: why aiofiles despite the fact that we need only run in executor a couple IO calls?

[kxepal]

Yes, I did, but some abstraction may be useful to not care about that bit if it will eventually have changed. If that's the only reason than it satisfied my curiousity (: