-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
StreamReader.iter_chunked(n) yields chunk_sizes < n before end of stream #7929
Comments
It says 'with maximum size limit', which atleast to me, suggests that chunks may be smaller. But, feel free to make a PR to make it clearer. I read that as a deliberate design. To have chunks of a fixed size would require not sending data that is already available, which could cause unnecessary delays. |
Got it, thanks! So in order to get precise chunks of size I think it's important to address these since the chunked reading feature is a widely used part of I can make a PR updating docs. Let me know if there's any other relevant information/context. |
I guess readexactly() works, or just adding chunks together yourself, maybe something like:
I guess it's possible we could add an option if there's some solid use cases for it, but I'm not clear what those are yet. I think the main use case for the chunk size is to limit the amount of memory the application uses (e.g. downloading a large file and writing it to disk, on a tiny embedded system you may want to limit the amount of memory used for this to 1KB or something, while a desktop application might be good with 10MB as a limit). |
My use case was for blowfish decryption, which was done on a chunk by chunk basis. This behavior was an issue because for any chunk with length # from iteration i from iteration i+1
decrypt(chunk[:n]) + decrypt(chunk[n:]) is not the same as # from iteration i
decrypt(chunk) which is what I want. I don't know any other use cases off the top of my head but I feel as though reading constant size blocks from a stream should be common enough. |
This feels to me like an itertools job. Looks like there are async versions of itertools at https://asyncstdlib.readthedocs.io/en/stable/source/api/itertools.html and https://aioitertools.omnilib.dev/en/stable/api.html (the former doesn't look like it has batched() yet, and the latter appears to have named it chunked()). |
Describe the bug
I was having compatibility issues when moving from requests to aiohttp in my application. Issue seems to be that aiohttp's iter_chunked method may yield small chunks in the middle of the stream, unlike requests.Response.iter_content(n).
Not sure if this behavior is intended but it was not indicated by the documentation or the source code.
To Reproduce
Run this when downloading a large file.
Expected behavior
All chunk sizes are 6144 except the last one.
Logs/tracebacks
aiohttp Version
multidict Version
yarl Version
OS
macOS 12
Related component
Client
Additional context
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: