New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Copying large files using mmap-able source streams may exhaust available memory and fail #13071
Comments
This clearly is a regression as it used to work in previous versions, but broke since the commit you linked. The obvious solution is to use chunking with userspace stream writes, that would look something like this: https://gist.github.com/nielsdos/d4cc1fbc3db1abe88f517919b2ad61e8 Out of curiosity I tried an alternative fix by fixing this at the stream_copy_to_stream_ex side, in the mmap code. So all in all, I prefer the first patch but wrapped in a nicer API. |
I have had a proper look and agree that the first solution is better - fwrite and stream_copy_to_stream should be consistent. I think write chunking makes sense for user wrappers as there's extra buffering in callbacks (variables) which is what causes issue here. And as you say chunking can be controlled by users so if anyone wants to increase it, they can (although they need to open stream first so not that useful for Thinking about it, it might be also interesting to see how those big writes work for other streams like TLS and if it results in bigger memory usage there as well (this won't be PHP allocated so users might not see it but might increase the process / system memory usage). Might not be an issue but would be probably good to do some testing so I will add this to my TODO list. |
…ay exhaust available memory and fail Commit 5cbe5a5 disabled chunking for all writes to streams. However, user streams have a callback where code is executed on data that is subject to the memory limit. Therefore, when using large writes or stream_copy_to_stream/copy the memory limit can easily be hit with large enough data. To solve this, we reintroduce chunking for userspace streams. Users have control over the chunk size, which is neat because they can improve the performance by setting the chunk size if that turns out to be a bottleneck. In an ideal world, we add an option so we can "ask" the stream whether it "prefers" chunked writes, similar to how we have php_stream_mmap_supported & friends. However, that cannot be done on stable branches.
…ay exhaust available memory and fail Commit 5cbe5a5 disabled chunking for all writes to streams. However, user streams have a callback where code is executed on data that is subject to the memory limit. Therefore, when using large writes or stream_copy_to_stream/copy the memory limit can easily be hit with large enough data. To solve this, we reintroduce chunking for userspace streams. Users have control over the chunk size, which is neat because they can improve the performance by setting the chunk size if that turns out to be a bottleneck. In an ideal world, we add an option so we can "ask" the stream whether it "prefers" chunked writes, similar to how we have php_stream_mmap_supported & friends. However, that cannot be done on stable branches.
* PHP-8.2: Fix GH-13071: Copying large files using mmap-able source streams may exhaust available memory and fail
* PHP-8.3: Fix GH-13071: Copying large files using mmap-able source streams may exhaust available memory and fail
Description
The following script:
up
protocol which doesfwrite
under the hood.copy
operation of a source file into a destination file using the defined stream.Ideally, this stream wrapper should be able to copy any file, but if the file is large enough it dies because memory gets exhausted.
Example to reproduce
Resulted in this output:
While no output and a successful copy was expected.
This seems to happen after 5cbe5a5, where chunking was disabled for streams.
copy
passes a buffer to_php_stream_write_buffer
that is as large asPHP_STREAM_MMAP_MAX
, defined at 512MB. This means that files smaller than 512MB require at least their size in terms of memory to be copied, and files larger than 512MB requires at least 512MB of memory available.This has been tested with PHP 8.0, 8.1, and now lastly with
PHP Version
PHP 8.3.1
Operating System
No response
The text was updated successfully, but these errors were encountered: