`shutil.copyfileobj()` doesn't flush at end of copy

[freakboy3742]

Bug report

Bug description:

If you use shutil.copyfileobj() to copy a file-like object to a file-like object, the destination isn't flushed at the end of the copy. This can lead to the destination file being in a written, but unflushed state, which could be counterintuitive.

This was found as part of the Emscripten build script, which works reliably on Python 3.13.5, but breaks on 3.14.0b3:

import shutil
import tempfile
from urllib.request import urlopen

workingdir = "./tmp"
shutil.rmtree(workingdir, ignore_errors=True)
with tempfile.NamedTemporaryFile(suffix=".tar.gz") as tmp_file:
    with urlopen(
        "https://github.com/libffi/libffi/releases/download/v3.4.6/libffi-3.4.6.tar.gz"
    ) as response:
        shutil.copyfileobj(response, tmp_file)
    # Uncomment this flush to make the example work on 3.14
    # tmp_file.flush()
    shutil.unpack_archive(tmp_file.name, workingdir)

I'm guessing this discrepancy was caused by #119783 (/cc @morotti), which increased the buffer size for copyfileobj(). I'm guessing that the bug existed prior to that change, but increasing the buffer size also increased the potential exposure to buffer flush errors/inconsistencies.

I'd argue that the documented API for copyfileobj() reads that the method is a "complete" operation, including an implied flush on completion.

However, I can also see the argument that the decision to flush should be left to the user - in which case, the resolution for this issue would be updating the documentation to make it clear that users should flush after a copy.

There could also be an argument for adding a flush argument (defaulting to True) so that flushing could be optionally omitted for cases where the overhead of a flush might matter.

CPython versions tested on:

3.14

Operating systems tested on:

macOS

Linked PRs

• gh-135648: Document that shutil.copyfileobj doesn't flush #135737

[vstinner]

I would suggest to always call flush().

[morotti]

Hello,

TL;DR you need to flush

It's a long standing bug in your code. It opens a file to write data, then opens the file again to read data (new file descriptor), but your code did not write the data between the two so there is nothing to read back.

It's not related to copyfileobj(), you'd have the same problem with write() and all other I/O functions.
All file objects and functions in python are buffered by default, the filesystem is likely to be buffered too, you need to flush to ensure data was written. (It's not limited to python, you'd have the same issue in most programming languages).

The issue is not due to the PR you linked. It's likely due to other improvements we made to the buffering subsystem to improve I/O performance. There are quite a few. (I've measured Python 3.14 to be as much as 5 times faster on basic code to read a file from HTTP and write to disk, similar to your code).
Overall, I'd summarize that the buffering was bugged and could have been set to as little as 4kB on Linux and 8kB on Windows. Your code was buggy but it might only have been noticeable on tiny files.

longer explanation:

1. You need to flush the data

2. Or you need to end the with open(...): block before you start reading the file. The end of the block should imply flush() and close().

Caveat on ending the with block, you're using tempfile functions to open the file and some tempfile functions may automatically delete the file at the end of the with block. :D

3. Some people may say you need to fsync() to ensure data is written to the device, but that's very slow and unnecessary on modern OS and filesystems.

[morotti]

On whether copyfileobj should flush, copyfileobj is a stream API that operates on raw streams, it has no concept of file size or buffer.

IMO: copyfilobj() can't flush because it supports objects that are not flushable.
IMO: copyfileobj() can't support a flush argument because the argument would be broken depending on the type of object that is passed. (That would be a major problem in libraries using copyfileobj(flush=True), they would fail when given non-buffered objects to read/write from).

That being said, I can understand the confusion. The doc could certainly state that it doesn't flush.

[freakboy3742]

It's not related to copyfileobj(), you'd have the same problem with write() and all other I/O functions.

I'm aware of how file descriptors work; what I'm mostly flagging is that the behavior of copyfileobj() was contrary to my expectation. In every other instance I can think of, "copy" is an atomic and complete transaction, and in this case, and the documentation for copyfileobj() doesn't say anything to suggest otherwise. I completely understand how this issue manifested at a technical level, and that based on the implementation, this bug would have existed in some form in the previous code - the recent changes 3.14 just had the effect of forcing the bug to the surface.

The issue is not due to the PR you linked. It's likely due to other improvements we made to the buffering subsystem to improve I/O performance.

Understood. And those optimisations are definitely appreciated; but they've highlighted an issue that needs a resolution, even if that is only in documentation (plus a fix in the Emscripten build script).

On whether copyfileobj should flush, copyfileobj is a stream API that operates on raw streams, it has no concept of file size or buffer.

copyfileobj() is documented as operating on "file-like objects", and flush() is part of the definition of IOBase. To me, that seems to suggest that flush() should be reliably available. The glossary entry for "file object" says "file" and "stream" are interchangable (and I understand the extent to which the two aren't strictly interchangeable, but is the documented interface).

Have you got a specific example in mind of a stream that might want to use copyfileobj(), but wouldn't provide flush()?

[morotti]

the behavior of copyfileobj() was contrary to my expectation

I certainly agree that the absence of flush could surprise users. It's most definitely worth a comment in the doc.
I see you are a python core developer in your github profile, can you send a PR to update the doc?

copyfileobj() does not flush(). data may not be written immediately when using buffered file-like objects.

Have you got a specific example in mind of a stream that might want to use copyfileobj(), but wouldn't provide flush()?

Again, I don't really have time to spend hours to look into the interpreter code right now. ^^
I was thinking of some edge cases like FileIO objects, mmap files, socket objects, etc... (lots of things that can potentially be used as a file like object).
I tried a few things just now. They either support flush or they can't be used as a file object. I guess you're right and flush is generally available.

If flush() is generally available, the next question is whether copyfileobj() should flush automatically? 🤔

That leaves the major question of performance. While I agree that flushing could make sense for user usability, I'm deeply concerned about the performance cost of an extra flush() syscall which is useless for nearly 99% of usage.
TL;DR I think copyfileobj() should not flush out-of-the-box due to performance impact.

The typical usage is to have a with open(): and sh.copyfileobj() and end the block. It works fine, the data is flushed and copied at the end of the with block.
Data will always be flushed when the file-like object is closed (either close() or reaching the end of the with block).
Adding an explicit flush() as part of copyfileobj() would be adding an expensive syscall every time, which is useless the vast majority of the time.
(Your usage was particular for reading and writing to tempfiles, which needs some care in all languages. it's not the usual pattern.)

I've been doing optimizations to make pip twice as fast and I spent quite a lot of time looking at I/O code recently.
A flush() call can be ~5 microseconds on my machine. It's a syscall. It's expensive.
If copyfileobj() were to flush(), it would make all usage of it potentially a few percent slower, for no practical reason. It doesn't benefit the enormous majority of usage that is just copying in a with block and immediately close.

There are a lot of nice I/O improvements coming with python 3.14, like an large reduction in stats() call because one guy fixed a lot of code in the interpreter that was needlessly calling stats() when files are opened and manipulated. syscalls are expensive.
So I think we should not add a preemptive call to flush as part of copyfileobj, since it's counter productive the vast majority of the time.

Sorry for the long text.

[freakboy3742]

I've been doing optimizations to make pip twice as fast and I spent quite a lot of time looking at I/O code recently. A flush() call can be ~5 microseconds on my machine. It's a syscall. It's expensive. If copyfileobj() were to flush(), it would make all usage of it potentially a few percent slower, for no practical reason. It doesn't benefit the enormous majority of usage that is just copying in a with block and immediately close.

That's a fair comment. On that basis, it sounds like documentation is our best option. I'll prepare a PR shortly.