Skip to content

Commit

Permalink
pythongh-117151: increase default buffer size of shutil.copyfileobj()…
Browse files Browse the repository at this point in the history
… to 256k.

it was set to 16k in the 1990s.
it was raised to 64k in 2019. the discussion at the time mentioned another 5% improvement by raising to 128k and settled for a very conservative setting.

it's 2024 now, I think it should be revisited to match modern hardware. I am measuring 0-15% performance improvement when raising to 256k on various types of disk. there is no downside as far as I can tell.

this function is only intended for sequential copy of full files (or file like objects). it's the typical use case that benefits from larger operations.

for reference, I came across this function while trying to profile pip that is using it to copy files when installing python packages.
  • Loading branch information
rmmancom committed Apr 30, 2024
1 parent 287d939 commit a8c9407
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion Lib/shutil.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
else:
_winapi = None

COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 64 * 1024
COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 256 * 1024
# This should never be removed, see rationale in:
# https://bugs.python.org/issue43743#msg393429
_USE_CP_SENDFILE = (hasattr(os, "sendfile")
Expand Down

0 comments on commit a8c9407

Please sign in to comment.