- Default buffer size of file objects from
pfio.v2.S3
is 16MB, while default size ofshutil.COPY_BUFSIZE
is 64 as of Python 3.10. To avoid using default size, additional argument would help us align the buffer size:
import shutil
from pfio.v2 import open_url
with open_url('s3://bucket/very-large-file', 'wb') as dst:
with open('very-large-local-file', 'rb') as src:
shutil.copyfileobj(src, dst, length=16*1024*1024)