Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ValueError: buffer size must be strictly positive when opening an empty file in S3 with "rb" mode #269

Closed
belltailjp opened this issue Mar 1, 2022 · 0 comments · Fixed by #270
Labels
cat:bug Bug report or fix.
Milestone

Comments

@belltailjp
Copy link
Member

In case we try to open an empty (zero-bytes) file on S3 in "rb" mode using pfio v2 API with the default buffering option, we get this error.

>>> import pfio; pfio.__version__
'2.2.0'
>>> with pfio.v2.open_url('s3://<bucket>/empty_file.dat', 'rb') as f:
...     pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/pfio/v2/fs.py", line 310, in open_url
    with fs.open(filename, mode) as fp:
  File "/usr/local/lib/python3.8/site-packages/pfio/v2/s3.py", line 407, in open
    obj = io.BufferedReader(obj, buffer_size=bs)
ValueError: buffer size must be strictly positive

This is because
https://github.com/pfnet/pfio/blob/2.2.0/pfio/v2/s3.py#L401-L407

    bs = min(obj.content_length, DEFAULT_MAX_BUFFER_SIZE)

this will make bs to be zero, which is then passed to BufferedReader as the buffer size causing the error.

In case the content length of the object to be read is zero, we of course don't need buffering at all, so I think we can simply skip wrapping obj by BufferedReader.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cat:bug Bug report or fix.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants