Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't write any bytes-like object to S3 #360

Closed
gilbsgilbs opened this issue Oct 8, 2019 · 0 comments · Fixed by #361
Closed

Can't write any bytes-like object to S3 #360

gilbsgilbs opened this issue Oct 8, 2019 · 0 comments · Fixed by #361

Comments

@gilbsgilbs
Copy link
Contributor

gilbsgilbs commented Oct 8, 2019 •
edited

open built-in lets you use write method on any object as long as it implements Buffer Protocol. However, smart_open restricts writeable types to only bytes, bytearray and memoryview. Is there a specific reason behind that? I'm using smart_open to upload to S3 files that are created using pyarrow. PyArrow has a (new?) type called Buffer which implements Buffer Protocol. Therefore, when I try to write it through smart_open, it just crashes while it just works with built-in open function.

I can easily fix the crash by skipping the type check in smart_open which seem to work flawlessly in my case. See #361

Would you be confident with such change? Not sure why the isinstance check was added in the first place. The error thrown by Python is already very explicit:

TypeError: a bytes-like object is required, not 'xxx'
gilbsgilbs added a commit to gilbsgilbs/smart_open that referenced this issue Oct 8, 2019
@gilbsgilbs
Support writing any bytes-like object to S3.
97d4fc5 
Fixes piskvorky#360
@gilbsgilbs gilbsgilbs mentioned this issue Oct 8, 2019
Merged
gilbsgilbs added a commit to gilbsgilbs/smart_open that referenced this issue Oct 9, 2019
@gilbsgilbs
Support writing any bytes-like object to S3.
423d7db 
Fixes piskvorky#360
mpenkov pushed a commit that referenced this issue Oct 13, 2019
@gilbsgilbs @mpenkov
Support writing any bytes-like object to S3. (#361)
b921446 
* Support writing any bytes-like object to S3.

Fixes #360

* Add a link to python documentation about buffers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support writing any bytes-like object to S3. gilbsgilbs/smart_open
1 participant
@gilbsgilbs