Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS, WebHDFS, HTTP I/O is not encoding-aware #146

Closed
mpenkov opened this issue Nov 24, 2017 · 2 comments
Closed

HDFS, WebHDFS, HTTP I/O is not encoding-aware #146

mpenkov opened this issue Nov 24, 2017 · 2 comments

Comments

@mpenkov
Copy link
Collaborator

mpenkov commented Nov 24, 2017

The HDFS and Http subsystems does not do any text encoding or decoding.

  • HdfsOpenRead returns the stdout from a Popen, so it will be in whatever encoding Popen uses for pipes (could be UTF-8, or raw bytes)
  • HdfsOpenWrite writes bytes directly into the stdin of a Popen
  • WebHdfsOpenRead reads bytes directly (requests.get returns bytes when stream=True)
  • HttpOpenRead also reads bytes directly (uses requests in the same manner as above).
@mpenkov mpenkov changed the title HDFS I/O is not encoding-aware HDFS, WebHDFS, HTTP I/O is not encoding-aware Nov 24, 2017
menshikh-iv pushed a commit that referenced this issue Nov 25, 2017
…sed cases. Fix #145 (#147)

Resolve Issue #145, add warnings for Issue #146
@mpenkov
Copy link
Collaborator Author

mpenkov commented Apr 22, 2018

@menshikh-iv I think this is resolved by #185

@menshikh-iv
Copy link
Contributor

@mpenkov 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants