New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read_json(lines=True) broken for s3 urls in Python 3 (v0.20.3) #17200

Closed
alph486 opened this Issue Aug 8, 2017 · 6 comments

Comments

Projects
None yet
5 participants
@alph486
Copy link
Contributor

alph486 commented Aug 8, 2017

Code Sample, a copy-pastable example if possible

Using Python

import pandas as pd
inputdf = pd.read_json(path_or_buf="s3://path/to/python-lines/file.json", lines=True)

The file is similar to:

{"url": "blah", "other": "blah"}
{"url": "blah", "other": "blah"}
{"url": "blah", "other": "blah"}

Problem description

When attempting to read a python lines file into a DataFrame using the s3 protocol, the above code will error with:

2017-08-08 11:06:14,225 - image_rank_csv - ERROR - initial_value must be str or None, not bytes
Traceback (most recent call last):
  File "image_rank_csv.py", line 62, in run
    inputdf = pd.read_json(path_or_buf="s3://path/to/python-lines/file.json", lines=True)
  File "...env/lib/python3.6/site-packages/pandas/io/json/json.py", line 347, in read_json
    lines = list(StringIO(json.strip()))
TypeError: initial_value must be str or None, not bytes

This works fine if the file is local, e.g.:

import pandas as pd
inputdf = pd.read_json(path_or_buf="/local/path/to/python-lines/file.json", lines=True)

Expected Output

Expect to successfully read the file and error above not to occur.

My current thinking is that when we get the file handle: https://github.com/pandas-dev/pandas/blob/v0.20.3/pandas/io/json/json.py#L333 , you delegate to s3fs, which documents that it only operates in Binary mode. Therefore when you read(): https://github.com/pandas-dev/pandas/blob/v0.20.3/pandas/io/json/json.py#L335, Therefore passing to StringIO will fail here: https://github.com/pandas-dev/pandas/blob/v0.20.3/pandas/io/json/json.py#L347 . Maybe it needs a different handler for BytesIO?

Output of pd.show_versions()

[paste the output of ``pd.show_versions()`` here below this line] ``` INSTALLED VERSIONS ------------------ commit: None python: 3.6.1.final.0 python-bits: 64 OS: Darwin OS-release: 16.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.20.3
pytest: None
pip: 9.0.1
setuptools: 36.2.7
Cython: None
numpy: 1.12.0
scipy: 0.19.1
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: None
s3fs: 0.1.2
pandas_gbq: None
pandas_datareader: None

</details>

@alph486 alph486 changed the title read_json(lines=True) broken for s3 urls in Python 3 read_json(lines=True) broken for s3 urls in Python 3 (v0.20.3) Aug 8, 2017

@gfyoung gfyoung added the IO JSON label Aug 8, 2017

@gfyoung

This comment has been minimized.

Copy link
Member

gfyoung commented Aug 8, 2017

Well we do have a BytesIO class in pandas.compat. If we can condition on the data returned to us, that would be reasonable I think.

@alph486

This comment has been minimized.

Copy link
Contributor

alph486 commented Aug 8, 2017

@gfyoung I'm not intimately familiar with the codebase, but I have a (possibly naive) fix that just attempts to decode the json based on whatever encoding is. It seems to fix. Would you like to review if do a PR?

@gfyoung

This comment has been minimized.

Copy link
Member

gfyoung commented Aug 8, 2017

I'm not intimately familiar with the codebase

Don't worry, that's a pretty tall order 😄

Absolutely! Submit a PR, and we'll certainly review it.

@TomAugspurger

This comment has been minimized.

Copy link
Contributor

TomAugspurger commented Aug 8, 2017

FYI, here's how we handle it on the CSV side:

if compat.PY3 and is_text and\

that f would be an instance of S3File, which is one of the classes in need_text_wrapping. By the TextIOWrapper can go around buffer of bytes and it'll do the encoding. You might be able to reuse parts of that for read_json, or just do something similar.

@alph486

This comment has been minimized.

Copy link
Contributor

alph486 commented Aug 8, 2017

Naive PR submitted ;) @gfyoung

@jreback jreback added this to the 0.21.0 milestone Aug 8, 2017

@jreback jreback modified the milestones: 0.21.0, Next Major Release Sep 23, 2017

@jreback jreback modified the milestones: Next Major Release, 0.21.1 Nov 27, 2017

jreback added a commit that referenced this issue Nov 27, 2017

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Dec 8, 2017

TomAugspurger added a commit that referenced this issue Dec 11, 2017

@sylviawhoa

This comment has been minimized.

Copy link
Contributor

sylviawhoa commented Jan 2, 2018

workaround:

with s3_filename.open("r") as s3_file:
            data=s3_file.read()
df = pd.read_json(data.decode('utf8'), lines=True)

my data was gzipped - you prob don't need the decode...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment