Read Json in chunks

[parasml]

Hi @igorborgest,

I am reading my JSON file in chunks as it is too big in size, In below code.

df = wr.s3.read_json(path1, chunksize=2, lines=True)

df type return is Generator.

I am struggling to save above chunk (df) to parquet format, please share your thoughts on how to achieve this or if we can save it to dataframe.

Thanks,
Prashan

[parasml]

Hi @igorborgest

Thanks for all the help in past.

Just tried with below code, but still not able to read json lines in for loop. Please suggest.

code:

df = wr.s3.read_json(path1, chunksize=5, lines=True)
i = 0
for row in df:
print("row = ", row)

print("row = ", json.loads(row))
print("--------------")

=======================================

Regards,
Prashan

[igorborgest]

Hi @parasml,

Actually I found a bug that I.ve just fixed in the json reading with chunksize.

Do you mind to test our development branch?

pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev

Example:

for df in wr.s3.read_json(paths, lines=True, chunksize=1):
    print(df)

I would like to make sure that it will be fixed on our next version 1.2.0. Thanks!

[igorborgest]

Released on version 1.2.0