s3.to_json ignores pandas_kwargs

[adrigru]

Describe the bug

I have a pandas DataFrame that I want to write to an S3 bucket as .json.
According to the documentation, I should be able to pass Pandas arguments to the wr.s3.to_json() function.
However, none of the specified arguments actually work.

Environment

https://pastebin.com/raw/pwiTmUkF

To Reproduce

1. Create a Pandas DataFrame
2. Call wr.s3.to_json(path='s3://my-bucket/path/', df=my_df, dataset=True, mode='overwrite', orient='records', lines=True, double_precision=2)
3. The file is created in the S3 as expected, however the orientation, lines, and double_precision arguments are not applied.

[kukushking]

By the looks of it pandas_kwargs aren't respected if dataset=True. Thanks for reporting this!

[adrigru]

Do you think it might be a future feature request to implement this functionality?

I would like to use the mode='overwrite' while exporting in json lines format.

[jaidisido]

Yes @adrigru, I think @kukushking is saying that this is an omission on our part where we inadvertently forgot to add pandas_kwargs here. And we will fix it in the next release

[jaidisido]

Now merged to main and can be tested from the branch:

pip uninstall awswrangler -y
pip install git+https://github.com/awslabs/aws-data-wrangler.git@main

until it's available in the next release

[adrigru]

Thanks for your quick intervention. I tested it now, but it still seems to exhibit the same behaviour.

I run

pip uninstall awswrangler -y
pip install git+https://github.com/awslabs/aws-data-wrangler.git@main

on my notebook instance as instructed. The pip log looks fine. However, running

wr.s3.to_json(path='s3://my-bucket/path/', df=my_df, dataset=True, mode='overwrite', orient='records', lines=True, double_precision=2)

still does not create the format that I expect. Removing the dataset=True parameter together with mode='overwrite' does produce the expected output though.

[jaidisido]

Are you sure you have the latest version of the library (i.e. with the fix)?

When I run this example:

import awswrangler as wr
import pandas as pd

path = "s3://my_bucket/my_prefix/"

wr.s3.delete_objects(path)

df = pd.DataFrame(
    {
        "id": [1, 2, 3],
        "string_object": ["foo", None, "boo"],
        "string": ["foo", None, "boo"],
        "float": [1.0, None, 2.0],
        "int": [1, None, 2],
        "bool": [True, None, False],
        "par0": [1, 1, 2],
        "par1": ["a", "b", "b"],
    }
)

# Dataset False
wr.s3.to_json(
    path=f"{path}test.json",
    df=df,
    dataset=False,
    orient="records",
    lines=True,
    double_precision=2,
)

# Dataset True
wr.s3.to_json(
    path=path,
    df=df,
    dataset=True,
    mode="overwrite"
    orient="records",
    lines=True,
    double_precision=2,
)

They both honor the pandas kwargs in a similar way, that is:

{
  "id": 1,
  "string_object": "foo",
  "string": "foo",
  "float": 1,
  "int": 1,
  "bool": true,
  "par0": 1,
  "par1": "a"
}
{
  "id": 2,
  "string_object": null,
  "string": null,
  "float": null,
  "int": null,
  "bool": null,
  "par0": 1,
  "par1": "b"
}
{
  "id": 3,
  "string_object": "boo",
  "string": "boo",
  "float": 2,
  "int": 2,
  "bool": false,
  "par0": 2,
  "par1": "b"
}

Can you try this example please? If you have different results than me then I suspect you don't have the latest version (i.e. from the branch) installed

[adrigru]

Hey @jaidisido,

sorry for the delay. I just tried out the example above and I get wrong results, i.e. pandas kwargs are being ignored while setting dataset=True.

# dataset True
{"id":{"0":1,"1":2,"2":3},"string_object":{"0":"foo","1":null,"2":"boo"},"string":{"0":"foo","1":null,"2":"boo"},"float":{"0":1.0,"1":null,"2":2.0},"int":{"0":1.0,"1":null,"2":2.0},"bool":{"0":true,"1":null,"2":false},"par0":{"0":1,"1":1,"2":2},"par1":{"0":"a","1":"b","2":"b"}}

# dataset False
{"id":1,"string_object":"foo","string":"foo","float":1.0,"int":1.0,"bool":true,"par0":1,"par1":"a"}
{"id":2,"string_object":null,"string":null,"float":null,"int":null,"bool":null,"par0":1,"par1":"b"}
{"id":3,"string_object":"boo","string":"boo","float":2.0,"int":2.0,"bool":false,"par0":2,"par1":"b"}

It's difficult to say where the problem lies, I installed the version using:

pip uninstall awswrangler -y
pip install git+https://github.com/awslabs/aws-data-wrangler.git@main

Anyways, I guess I will just wait for the official release and see :)