AWS S3 overwrite mode bug?

[nskrjabins]

Hi,

Maybe I am missing something very obvious but for me the overwrite mode in wr.pandas.read_parquet() is not overwriting but appending. I have followed the tutorial to overwrite from here and here is what I am getting:

The initial overwrite:

The subsequent overwrite is appending:

It is not the end of the world, as I can simply manually delete everything in the target directory but would be good to understand the cause if the issue.

Thanks!

[igorborgest]

Hi @nskrjabins,

I'm don't know what is happening in your environment yet. But I will analyse it on the weekend.

[igorborgest]

@nskrjabins I've tried several times to replicate your situation, but without success.

Even with this extreme situation running on SageMaker with low latency with S3 and overwriting 100 files everything worked like a charm for me.

I also added this case of sequential overwrites in our tests.

[igorborgest]

@nskrjabins can you give more details about your environment?
Which version of wrangler are you using?
In which platform are you running?
Are you sure that you have all permissions related with list and delete objects on S3?

[nskrjabins]

@igorborgest:
Environment details: - conda 4.8.1, Python 3.7.4, pandas 0.25.1, jupyter lab 1.1.4 (for IDE)
Version of wrangler: 0.3.0
Platform: macOS Catalina v10.15.2
Permissions in S3: Yes, I do have all permissions, as I can manually delete object in the same S3 path. Assuming that permission to delete objects manually also allows deleting them with wrangle.

[igorborgest]

@nskrjabins

Could you please update your version to the new 0.3.2 and retry it?

[nskrjabins]

@igorborgest

I've updated it to the 0.3.2 and retried it but still the same error.

[nskrjabins]

I've also tried running the same script from a new virtual environment with Python 3.7.4 and just awswrangler 0.3.2, still had the same error.

[nskrjabins]

Ok - this is really weird. So I double-checked the permission for the bucket - I had both read and write permissions for my account. But I thought, well let's try to make the bucket public - giving read and write permissions to everyone. Once I did it, the overwrite method worked properly.

Now what is weird, is that when I switched I made the bucket private again - removing read and write permissions from everyone, the overwrite method still continued to work properly...

[igorborgest]

@nskrjabins thanks for update us.