Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

delete_object() does not delete object (or seem to do anything) #2290

Closed
crooksey opened this issue Feb 13, 2020 · 4 comments
Closed

delete_object() does not delete object (or seem to do anything) #2290

crooksey opened this issue Feb 13, 2020 · 4 comments
Assignees
@swetashre
Labels
s3

Comments

@crooksey
Copy link

crooksey commented Feb 13, 2020 •
edited
Loading

I am using the boto3 libary, and trying to delete objects. Note, I am not using versioning.

So I have a simple function:

def remove_aws_object(bucket_name, item_key):
    ''' Provide bucket name and item key, remove from S3
    '''
    s3_client = boto3.client('s3',
                             aws_access_key_id=AWS_ACCESS_KEY,
                             aws_secret_access_key=AWS_SECRET)
    delete = s3_client.delete_object(Bucket=bucket_name, Key=item_key)
    print(delete)

And an example response:

{'ResponseMetadata': {'RequestId': '61F3C195D373B0C5', 'HostId': 'pigcx1wtIN+Y8RU3zJKliWXcXrHXHzpdAOuGBL64x3V9YefQbWXPZi9B/9F', 'HTTPStatusCode': 204, 'HTTPHeaders': {'x-amz-id-2': 'pigcx1wtIN+Y8RU3zJKliWXcXrHXHzpdAOuGBL64x3V9YefQbWXPZi9B/9FLazUSsCds8f4=', 'x-amz-request-id': '61F3C195D373B0C5', 'date': 'Thu, 13 Feb 2020 09:07:50 GMT', 'server': 'AmazonS3'}, 'RetryAttempts': 0}}

I check the S3 console after performing the above, and the file still shows. What am I doing wrong? Also if I then click the tags of the item, there are no tags associated with it.

Note: My API User has full S3 Full Access

I have also tired deleting via an object which has the same results:

def remove_aws_object(bucket_name, item_key):
    ''' Provide bucket name and item key, remove from S3
    '''
    s3_client = boto3.resource('s3',
                             aws_access_key_id=AWS_ACCESS_KEY,
                             aws_secret_access_key=AWS_SECRET)

    my_object = s3_client.Object(bucket_name, item_key)
    my_object.delete()
@crooksey crooksey reopened this Feb 13, 2020
@swetashre swetashre self-assigned this Feb 13, 2020
@swetashre swetashre added the s3 label Feb 13, 2020
@swetashre
Copy link
Contributor

@crooksey - Thank you for your post. S3 delete requests are eventually consistent, which means the delete request can be successful but may not be reflected right away since the delete has to propagate across the region. Its detailed further here https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel

Anyway i am not able to reproduce the issue. Can you please provide me full debug logs ? You can enable log by adding boto3.set_stream_logger('') to your code.

@swetashre swetashre added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 13, 2020
@crooksey
Copy link
Author

Ok, so this is the exact code used, as deleting via an object seems the most widely used way. This is tested on a brand new bucket, all I did was create it, then enable public access.

This is the code used to add the item to the bucket:

with open(temp_file_path, "rb") as f:
        # lets now save this file to S3
        s3_client = boto3.client('s3',
                                 aws_access_key_id=AWS_ACCESS_KEY,
                                 aws_secret_access_key=AWS_SECRET)
        s3_client.upload_fileobj(f, bucket_name, new_key, ExtraArgs={'ACL':'public-read'})

Then I checked the console, my object is there, I then deleted with the following code:

def remove_aws_object(bucket_name, item_key):
    ''' Provide bucket name and item key, remove from S3
    '''
    s3_client = boto3.resource('s3',
                             aws_access_key_id=AWS_ACCESS_KEY,
                             aws_secret_access_key=AWS_SECRET)
    boto3.set_stream_logger('')
    my_object = s3_client.Object(bucket_name, item_key)
    response = my_object.delete()
    print(response)
    return True

And this is my output:

2020-02-14 10:04:08,731 DEBUG [boto3.resources.factory:66][waitress] Loading s3:Object
2020-02-14 10:04:08,731 boto3.resources.factory [DEBUG] Loading s3:Object
2020-02-14 10:04:08,732 DEBUG [botocore.hooks:210][waitress] Event creating-resource-class.s3.Object: calling handler <function lazy_call.<locals>._handler at 0x7fc843ea8290>
2020-02-14 10:04:08,732 botocore.hooks [DEBUG] Event creating-resource-class.s3.Object: calling handler <function lazy_call.<locals>._handler at 0x7fc843ea8290>
2020-02-14 10:04:08,732 DEBUG [boto3.resources.action:81][waitress] Calling s3:delete_object with {'Bucket': 'MYBUCKETNAME', 'Key': 'Aura837e508d-463d-4860-9972-e946e91b40e3.jpg'}
2020-02-14 10:04:08,732 boto3.resources.action [DEBUG] Calling s3:delete_object with {'Bucket': 'MYBUCKETNAME', 'Key': 'Aura837e508d-463d-4860-9972-e946e91b40e3.jpg'}
2020-02-14 10:04:08,732 DEBUG [botocore.hooks:210][waitress] Event before-parameter-build.s3.DeleteObject: calling handler <function validate_bucket_name at 0x7fc852f14ef0>
2020-02-14 10:04:08,732 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <function validate_bucket_name at 0x7fc852f14ef0>
2020-02-14 10:04:08,732 DEBUG [botocore.hooks:210][waitress] Event before-parameter-build.s3.DeleteObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fc8439a4e50>>
2020-02-14 10:04:08,732 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fc8439a4e50>>
2020-02-14 10:04:08,732 DEBUG [botocore.hooks:210][waitress] Event before-parameter-build.s3.DeleteObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7fc8439ee2d0>>
2020-02-14 10:04:08,732 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7fc8439ee2d0>>
2020-02-14 10:04:08,732 DEBUG [botocore.hooks:210][waitress] Event before-parameter-build.s3.DeleteObject: calling handler <function generate_idempotent_uuid at 0x7fc852f14b00>
2020-02-14 10:04:08,732 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <function generate_idempotent_uuid at 0x7fc852f14b00>
2020-02-14 10:04:08,733 DEBUG [botocore.hooks:210][waitress] Event before-call.s3.DeleteObject: calling handler <function add_expect_header at 0x7fc852f1a290>
2020-02-14 10:04:08,733 botocore.hooks [DEBUG] Event before-call.s3.DeleteObject: calling handler <function add_expect_header at 0x7fc852f1a290>
2020-02-14 10:04:08,733 DEBUG [botocore.hooks:210][waitress] Event before-call.s3.DeleteObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fc8439a4e50>>
2020-02-14 10:04:08,733 botocore.hooks [DEBUG] Event before-call.s3.DeleteObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fc8439a4e50>>
2020-02-14 10:04:08,733 DEBUG [botocore.hooks:210][waitress] Event before-call.s3.DeleteObject: calling handler <function inject_api_version_header_if_needed at 0x7fc852f1b5f0>
2020-02-14 10:04:08,733 botocore.hooks [DEBUG] Event before-call.s3.DeleteObject: calling handler <function inject_api_version_header_if_needed at 0x7fc852f1b5f0>
2020-02-14 10:04:08,733 DEBUG [botocore.endpoint:101][waitress] Making request for OperationModel(name=DeleteObject) with params: {'url_path': '/MYBUCKETNAME/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg', 'query_string': {}, 'method': 'DELETE', 'headers': {'User-Agent': 'Boto3/1.11.16 Python/3.7.5 Linux/5.3.0-29-generic Botocore/1.14.16 Resource'}, 'body': b'', 'url': 'https://s3.eu-west-1.amazonaws.com/MYBUCKETNAME/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7fc865591b50>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'MYBUCKETNAME'}}}
2020-02-14 10:04:08,733 botocore.endpoint [DEBUG] Making request for OperationModel(name=DeleteObject) with params: {'url_path': '/MYBUCKETNAME/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg', 'query_string': {}, 'method': 'DELETE', 'headers': {'User-Agent': 'Boto3/1.11.16 Python/3.7.5 Linux/5.3.0-29-generic Botocore/1.14.16 Resource'}, 'body': b'', 'url': 'https://s3.eu-west-1.amazonaws.com/MYBUCKETNAME/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7fc865591b50>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'MYBUCKETNAME'}}}
2020-02-14 10:04:08,733 DEBUG [botocore.hooks:210][waitress] Event request-created.s3.DeleteObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fc85218ad50>>
2020-02-14 10:04:08,733 botocore.hooks [DEBUG] Event request-created.s3.DeleteObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fc85218ad50>>
2020-02-14 10:04:08,733 DEBUG [botocore.hooks:210][waitress] Event choose-signer.s3.DeleteObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fc8520a52d0>>
2020-02-14 10:04:08,733 botocore.hooks [DEBUG] Event choose-signer.s3.DeleteObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fc8520a52d0>>
2020-02-14 10:04:08,733 DEBUG [botocore.hooks:210][waitress] Event choose-signer.s3.DeleteObject: calling handler <function set_operation_specific_signer at 0x7fc852f149e0>
2020-02-14 10:04:08,733 botocore.hooks [DEBUG] Event choose-signer.s3.DeleteObject: calling handler <function set_operation_specific_signer at 0x7fc852f149e0>
2020-02-14 10:04:08,733 DEBUG [botocore.hooks:210][waitress] Event before-sign.s3.DeleteObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fc843a1a2d0>>
2020-02-14 10:04:08,733 botocore.hooks [DEBUG] Event before-sign.s3.DeleteObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fc843a1a2d0>>
2020-02-14 10:04:08,733 DEBUG [botocore.utils:1560][waitress] Defaulting to S3 virtual host style addressing with path style addressing fallback.
2020-02-14 10:04:08,733 botocore.utils [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
2020-02-14 10:04:08,733 DEBUG [botocore.utils:990][waitress] Checking for DNS compatible bucket for: https://s3.eu-west-1.amazonaws.com/MYBUCKETNAME/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg
2020-02-14 10:04:08,733 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.eu-west-1.amazonaws.com/MYBUCKETNAME/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg
2020-02-14 10:04:08,733 DEBUG [botocore.utils:1008][waitress] URI updated to: https://MYBUCKETNAME.s3.eu-west-1.amazonaws.com/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg
2020-02-14 10:04:08,733 botocore.utils [DEBUG] URI updated to: https://MYBUCKETNAME.s3.eu-west-1.amazonaws.com/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg
2020-02-14 10:04:08,734 DEBUG [botocore.auth:364][waitress] Calculating signature using v4 auth.
2020-02-14 10:04:08,734 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-02-14 10:04:08,734 DEBUG [botocore.auth:365][waitress] CanonicalRequest:
DELETE
/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg

host:MYBUCKETNAME.s3.eu-west-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200214T100408Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-02-14 10:04:08,734 botocore.auth [DEBUG] CanonicalRequest:
DELETE
/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg

host:MYBUCKETNAME.s3.eu-west-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200214T100408Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-02-14 10:04:08,734 DEBUG [botocore.auth:367][waitress] StringToSign:
AWS4-HMAC-SHA256
20200214T100408Z
20200214/eu-west-1/s3/aws4_request
f1e7fff7199f9781983bdebfe3b89de417f7b1acadf7ac1b8fa46a9e933dab23
2020-02-14 10:04:08,734 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20200214T100408Z
20200214/eu-west-1/s3/aws4_request
f1e7fff7199f9781983bdebfe3b89de417f7b1acadf7ac1b8fa46a9e933dab23
2020-02-14 10:04:08,734 DEBUG [botocore.auth:369][waitress] Signature:
09105681575615a5637f3781b028589217d37d7dd38ae1209793bdd891e3ba95
2020-02-14 10:04:08,734 botocore.auth [DEBUG] Signature:
09105681575615a5637f3781b028589217d37d7dd38ae1209793bdd891e3ba95
2020-02-14 10:04:08,734 DEBUG [botocore.endpoint:187][waitress] Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=https://MYBUCKETNAME.s3.eu-west-1.amazonaws.com/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg, headers={'User-Agent': b'Boto3/1.11.16 Python/3.7.5 Linux/5.3.0-29-generic Botocore/1.14.16 Resource', 'X-Amz-Date': b'20200214T100408Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA342S3E2UN5L36MAR/20200214/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=09105681575615a5637f3781b028589217d37d7dd38ae1209793bdd891e3ba95', 'Content-Length': '0'}>
2020-02-14 10:04:08,734 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=https://MYBUCKETNAME.s3.eu-west-1.amazonaws.com/Aura837e508d-463d-4860-9972-e946e91b40e3.jpg, headers={'User-Agent': b'Boto3/1.11.16 Python/3.7.5 Linux/5.3.0-29-generic Botocore/1.14.16 Resource', 'X-Amz-Date': b'20200214T100408Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA342S3E2UN5L36MAR/20200214/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=09105681575615a5637f3781b028589217d37d7dd38ae1209793bdd891e3ba95', 'Content-Length': '0'}>
2020-02-14 10:04:08,734 DEBUG [urllib3.connectionpool:823][waitress] Starting new HTTPS connection (1): MYBUCKETNAME.s3.eu-west-1.amazonaws.com:443
2020-02-14 10:04:08,734 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): MYBUCKETNAME.s3.eu-west-1.amazonaws.com:443
2020-02-14 10:04:08,821 DEBUG [urllib3.connectionpool:393][waitress] https://MYBUCKETNAME.s3.eu-west-1.amazonaws.com:443 "DELETE /Aura837e508d-463d-4860-9972-e946e91b40e3.jpg HTTP/1.1" 204 0
2020-02-14 10:04:08,821 urllib3.connectionpool [DEBUG] https://MYBUCKETNAME.s3.eu-west-1.amazonaws.com:443 "DELETE /Aura837e508d-463d-4860-9972-e946e91b40e3.jpg HTTP/1.1" 204 0
2020-02-14 10:04:08,823 DEBUG [botocore.parsers:234][waitress] Response headers: {'x-amz-id-2': 'wIaAthw+C7Pm9kJR3panhPMOg8lRgFtaPWBu9scPNi7hxYCm3ibj04c8RqSyBMugXwhB0eDoVOA=', 'x-amz-request-id': '39B3528814AFC496', 'Date': 'Fri, 14 Feb 2020 10:06:28 GMT', 'Server': 'AmazonS3'}
2020-02-14 10:04:08,823 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': 'wIaAthw+C7Pm9kJR3panhPMOg8lRgFtaPWBu9scPNi7hxYCm3ibj04c8RqSyBMugXwhB0eDoVOA=', 'x-amz-request-id': '39B3528814AFC496', 'Date': 'Fri, 14 Feb 2020 10:06:28 GMT', 'Server': 'AmazonS3'}
2020-02-14 10:04:08,823 DEBUG [botocore.parsers:235][waitress] Response body:
b''
2020-02-14 10:04:08,823 botocore.parsers [DEBUG] Response body:
b''
2020-02-14 10:04:08,824 DEBUG [botocore.hooks:210][waitress] Event needs-retry.s3.DeleteObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fc8439ee410>
2020-02-14 10:04:08,824 botocore.hooks [DEBUG] Event needs-retry.s3.DeleteObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fc8439ee410>
2020-02-14 10:04:08,825 DEBUG [botocore.retryhandler:187][waitress] No retry needed.
2020-02-14 10:04:08,825 botocore.retryhandler [DEBUG] No retry needed.
2020-02-14 10:04:08,825 DEBUG [botocore.hooks:210][waitress] Event needs-retry.s3.DeleteObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fc8439a4e50>>
2020-02-14 10:04:08,825 botocore.hooks [DEBUG] Event needs-retry.s3.DeleteObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fc8439a4e50>>
2020-02-14 10:04:08,825 DEBUG [boto3.resources.action:85][waitress] Response: {'ResponseMetadata': {'RequestId': '39B3528814AFC496', 'HostId': 'wIaAthw+C7Pm9kJR3panhPMOg8lRgFtaPWBu9scPNi7hxYCm3ibj04c8RqSyBMugXwhB0eDoVOA=', 'HTTPStatusCode': 204, 'HTTPHeaders': {'x-amz-id-2': 'wIaAthw+C7Pm9kJR3panhPMOg8lRgFtaPWBu9scPNi7hxYCm3ibj04c8RqSyBMugXwhB0eDoVOA=', 'x-amz-request-id': '39B3528814AFC496', 'date': 'Fri, 14 Feb 2020 10:06:28 GMT', 'server': 'AmazonS3'}, 'RetryAttempts': 0}}
2020-02-14 10:04:08,825 boto3.resources.action [DEBUG] Response: {'ResponseMetadata': {'RequestId': '39B3528814AFC496', 'HostId': 'wIaAthw+C7Pm9kJR3panhPMOg8lRgFtaPWBu9scPNi7hxYCm3ibj04c8RqSyBMugXwhB0eDoVOA=', 'HTTPStatusCode': 204, 'HTTPHeaders': {'x-amz-id-2': 'wIaAthw+C7Pm9kJR3panhPMOg8lRgFtaPWBu9scPNi7hxYCm3ibj04c8RqSyBMugXwhB0eDoVOA=', 'x-amz-request-id': '39B3528814AFC496', 'date': 'Fri, 14 Feb 2020 10:06:28 GMT', 'server': 'AmazonS3'}, 'RetryAttempts': 0}}
{'ResponseMetadata': {'RequestId': '39B3528814AFC496', 'HostId': 'wIaAthw+C7Pm9kJR3panhPMOg8lRgFtaPWBu9scPNi7hxYCm3ibj04c8RqSyBMugXwhB0eDoVOA=', 'HTTPStatusCode': 204, 'HTTPHeaders': {'x-amz-id-2': 'wIaAthw+C7Pm9kJR3panhPMOg8lRgFtaPWBu9scPNi7hxYCm3ibj04c8RqSyBMugXwhB0eDoVOA=', 'x-amz-request-id': '39B3528814AFC496', 'date': 'Fri, 14 Feb 2020 10:06:28 GMT', 'server': 'AmazonS3'}, 'RetryAttempts': 0}}

The file still shows in the S3 console. The files I deleted in my other bucket yesterday, are still showing there this morning some 16 hours later.

@no-response no-response bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 14, 2020
@swetashre
Copy link
Contributor

swetashre commented Feb 14, 2020 •
edited
Loading

@crooksey - Thank you for providing me the debug logs. please make sure if your object is inside a folder then you have to provide the entire path in order to successfully delete the object.

For example if your object path is bucket/folder/object and if you only specify bucket/object then the object won't be deleted. You have to specify the entire path bucket/folder/object something like this:

s3.delete_object(Bucket = 'bucketname', Key = 'folder/object')

I suspect this can be the cause of issue. if this is not the issue i would recommend contacting the s3 team either on their forum or creating a ticket to AWS Support.

@swetashre swetashre added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 14, 2020
@crooksey
Copy link
Author

Thanks, I can confirm this was the issue.

@no-response no-response bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@swetashre swetashre

Labels
s3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@crooksey @swetashre