Skip to content

[Bugfix] Batch delete objects in chunks of 1000 elements when calling delete_objects boto3 s3 api#179

Merged
joakimen merged 1 commit intocapralifecycle:masterfrom
jakob-lj:batch-delete-objects
Dec 9, 2025
Merged

[Bugfix] Batch delete objects in chunks of 1000 elements when calling delete_objects boto3 s3 api#179
joakimen merged 1 commit intocapralifecycle:masterfrom
jakob-lj:batch-delete-objects

Conversation

@jakob-lj
Copy link
Copy Markdown
Contributor

@jakob-lj jakob-lj commented Dec 8, 2025

Todays version of webapp deploy does not support cases when more than 1000 files is scheduled for deletion.
Having more than 1000 files in the objects-list for cleanup, the current error is thrown:

[ERROR] ClientError: An error occurred (MalformedXML) when calling the DeleteObjects operation: The XML you provided was not well-formed or did not validate against our published schema
Traceback (most recent call last):
File "/var/task/webapp_deploy/main.py", line 262, in handler
process(event["ResourceProperties"]["artifactS3Url"])
File "/var/task/webapp_deploy/main.py", line 247, in process
deploy_items = cleanup(deploy_items, target_bucket_url, older_than)
File "/var/task/webapp_deploy/main.py", line 94, in cleanup
cleanup_delete_files(s3_target, delete_files)
File "/var/task/webapp_deploy/main.py", line 83, in cleanup_delete_files

This is due to a limit in the boto3 s3 api, where delete_objects only support a list of 1000 objects [1].

This pull request reduces the number of objects per delete_objects-call, into chunks of 1000 elements per call. This way, the construct supports SPA applications where there might be over 1000 assets to be deleted.

[1] Url fetched at 8th December 2025 20:38 Europe/Oslo: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/delete_objects.html
Text:

The request can contain a list of up to 1,000 keys that you want to delete. In the XML, you provide the object key names, and optionally, version IDs if you want to delete a specific version of the object from a versioning-enabled bucket. For each key, Amazon S3 performs a delete operation and returns the result of that delete, success or failure, in the response. If the object specified in the request isn’t found, Amazon S3 confirms the deletion by returning the result as deleted.

@jakob-lj jakob-lj requested a review from a team as a code owner December 8, 2025 19:40
@joakimen joakimen merged commit 654dd77 into capralifecycle:master Dec 9, 2025
@github-actions
Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 2.5.3 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants