Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete no-longer required delivery artifacts #430

Open
donaldgray opened this issue Dec 1, 2022 · 4 comments · May be fixed by #882
Open

Delete no-longer required delivery artifacts #430

donaldgray opened this issue Dec 1, 2022 · 4 comments · May be fixed by #882
Assignees
Labels
API https://dlcs-book.readthedocs.io/en/latest/ delivery-channels engine jira marks issue for crossposting to JIRA sustainability

Comments

@donaldgray
Copy link
Member

Ticket #393 introduced the concept of delivery-channels.

Different combinations of deliveryChannel and mediaType can result in different resources being created and uploaded to S3. We need to catch if a change in deliveryChannel means that some of these resources are now no-longer required and delete them.

This would be difficult to do in the Engine as it gets posted a payload containing the current state of the Asset, at which time the database record has been written to show it is being processed.

A notification raised by IAssetNotificationSender.SendAssetModifiedNotification(ChangeType changeType, Asset? before, Asset? after) would be a sensible place to implement this logic. It's already hooked up for single PUT operations but bulk operations would need extended as they only send details of the AssetId, rather than fully hydrated Asset. This was done to avoid additional queries to database (e.g. 100 updates would require 100 db calls) but it sounds like this is the best place to handle this requirement.

@donaldgray donaldgray added sustainability API https://dlcs-book.readthedocs.io/en/latest/ labels Dec 1, 2022
@donaldgray donaldgray added this to To do in Wellcome 2022 via automation Dec 1, 2022
@donaldgray donaldgray changed the title Delete no-longer required transition artifacts Delete no-longer required delivery artifacts Dec 1, 2022
@donaldgray
Copy link
Member Author

This should also account for items that change Thumbnail policy or auth service.

@sarahos sarahos added the jira marks issue for crossposting to JIRA label Feb 16, 2024
@donaldgray
Copy link
Member Author

Note that this will handle changes to current assets, would require a one off bulk cleanup to remove legacy 'old' derivatives.

@JackLewis-digirati
Copy link
Contributor

This ticket now covers cleaning up thumbnail data from the AssetApplicationMetadata table

@donaldgray
Copy link
Member Author

Note: this requires the addition of a new SQS queue subscribed to existing SNS queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API https://dlcs-book.readthedocs.io/en/latest/ delivery-channels engine jira marks issue for crossposting to JIRA sustainability
Projects
Status: In Progress
4 participants