Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Looking up versions of storage manifests is very inefficient (and possibly broken for C&D) #1095

Open
alexwlchan opened this issue Jul 29, 2020 · 0 comments

Comments

@alexwlchan
Copy link
Contributor

alexwlchan commented Jul 29, 2020

There's an API to get all the versions of a storage manifest, which:

  • Looks up all the versions in DynamoDB
  • Goes and opens every storage manifest that's listed

This is because we include the creation date of the manifest in the API response, but that's only stored in the S3 object and not in DynamoDB. We should consider if there's a better way to store/present a list of versions – possibly by storing the creation date metadata in DynamoDB.

I suspect looking it up will actually just fail for C&D, but I didn't wait around to find out.

[C&D = Chemist & Druggist, a bag with a very large storage manifest. See https://docs.wellcomecollection.org/storage-service/wellcome-specific-information/awkward-files-and-bags]

@alexwlchan alexwlchan transferred this issue from wellcomecollection/platform Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant