102 - persist bundle metadata in separate table #185
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* use separate entity for bundle metadata; validate bundle checksum after download * changes to accommodate new data model * use persisted metadata instead of constructing bundle name * created endpoints * moved block down * use Path * use Path * removed comments * removed comments * tested endpoints * fixed access * throw exception in case of checksum mismatch * show bundle size * seed data * show download option * minor fixes * migration file * cleanup * removed testing seed code * added logging * removed logging * delete datasets before seeding * fixed button style * re-added comment * added logging * removed workflow association from seeded data * fixed duplicate dataset names; added static ids for seeding; updated association * fixed path in seed data * updated origin_path * typo fix * added logging * delete datasets before seeding * made relational not optional on owning side * removed migration * recreated migration * removed hardcoded ids * temporarily removed additional admins in seeding * use .name * WIP - testing * edited seed data * changed endpoint * use /datasets endpoints * tested archive step * tested stage step * cleanup * WIP - testing * use bundle alias for bundle downloads * include bundle info * worker side changes * eslint fixes * cleanup * fixed download path * updated bundle path * cleanup * cleanup * removed logging * moved into var * retrieve bundle info on dataset page * fixed text * added script for bundle population * added script/workflow bundle population * fixed imports/paths; catch exceptions * use _id * added new script to ecosystem.config * trimmed logging * 102 - updated usages of bundle_size --------- Co-authored-by: scadev <scadev@bioloop-dev1.sca.iu.edu> Co-authored-by: scadev@colo25 <scadev@colo25.carbonate.uits.iu.edu>
* 102 - remove outdated usage * use arg --------- Co-authored-by: scadev <scadev@bioloop-dev1.sca.iu.edu> Co-authored-by: scadev@colo25 <scadev@colo25.carbonate.uits.iu.edu>
ri-pandey
commented
Feb 29, 2024
| @@ -85,7 +86,7 @@ def request(self, method, url, *args, **kwargs): | |||
|
|
|||
|
|
|||
| def str_to_int(d: dict, key: str): | |||
| d['du_size'] = utils.parse_number(d.get(key, None)) | |||
| d[key] = utils.parse_number(d.get(key, None)) | |||
There was a problem hiding this comment.
@deepakduggirala I am assuming this was not intentional (maybe it was, because size and du_size are effectively the same value?). Let me know otherwise.
ryanlong89
approved these changes
Feb 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Persist bundle metadata in separate table.
Related Issue(s)
Closes #102, possibly #37 as well.
Changes Made
List the main changes made in this PR. Be as specific as possible.
Checklist
Before submitting this PR, please make sure that:
Additional Information
bundle). This is linked todatasetviadataset_id.dataset.bundle_sizehave been updated to use the size from the newbundletable.bundle_aliasis used to obfuscate the actual location of the bundle on Slate-scratch. I am not reusing thestage_aliasfor this because the stage_alias directory symlinks to the top-level directory inside dataset, instead of the dataset's directory. Therefore downloading the staged dataset with this approach would end up having the bundle inside the dataset. This is the alias provided to users who attempt to download bundles.I have also added a new script (
populate_bundles.py) thathsiutility) from SDA.sync_archived_bundlesworkflow on these datasets, which runs the tasks archive (which populates the bundle metadata in thebundletable), stage, validate, and setup_download steps on each of them, thus preparing them for download.