✨♻️ Storage refactoring step3 #3144
Conversation
Codecov Report
@@ Coverage Diff @@
## master #3144 +/- ##
======================================
Coverage 77.3% 77.3%
======================================
Files 591 591
Lines 24266 24266
Branches 3148 3148
======================================
Hits 18761 18761
Misses 4923 4923
Partials 582 582
Flags with carried forward coverage won't be shown. Click here to find out more. |
sanderegg
force-pushed
the
enhancement/storage_refactoring_step3
branch
3 times, most recently
from
604d2f2
to
9fe0ae0
Compare
packages/postgres-database/src/simcore_postgres_database/models/file_meta_data.py
Show resolved
Hide resolved
services/web/server/src/simcore_service_webserver/api/v0/openapi.yaml
Outdated
Show resolved
Hide resolved
sanderegg
force-pushed
the
enhancement/storage_refactoring_step3
branch
2 times, most recently
from
8e3100c
to
b859bf9
Compare
packages/postgres-database/src/simcore_postgres_database/models/file_meta_data.py
Show resolved
Hide resolved
| # Rationale: | ||
| - for each upload an entry is created in the file_meta_data table in the database | ||
| - then an upload link (S3/HTTP URL) is created through S3 backend and sent back to the client | ||
| - the client shall upload the file and then notify DSM of completion | ||
| - upon completion the corresponding entry in file_meta_data is updated: | ||
| - the file_size of the uploaded file is set | ||
| - the upload_expiration_date is set to null | ||
|
|
||
| # DSM cleaner: | ||
| - runs at an interval | ||
| - list the entries that are expired in the database by checking "upload_expires_at" column | ||
| - tries to update from S3 the database first, if that fails: | ||
| - removes the entries in the database that are expired: | ||
| - removes the entry |
There was a problem hiding this comment.
Very nice!
Could this also be a good opportunity to clean up the S3 storage?
When people remove nodes, sometimes there is trash left in the S3 storage, this is especially valid for bing/long running studies.
When the DSM cleaner ends it could also check if files in S3 are no longer being used (or are no longer referenced) and remove them.
Maybe such a step could be triggered every X times the DSM cleaner finishes, since it might take some time to check.
There was a problem hiding this comment.
hmm yes, but in a next iteration. let's keep it simple now
services/storage/src/simcore_service_storage/handlers_datasets.py
Outdated
Show resolved
Hide resolved
sanderegg
force-pushed
the
enhancement/storage_refactoring_step3
branch
from
d6c2bed
to
84142e9
Compare
|
Kudos, SonarCloud Quality Gate passed!
|
What do these changes do?
pre-requisite to #3021
This PR brings the step 3 of storage refactoring. It is the biggest step of the refactoring and replaces a lot of the parts in storage micro-service.
Highlights:
file_meta_datatable in postgres (👋 file_uuid, project_name, node_name, file_name, user_name, raw_file_path, display_file_path)file_meta_datatable contains expired uploads (e.g. uploads that were created at some time but never completed, these are then removed from the table to maintain sync)dsm.pymoduledsm.pyfor simcore and datcore following the factory pattern to simplifyRelated issue/s
How to test
Checklist