✨Storage can return a S3 link instead of a presigned link on demand

[sanderegg]

What do these changes do?

This PR let storage return a valid S3 link for upload and download links.
This is a pre-requisite to be able to avail the 5GiB issue in AWS.

Modifies storage micro-service REST API like so:

• adds a link_type query parameter that accepts s3 or presigned (default) when calling GET/PUT entrypoints for download/upload link respectively

Related issue/s

How to test

Checklist

[codecov]

Codecov Report

Merging #3004 (cb2a133) into master (f1ab2d7) will increase coverage by 0.0%.
The diff coverage is 86.9%.

@@          Coverage Diff           @@
##           master   #3004   +/-   ##
======================================
  Coverage    79.6%   79.7%           
======================================
  Files         688     688           
  Lines       28735   28748   +13     
  Branches     3705    3707    +2     
======================================
+ Hits        22899   22922   +23     
+ Misses       5026    5015   -11     
- Partials      810     811    +1     

Flag	Coverage Δ
integrationtests	65.6% <ø> (+<0.1%)	⬆️
unittests	75.4% <86.9%> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ervices/storage/src/simcore_service_storage/dsm.py	72.3% <83.3%> (-0.1%)	⬇️
...es/storage/src/simcore_service_storage/handlers.py	71.6% <100.0%> (+0.3%)	⬆️
...ore_service_director_v2/utils/dask_client_utils.py	78.2% <0.0%> (-0.9%)	⬇️
.../director/src/simcore_service_director/producer.py	62.1% <0.0%> (+0.4%)	⬆️
...tor_v2/modules/dynamic_sidecar/scheduler/events.py	93.8% <0.0%> (+1.1%)	⬆️
...e_service_director_v2/modules/dask_clients_pool.py	93.9% <0.0%> (+1.5%)	⬆️
...rector_v2/modules/comp_scheduler/base_scheduler.py	88.6% <0.0%> (+1.8%)	⬆️
...c/simcore_service_catalog/core/background_tasks.py	68.4% <0.0%> (+2.1%)	⬆️
...ector_v2/modules/comp_scheduler/background_task.py	91.6% <0.0%> (+8.3%)	⬆️

[GitHK]

Very nice. Some observations and questions below.

[GitHK]

Obsservations:

1. if you look above there is this extra_source and extra_location which ware optional. You should raise an error if link_type is also present.
2. If you select dactore/pennsieve as a backend, the s3 links should probably be disabled both in upload and in download. Do you agree?

[sanderegg]

yes. I will check that in the next PR as I anyway need to bring that s3 link also to the frontend.

[GitHK]

👍 thanks!