computational backend shall make use of multipart links for external clusters

[sanderegg]

currently external clusters are limited to 1 single presigned link, thus limited to 5Gb uploads

[sanderegg]

@elisabettai
currently the ENV VAR STORAGE_DEFAULT_PRESIGNED_LINK_EXPIRATION_SECONDS is set as 6 hours until the computational sidecar can properly request an upload link

[sanderegg]

webserver

• start computation
• creates a token, pass it to director-v2 (ideally with expiration time)
• delete at the end of computation

director-v2

• director-v2 creates a token to access the osparc-api for the project or node that must run (this token is then deleted after the project/node has run) for output files and log file
• instead of passing the links to the output files, the director-v2 passes the token to the dask-sidecar and the url of the public api

dask-sidecar

• after the computational service has run, the sidecar will use the osparc public-api to upload any retrieved output file
• same will happen with the log file

[elisabettai]

fyi, @sanderegg we have now a comp. service (tvb-recon) that takes potentially more than 6h to run. This is expected according to TN.
I'm still testing if it all works on dalco staging.
As a precautionary measure I've set STORAGE_DEFAULT_PRESIGNED_LINK_EXPIRATION_SECONDS in the webserver (on dalco staging) to 43200. Maybe we should change the default value to accommodate this use-case.