working group efs inpus outputs

[sanderegg]

Discuss how EFS might be used for inputs/outputs folders

inputs/outputs folders are different from workspace folder:

• registered in the osparc DB (projects, comp_tasks, file_meta_data)
• when an output port has more than one file, it is added to a zip file that is always uploaded to S3, this happens everytime, since there is no way to know to compare the zip files in S3 and local,
• when pulling inputs, the same happen in reverse,

--> if we go away from zipping then we need:

• to find a way to have these inputs, outputs in the projects table for reproducibility of the projects or copying, (this might need discussion with @pcrespov , I am not sure if the JSON definition of inputs/outputs could help there), --> changes in how we handle the projects and comp_tasks table, --> simcore-sdk package, webserver, director-v2, dask-sidecar, and possibly api-server
• the file_meta_data, which could maybe be completely erased, --> refactoring of storage and we dump the whole table, but expect slower times on listing from S3 instead of the DB, (unless accessing directly S3 metadata is fast which I honestly doubt),
• changes in the simcore-sdk package to not zip/unzip and be backward compatible (at least when a zip is found),
• changes in the dask-sidecar service to not zip/unzip and be backward compatible,

On another hand, we could also not list what is inside the folders but just the base folder name.
Nevertheless we currently rely on outputs hashes to know whether a computational service needs to re-compute, this would have to change as well. If some file change inside the folder we need to mark it somehow and this currently is done automatically without accessing S3 but directly via the projects/comp_tasks tables.

[mguidon]

Regarding zips/hashes:

• Can we use a packager (zip/tar) that does not add additional meta-data and thus would give the same hash if the actual data did
  not change?
• Can we calculate the hash for all the files in the package ourselves and attach it as postifx to the filename in S3 or an additional file that we can use to check if a download is required?

In general I think we should not change anything in how the platform handles the inputs/outputs update in this iteration. I would just like to improve startup times by leveraging the EFS infrastructure we developed for the workspace in dynamic services.

[GitHK]

Regarding zips/hashes:

• Can we use a packager (zip/tar) that does not add additional meta-data and thus would give the same hash if the actual data did
  not change?

• FYI: your request implies maintaining two sperate systems for acrhiving/unarchiving the zip one and something else
• Additionally: you need to identify a compression algorithm that produces the same binary each time and that is on par with our current solution (speed wise) we are currently using uncompressed zip archives.

• Can we calculate the hash for all the files in the package ourselves and attach it as postifx to the filename in S3 or an additional file that we can use to check if a download is required?

• we can compute the hash
• the tolling does the hash checks internally
• if you want this it's equivalent to writing your own rclone or s3 sync

In general I think we should not change anything in how the platform handles the inputs/outputs update in this iteration. I would just like to improve startup times by leveraging the EFS infrastructure we developed for the workspace in dynamic services.

I agree with this

[sanderegg]

some interesting article about "deterministic zip": https://zerostride.medium.com/building-deterministic-zip-files-with-built-in-commands-741275116a19

• So it is possible to ensure the zip file keep the same hash if the file did not change (although in our case, since we extract the files, I think their creation/modification times might change between 2 runs - to be verified).
• Nevertheless if 1 file of 1byte changes and in the end we have a 20Gib zip file... this is still not amazing;)

[sanderegg]

in general, if we do not change anything. just adding EFS to the inputs/outputs folder, will probably mostly:

• might make downloading inputs faster when they are unchanged when restarting a service (but not when the service before was run as well - zip sha changes)
• will make downloading outputs faster when restarting the same service
• what kind of costs are we talking about here? https://www.vantage.sh/blog/amazon-efs-pricing

[sanderegg]

Actions for now

• enable EFS for workspace in production -> Great 8.0.2
• see if using deterministic zipping is a quick win (upload) -> Great 8.0.2
• comparing the zip SHA before downloading (download) -> November 8.2

Usage of EFS

• start opening for specific users
• then need some kind of frontend with payments or similar

[sanderegg]

discussed and issues were created accordingly in the main case.