Fix duplicate feature for large studies

[odeimaiz]

Duplicating large studies fails

[KZzizzle]

related to #466

[pcrespov]

Update on sprint Croissant

Planned

This issue aims to enhance cloning of large studies. We are

• Redesigning the clone feature using the feedback gathered in related issues. For completeness, these are the most relevant:
  • Duplicate should use the mechanism of the templates instead of exporting/importing osparc-simcore#2152: SAN gives guidelines to develop
  • Export/Copy/Publish_Template : Structure & Code Only #560: TN reports bug
  • duplicate/copy an existing osparc node #259: copy nodes! including state data could be achieved as well
  • Fix duplicate feature for large studies #558: bug report
  • 🐛 re: Incident. Webserver: Load and error when exporting / duplicating study osparc-simcore#2805: ops shows heavy use of resources with export
• Redesign will be benchmarked against realistic use case (complexity and size)

Done

• @Konohana0608 (MS) defined a use-case
• ✨ Is560/aiohttp request validations osparc-simcore#3048

Ongoing

• WIP: ♻️✨ Is560/project rest models osparc-simcore#3082: improving API models
• WIP: ♻️ Is560/clone projects osparc-simcore#2974: clone handler and long polling operations

Open

• clone in storage service (awaiting rclone assessment and 5GB upload feature 5Gb limit #617)
• benchmarks at least with MS use case

[pcrespov]

Update on sprint Diolkos

No progress on this issue during this sprint

[sanderegg]

Update on sprint Brutalism

Done

• ✨ Allow copy of closed studies through same mechanism as templates osparc-simcore#3234
• ✨ long running creation of studies osparc-simcore#3235
• ✨ Frontend side: Long running creation of studies osparc-simcore#3260
• ✨ Long running tasks: allow retrieving tasks by user osparc-simcore#3268
• ✨ Add ENV to prevent copying if project too large (⚠️ devops) osparc-simcore#3286

Summary

Thanks to ITISFoundation/osparc-simcore#3077, ITISFoundation/osparc-simcore#3100, ITISFoundation/osparc-simcore#3144 and ITISFoundation/osparc-simcore#3021 a.k.a. complete refactoring of the storage micro-service the problem with copying studies containing files >5Gb is possible and executed in place.

The webserver micro-service and the frontend were modified to allow to use the same mechanism as "creating a project from a template". The operation which can potentially be very long (depending on the amount of data) has been modified to leverage "long running operations in a REST API" which will allow to report a progress (still ongoing). A way to retrieve these long running operations of a specific user will allow the frontend to be F5-refreshed and recover its list of tasks.

1st version caveats

• No progress bar to report study copy
• If webbrowser is disconnected from osparc --> copy will be cancelled
• If webbrowser is refreshed (F5) --> copy will be cancelled
• If webserver is restarted --> copy will be cancelled
• Currently limited to studies <= 30GB until tested with more data --> error message displayed

Ongoing/Next steps

• Generate progress for copy operation from the backend
• Improve UI (auto retries in case of network issue, show copy progress bar, recover copy tasks after refresh or reconnection)
• Handle frontend disconnection/refresh
• Possibly handle webserver restarts
• e2e tests to increase current limit

[sanderegg]

Update on sprint Vaporwave

Done

• ✨ Aiohttp long running tasks client osparc-simcore#3290
• ✨ Webserver reports copy progress osparc-simcore#3287
• remove auto-cancellation when duplicating a study osparc-simcore#3335
• project copy/template copy improvements in the frontend osparc-simcore#3326

Ongoing

• ✨New E2E test to check transfer of large files osparc-simcore#3168

Open

• Not reproducible: need input from @elisabettai Storage cannot handle file links in when saving study to template #604
• In case of web browser refresh or disconnection/later reconnection, currently no recovery of copy tasks (but they will still eventually complete) - will not be implemented for now.

[elisabettai]

Closing this, as discussed with @sanderegg. The only open part is an e2e test, that will be tackled on the side at another point.
Note: as of today, this has not been released to production yet (but will be).
fyi @esraneufeld