Possible memory leak in storage when copying

[sanderegg]

As discovered it seems the memory usage of the storage service might be leaking.

It could be identified as linked to copying process which uses aioboto3 copy function.

Investigation needed:

• create a test or enhance the test for copying and check the memory usage
• maybe the copy parameters need to be modified

Issue created from a Mattermost message by @mrnicegyu11.

[sanderegg]

https://monitoring.osparc.io/prometheus/graph?g0.expr=container_memory_usage_bytes%7Bname%3D~%22.*simcore.*storage.*%22%7D-container_memory_cache%7Bname%3D~%22.*simcore.*%22%7D&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1d

[mrnicegyu11]

Storage now takes 7.5 Gb on aws-prod:

[sanderegg]

From today, storage seems to go up in memory by steps. It is actually still not quite clear where the memory accumulates. I will need some more testing to resolve this, and find a way to reproduce locally. In AWS prod, the memory increases by steps over the last 3 weeks.

[sanderegg]

Also it seems that this problem is affecting AWS prod only.

[sanderegg]

the latest graphs from grafana look good. closing this case