Problem: Storage Service leaks memory

[sevein]

Expected behaviour
Storage Service's allocated memory should be eventually released. This should be true for both synchronous and asynchronous code paths.

A good initial strategy that we've used in MCPClient is to allow workers to complete only a set amount of jobs. In Gunicorn, we have max_requests and max_requests_jitter. This is just a configuration change, but we'd need to see how it impacts the async manager that the process is running. Gunicorn's worker_exit could be used as a window to exit the worker's watchdog gracefully, but in practice we may never want to do that if the watchdog is busy processing tasks that are blocking. We may have to move the async manager into its own process. At that point the manager will need its own max-tasks-per-child.

Current behaviour
Similar to #1482, we observe that the Gunicorn workers that run Storage Service do not release allocated memory. At a first glance this is also due to how we're using lxml, e.g. parsers setting collect_ids=True by default, and writers not being capable to release.

Steps to reproduce

1. Process multiple transfer in Archivematica.
2. Observe memory usage.

Your environment (version of Archivematica, operating system, other relevant details)
SS 0.18 and older.

---

For Artefactual use:

Before you close this issue, you must check off the following:

• All pull requests related to this issue are properly linked
• All pull requests related to this issue have been merged
• A testing plan for this issue has been implemented and passed (testing plan information should be included in the issue body or comments)
• Documentation regarding this issue has been written and merged (if applicable)
• Details about this issue have been added to the release notes (if applicable)

[replaceafill]

This issue is being closed as part of a bulk closure process. Please click here for further information.