Out-of-memory exception with restricted memory and large artifact size

[pdrd]

Describe the bug
Download of a large artifact (1 GB) from a connector with a restricted memory limit (2 GB) can lead to an Out-Of-Memory exception. A test with a 100 MB file succeeded.

To Reproduce

1. Restrict memory limit of connector to 1-2GB
2. Add a new artifact with data size of 1GB+, e.g. https://speed.hetzner.de/1GB.bin
3. Trying to download the artifact will result in Internal Server Error 500
4. Logs contain OOM-Exception

Expected behavior
The available memory should not restrict the maximum size of a file. It seems like the file might be fully cached in the dsc and then forwarded to the client. This should be done with appropiate chunk sizes/buffers to avoid OOM problems, otherwise the request of multiple big files in parallel might always be problematic - even on machines with more memory.

Logs

2022-04-06T08:54:03,029 [scheduling-1] �[32mINFO�[m - Scanning agreements...
2022-04-06T08:55:04,588 [http-nio-8080-exec-9] �[32mINFO�[m - Sending request to https://speed.hetzner.de/1GB.bin ...
2022-04-06T08:55:04,688 [http-nio-8080-exec-9] �[32mINFO�[m - Successfully received response to request.

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "mssql-jdbc-TimeoutPoller"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Catalina-utility-2"

Exception in thread "http-nio-8080-Poller" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Catalina-utility-1" java.lang.OutOfMemoryError: Java heap space
2022-04-06T08:55:20,047 [http-nio-8080-exec-9] �[31mERROR�[m - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed  nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
java.lang.OutOfMemoryError: Java heap space

[tmberthold]

Duplicate of #27 ?

[tmberthold]

will close it since it's a duplicate, linked this issue in the other for more info