When a request zip is too large too unzip we get an error

[mathiaswking]

We can change the launch argument to the server to increase the heap size, but we'd like to investigate if we can stream the zip file to disc before unzipping. This should keep the memory usage small and also make for a more robust concurrency situation.

Build a huge extension: https://forum.defold.com/t/native-extensions/4946/178
with a heap size of 500mb

Callstack:

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_161]
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_161]
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_161]
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.8.0_161]
at org.eclipse.jetty.util.MultiPartInputStreamParser$MultiPart.write(MultiPartInputStreamParser.java:135) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.MultiPartInputStreamParser.parse(MultiPartInputStreamParser.java:728) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.MultiPartInputStreamParser.getParts(MultiPartInputStreamParser.java:422) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParts(Request.java:2317) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.extractMultipartParameters(Request.java:519) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:441) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParameters(Request.java:365) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParameter(Request.java:996) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:70) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) ~[spring-boot-actuator-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Server.handle(Server.java:534) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]

[nasthu]

Maybe use Apache Commons FileUpload Streaming API?
https://commons.apache.org/proper/commons-fileupload/streaming.html

The streaming API allows you to trade a little bit of convenience for optimal performance and a low memory profile

You also need to disable Spring's default multipart resolver:
multipart.enabled=false

This has the added advantage that we'll get upload exceptions in the controller so it's easier to handle upload errors.

[nasthu]

How to limit file upload sizes is explained here:
https://commons.apache.org/proper/commons-fileupload/using.html