getResponseBodyAsStream throws OOM

[charlesk40]

OOM is thrown when requesting a file size around 500 mb.
I thought the getResponseBodyAsStream supposed to be used for this exact case where we are requesting a big size data and use stream for it. It seems to me that we are actually writing out the entire data here.

Is there any other ways to get this data using a synchronous call?

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /home/y/logs/yjava_tomcat/java_pid29120.hprof ...
Heap dump file created [1674393227 bytes in 3.792 secs]
Exception in thread "ajp-bio-/127.0.0.1-8009-exec-1" java.lang.OutOfMemoryError: Java heap space
at org.jboss.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42)
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:69)
at org.jboss.netty.buffer.DynamicChannelBuffer.ensureWritableBytes(DynamicChannelBuffer.java:78)
at org.jboss.netty.buffer.DynamicChannelBuffer.writeBytes(DynamicChannelBuffer.java:227)
at org.jboss.netty.buffer.AbstractChannelBuffer.writeBytes(AbstractChannelBuffer.java:441)
at com.ning.http.client.providers.netty.NettyResponse.getResponseBodyAsStream(NettyResponse.java:106)

[jfarcand]

Looking

[simonetripodi]

if you are processing a reply of 500mb you are maybe processing binary files, or a large document response?

[jfarcand]

My own comments

    /* @Override */
    public InputStream getResponseBodyAsStream() throws IOException {
        ChannelBuffer buf = ChannelBuffers.dynamicBuffer();
        for (HttpResponseBodyPart bp : bodyParts) {
            // Ugly. TODO
            // (1) We must remove the downcast,
            // (2) we need a CompositeByteArrayInputStream to avoid
            // copying the bytes.
            if (bp.getClass().isAssignableFrom(ResponseBodyPart.class)) {
                buf.writeBytes(bp.getBodyPartBytes());
            }
        }
        return new ChannelBufferInputStream(buf);
    }

Looks like i need to work on that :-) But I strongly recommend you write your own AsyncHandler and not use the Response (or build it yourself)

[charlesk40]

So, I take it that the Response should not be used in such case where you expect to receive large data in streams using a synchronous call. Is this because the default ResponseBuilder accumulates the response body internally? And are you suggesting to write my own ResponseBuilder that writes to a buffer so that it doesn't accumulate entire data?

[jfarcand]

Salut, I recommend you write your own AsyncHandler and works with the bytes as they comes and never accumulate body in order to later create the Response. You don't need the Response object, so just stream the bytes using an AsyncHandler#onBodyPartReceived.

Alternatively you can use the SimpleAsyncHttpClient with the OutputStreamBobyGenerator ... See this sample

Both solution will works.

[charlesk40]

Thank you.
I think that is what I need to do to NOT accumulate the body.

[charlesk40]

Okay, So I"m able to write to an outputstream (in my case to a file system) as I receive the bodyparts using onBodyPartReceived event. This is all good no accumulating entire data and no OOM.

However, trying to apply the same logic to other bits which is not writing to an outputstream but rather consume it as an inputstream, my original problem still exists.

My use case here is that I would need to read the bytes coming in using an InputStreamReader to read each line. I have found no clean way to do this unless I accumulate the entire data which goes back to my original problem.
Are there any other examples that uses InputStream as a consumer?

[jfarcand]

Salut,

take a look at

   http://sonatype.github.com/async-http-client/apidocs/com/ning/http/client/BodyDeferringAsyncHandler.html

This is what you are looking for. The code is

   https://github.com/sonatype/async-http-client/blob/master/src/main/java/com/ning/http/client/BodyDeferringAsyncHandler.java#L27

Closing the issue as will not fix as I'm not planning to change the API.